Advertisement
Guest User

Untitled

a guest
Jul 11th, 2016
81
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 0.54 KB | None | 0 0
  1. from pyspark.sql import SQLContext
  2. from pyspark import SparkContext
  3.  
  4. sc = SparkContext(appName="Connect Spark with Redshift")
  5. sql_context = SQLContext(sc)
  6. sc._jsc.hadoopConfiguration().set("fs.s3n.awsAccessKeyId", <ACCESSID>)
  7. sc._jsc.hadoopConfiguration().set("fs.s3n.awsSecretAccessKey", <ACCESSKEY>)
  8.  
  9. df = sql_context.read
  10. .option("url", "jdbc:redshift://example.coyf2i236wts.eu-central- 1.redshift.amazonaws.com:5439/agcdb?user=user&password=pwd")
  11. .option("dbtable", "table_name")
  12. .option("tempdir", "bucket")
  13. .load()
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement