Advertisement
Guest User

Untitled

a guest
Jan 28th, 2016
89
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 0.41 KB | None | 0 0
  1. df=sqlContext.read.format('jdbc')
  2. .options(driver='com.mysql.jdbc.Driver',url="""jdbc:mysql://<host>:3306/<>db?user=<usr>&password=<pass>""",
  3. dbtable='tbl',
  4. numPartitions=4 )
  5. .load()
  6.  
  7.  
  8. df2=df.withColumn('updated_date',to_date(df.updated_at))
  9. df2.write.parquet(path='s3n://parquet_location',mode='append',partitionBy=['updated_date'])
  10.  
  11. partitionColumn,
  12. lowerBound,
  13. upperBound,
  14. numPartitions
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement