Advertisement
Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- df=sqlContext.read.format('jdbc')
- .options(driver='com.mysql.jdbc.Driver',url="""jdbc:mysql://<host>:3306/<>db?user=<usr>&password=<pass>""",
- dbtable='tbl',
- numPartitions=4 )
- .load()
- df2=df.withColumn('updated_date',to_date(df.updated_at))
- df2.write.parquet(path='s3n://parquet_location',mode='append',partitionBy=['updated_date'])
- partitionColumn,
- lowerBound,
- upperBound,
- numPartitions
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement