daily pastebin goal
38%
SHARE
TWEET

Exor

a guest Jan 28th, 2016 57 Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
  1. df=sqlContext.read.format('jdbc')
  2.    .options(driver='com.mysql.jdbc.Driver',url="""jdbc:mysql://<host>:3306/<>db?user=<usr>&password=<pass>""",
  3.          dbtable='tbl',
  4.          numPartitions=4 )
  5.    .load()
  6.  
  7.  
  8. df2=df.withColumn('updated_date',to_date(df.updated_at))
  9. df2.write.parquet(path='s3n://parquet_location',mode='append',partitionBy=['updated_date'])
  10.    
  11. partitionColumn,
  12. lowerBound,
  13. upperBound,
  14. numPartitions
RAW Paste Data
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy. OK, I Understand
 
Top