Advertisement
Guest User

Untitled

a guest
Jun 25th, 2019
101
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 0.50 KB | None | 0 0
  1. def SaveData(row):
  2.  
  3. ...
  4.  
  5. # read csv string
  6. df = spark.read
  7. .option("header", True)
  8. .option("delimiter","|")
  9. .option("quote", """)
  10. .option("nullValue", "\N")
  11. .schema(schemaMapping)
  12. .csv(csvData)
  13.  
  14. df.write.format("delta").mode("append").save(tableLocation)
  15. #df.write.saveAsTable(tableName)
  16. #df.saveAsTable(tableName, format='parquet', mode='append')
  17.  
  18. query = dfDEHubStream.writeStream.foreach(SaveData).start()
  19.  
  20. CREATE TABLE tableName
  21. USING DELTA
  22. LOCATION tableLocation
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement