Advertisement
Guest User

Untitled

a guest
Jun 19th, 2019
79
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 0.63 KB | None | 0 0
  1. 123|"ABC"|hello
  2. 124|"AB|hello all
  3. 125|A"B"|hellll
  4.  
  5. val myDf = session.sqlContext.read.format("csv")
  6. .option("charset", "UTF8")
  7. .option("inferSchema", "true")
  8. .option("quote","u0000")
  9. .schema(mySchema)
  10. .option("delimiter", "|")
  11. .option("nullValue", "")
  12. .option("treatEmptyValuesAsNulls", "true")
  13. .load("path to file")
  14.  
  15. myDf.repartition(1).write
  16. .format("com.databricks.spark.csv")
  17. .option("delimiter", "|")
  18. .save("Path to save file")
  19.  
  20. 123|""ABC""|hello
  21. 124|""AB"|hello all
  22. 125|"A"B""|hellll
  23.  
  24. 123|"ABC"|hello
  25. 124|"AB|hello all
  26. 125|A"B"|hellll
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement