Guest User

Untitled

a guest
Oct 26th, 2018
113
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 0.72 KB | None | 0 0
  1. filepath, stringValue
  2. wasbs://container@myaccount.blob.core.windows.net/demo1,"demo string 1"
  3. wasbs://container@myaccount.blob.core.windows.net/demo2,"demo string 2"
  4. wasbs://container@myaccount.blob.core.windows.net/demo3,"demo string 3"
  5. wasbs://container@myaccount.blob.core.windows.net/demo4,"demo string 4"
  6. wasbs://container@myaccount.blob.core.windows.net/demo5,"demo string 5"
  7.  
  8. from pyspark.sql import *
  9. from pyspark.sql.types import StringType
  10.  
  11. df = spark.read.option("header", True).csv("wasbs://container@myaccount.blob.core.windows.net/demo-data.csv")
  12.  
  13. rowList = df.rdd.collect()
  14. for row in rowList:
  15. dfRow = spark.createDataFrame([str(row[1])], StringType())
  16. dfRow.coalesce(1).write.mode("overwrite").text(row[0])
Add Comment
Please, Sign In to add comment