Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- filepath, stringValue
- wasbs://container@myaccount.blob.core.windows.net/demo1,"demo string 1"
- wasbs://container@myaccount.blob.core.windows.net/demo2,"demo string 2"
- wasbs://container@myaccount.blob.core.windows.net/demo3,"demo string 3"
- wasbs://container@myaccount.blob.core.windows.net/demo4,"demo string 4"
- wasbs://container@myaccount.blob.core.windows.net/demo5,"demo string 5"
- from pyspark.sql import *
- from pyspark.sql.types import StringType
- df = spark.read.option("header", True).csv("wasbs://container@myaccount.blob.core.windows.net/demo-data.csv")
- rowList = df.rdd.collect()
- for row in rowList:
- dfRow = spark.createDataFrame([str(row[1])], StringType())
- dfRow.coalesce(1).write.mode("overwrite").text(row[0])
Add Comment
Please, Sign In to add comment