Guest User

Untitled

a guest
Jan 23rd, 2018
55
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 1.74 KB | None | 0 0
  1. SparkS = SparkSession.builder
  2. .appName("Test")
  3. .master("local[*]")
  4. .getOrCreate()
  5.  
  6. raw_data = SparkS
  7. .sparkContext
  8. .textFile("C:\Users\...\RawData\nasdaq.csv")
  9.  
  10. print(raw_data.take(3))
  11.  
  12. schema = StructType().add("date", StringType())
  13. .add("open", StringType())
  14. .add("high", StringType())
  15. .add("low", StringType())
  16. .add("close", StringType())
  17. .add("adj_close", StringType())
  18. .add("volume", StringType())
  19.  
  20. geioIP = SparkS.createDataFrame(raw_data,schema)
  21. print(geioIP)
  22.  
  23. DataFrame[date: string, open: string, high: string, low: string, close: string, adj_close: string, volume: string]
  24.  
  25. 18/01/23 12:58:48 ERROR Executor: Exception in task 0.0 in stage 1.0 (TID 1)
  26. org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  27. File "C:spark-2.2.1-bin-hadoop2.7pythonlibpyspark.zippysparkworker.py", line 177, in main
  28. File "C:spark-2.2.1-bin-hadoop2.7pythonlibpyspark.zippysparkworker.py", line 172, in process
  29. File "C:spark-2.2.1-bin-hadoop2.7pythonlibpyspark.zippysparkserializers.py", line 268, in dump_stream
  30. vs = list(itertools.islice(iterator, batch))
  31. File "C:Usersrajnish.kumarAppDataLocalProgramsPythonPython36libsite-packagespysparksqlsession.py", line 520, in prepare
  32. verify_func(obj, schema)
  33. File "C:spark-2.2.1-bin-hadoop2.7pythonlibpyspark.zippysparksqltypes.py", line 1371, in _verify_type
  34. raise TypeError("StructType can not accept object %r in type %s" % (obj, type(obj)))
  35. TypeError: StructType can not accept object '43084,6871.549805,6945.819824,6871.450195,6936.580078,6936.580078,3510420000' in type <class 'str'>
Add Comment
Please, Sign In to add comment