API tools faq
paste
Advertisement
Guest User

Untitled

a guest
May 12th, 2018
231
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
SPARK 22.09 KB | None | 0 0
raw download clone embed print report
  1. 2018-05-12 14:14:46 INFO  DAGScheduler:54 - ResultStage 0 (csv at NativeMethodAccessorImpl.java:0) finished in 0.326 s
  2. 2018-05-12 14:14:46 INFO  DAGScheduler:54 - Job 0 finished: csv at NativeMethodAccessorImpl.java:0, took 0.375615 s
  3. 2018-05-12 14:14:46 INFO  FileSourceStrategy:54 - Pruning directories with:
  4. 2018-05-12 14:14:46 INFO  FileSourceStrategy:54 - Post-Scan Filters:
  5. 2018-05-12 14:14:46 INFO  FileSourceStrategy:54 - Output Data Schema: struct<value: string>
  6. 2018-05-12 14:14:46 INFO  FileSourceScanExec:54 - Pushed Filters:
  7. 2018-05-12 14:14:46 INFO  CodeGenerator:54 - Code generated in 7.4815 ms
  8. 2018-05-12 14:14:46 INFO  MemoryStore:54 - Block broadcast_2 stored as values in memory (estimated size 277.6 KB, free 4.1 GB)
  9. 2018-05-12 14:14:46 INFO  MemoryStore:54 - Block broadcast_2_piece0 stored as bytes in memory (estimated size 23.4 KB, free 4.1 GB)
  10. 2018-05-12 14:14:46 INFO  BlockManagerInfo:54 - Added broadcast_2_piece0 in memory on a77d7f063d74:36956 (size: 23.4 KB, free: 4.1 GB)
  11. 2018-05-12 14:14:46 INFO  SparkContext:54 - Created broadcast 2 from csv at NativeMethodAccessorImpl.java:0
  12. 2018-05-12 14:14:46 INFO  FileSourceScanExec:54 - Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes.
  13. 2018-05-12 14:14:46 INFO  SparkContext:54 - Starting job: csv at NativeMethodAccessorImpl.java:0
  14. 2018-05-12 14:14:46 INFO  DAGScheduler:54 - Got job 1 (csv at NativeMethodAccessorImpl.java:0) with 1 output partitions
  15. 2018-05-12 14:14:46 INFO  DAGScheduler:54 - Final stage: ResultStage 1 (csv at NativeMethodAccessorImpl.java:0)
  16. 2018-05-12 14:14:46 INFO  DAGScheduler:54 - Parents of final stage: List()
  17. 2018-05-12 14:14:46 INFO  DAGScheduler:54 - Missing parents: List()
  18. 2018-05-12 14:14:46 INFO  DAGScheduler:54 - Submitting ResultStage 1 (MapPartitionsRDD[8] at csv at NativeMethodAccessorImpl.java:0), which has no missing parents
  19. 2018-05-12 14:14:46 INFO  MemoryStore:54 - Block broadcast_3 stored as values in memory (estimated size 13.0 KB, free 4.1 GB)
  20. 2018-05-12 14:14:46 INFO  MemoryStore:54 - Block broadcast_3_piece0 stored as bytes in memory (estimated size 7.0 KB, free 4.1 GB)
  21. 2018-05-12 14:14:46 INFO  BlockManagerInfo:54 - Added broadcast_3_piece0 in memory on a77d7f063d74:36956 (size: 7.0 KB, free: 4.1 GB)
  22. 2018-05-12 14:14:46 INFO  SparkContext:54 - Created broadcast 3 from broadcast at DAGScheduler.scala:1039
  23. 2018-05-12 14:14:46 INFO  DAGScheduler:54 - Submitting 1 missing tasks from ResultStage 1 (MapPartitionsRDD[8] at csv at NativeMethodAccessorImpl.java:0) (first 15 tasks are for partitions Vector(0))
  24. 2018-05-12 14:14:46 INFO  TaskSchedulerImpl:54 - Adding task set 1.0 with 1 tasks
  25. 2018-05-12 14:14:46 INFO  TaskSetManager:54 - Starting task 0.0 in stage 1.0 (TID 1, localhost, executor driver, partition 0, PROCESS_LOCAL, 8313 bytes)
  26. 2018-05-12 14:14:46 INFO  Executor:54 - Running task 0.0 in stage 1.0 (TID 1)
  27. 2018-05-12 14:14:46 INFO  FileScanRDD:54 - Reading File path: file:///tmp/src/input/csv/lasertriangulation.csv, range: 0-1726, partition values: [empty row]
  28. 2018-05-12 14:14:46 INFO  Executor:54 - Finished task 0.0 in stage 1.0 (TID 1). 1761 bytes result sent to driver
  29. 2018-05-12 14:14:46 INFO  TaskSetManager:54 - Finished task 0.0 in stage 1.0 (TID 1) in 51 ms on localhost (executor driver) (1/1)
  30. 2018-05-12 14:14:46 INFO  DAGScheduler:54 - ResultStage 1 (csv at NativeMethodAccessorImpl.java:0) finished in 0.074 s
  31. 2018-05-12 14:14:46 INFO  TaskSchedulerImpl:54 - Removed TaskSet 1.0, whose tasks have all completed, from pool
  32. 2018-05-12 14:14:46 INFO  DAGScheduler:54 - Job 1 finished: csv at NativeMethodAccessorImpl.java:0, took 0.079536 s
  33. 2018-05-12 14:14:46 INFO  FileSourceStrategy:54 - Pruning directories with:
  34. 2018-05-12 14:14:46 INFO  FileSourceStrategy:54 - Post-Scan Filters:
  35. 2018-05-12 14:14:46 INFO  FileSourceStrategy:54 - Output Data Schema: struct<InspectionType: string, Altitude_UM: double, Floor_UM: double, SurfaceFlatness_UM: double, Height_UM: double ... 18 more fields>
  36. 2018-05-12 14:14:46 INFO  FileSourceScanExec:54 - Pushed Filters:
  37. 2018-05-12 14:14:46 INFO  MemoryStore:54 - Block broadcast_4 stored as values in memory (estimated size 277.9 KB, free 4.1 GB)
  38. 2018-05-12 14:14:46 INFO  MemoryStore:54 - Block broadcast_4_piece0 stored as bytes in memory (estimated size 23.4 KB, free 4.1 GB)
  39. 2018-05-12 14:14:46 INFO  BlockManagerInfo:54 - Added broadcast_4_piece0 in memory on a77d7f063d74:36956 (size: 23.4 KB, free: 4.1 GB)
  40. 2018-05-12 14:14:46 INFO  SparkContext:54 - Created broadcast 4 from javaToPython at NativeMethodAccessorImpl.java:0
  41. 2018-05-12 14:14:47 INFO  FileSourceScanExec:54 - Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes.
  42. Traceback (most recent call last):
  43.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 235, in dump
  44.     return Pickler.dump(self, obj)
  45.   File "/usr/lib/python3.5/pickle.py", line 408, in dump
  46.     self.save(obj)
  47.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  48.     f(self, obj) # Call unbound method with explicit self
  49.   File "/usr/lib/python3.5/pickle.py", line 740, in save_tuple
  50.     save(element)
  51.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  52.     f(self, obj) # Call unbound method with explicit self
  53.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 378, in save_function
  54.     self.save_function_tuple(obj)
  55.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 529, in save_function_tuple
  56.     save(closure_values)
  57.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  58.     f(self, obj) # Call unbound method with explicit self
  59.   File "/usr/lib/python3.5/pickle.py", line 770, in save_list
  60.     self._batch_appends(obj)
  61.   File "/usr/lib/python3.5/pickle.py", line 794, in _batch_appends
  62.     save(x)
  63.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  64.     f(self, obj) # Call unbound method with explicit self
  65.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 378, in save_function
  66.     self.save_function_tuple(obj)
  67.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 529, in save_function_tuple
  68.     save(closure_values)
  69.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  70.     f(self, obj) # Call unbound method with explicit self
  71.   File "/usr/lib/python3.5/pickle.py", line 770, in save_list
  72.     self._batch_appends(obj)
  73.   File "/usr/lib/python3.5/pickle.py", line 794, in _batch_appends
  74.     save(x)
  75.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  76.     f(self, obj) # Call unbound method with explicit self
  77.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 378, in save_function
  78.     self.save_function_tuple(obj)
  79.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 529, in save_function_tuple
  80.     save(closure_values)
  81.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  82.     f(self, obj) # Call unbound method with explicit self
  83.   File "/usr/lib/python3.5/pickle.py", line 770, in save_list
  84.     self._batch_appends(obj)
  85.   File "/usr/lib/python3.5/pickle.py", line 794, in _batch_appends
  86.     save(x)
  87.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  88.     f(self, obj) # Call unbound method with explicit self
  89.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 378, in save_function
  90.     self.save_function_tuple(obj)
  91.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 529, in save_function_tuple
  92.     save(closure_values)
  93.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  94.     f(self, obj) # Call unbound method with explicit self
  95.   File "/usr/lib/python3.5/pickle.py", line 770, in save_list
  96.     self._batch_appends(obj)
  97.   File "/usr/lib/python3.5/pickle.py", line 797, in _batch_appends
  98.     save(tmp[0])
  99.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  100.     f(self, obj) # Call unbound method with explicit self
  101.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 378, in save_function
  102.     self.save_function_tuple(obj)
  103.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 529, in save_function_tuple
  104.     save(closure_values)
  105.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  106.     f(self, obj) # Call unbound method with explicit self
  107.   File "/usr/lib/python3.5/pickle.py", line 770, in save_list
  108.     self._batch_appends(obj)
  109.   File "/usr/lib/python3.5/pickle.py", line 797, in _batch_appends
  110.     save(tmp[0])
  111.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  112.     f(self, obj) # Call unbound method with explicit self
  113.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 372, in save_function
  114.     self.save_function_tuple(obj)
  115.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 529, in save_function_tuple
  116.     save(closure_values)
  117.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  118.     f(self, obj) # Call unbound method with explicit self
  119.   File "/usr/lib/python3.5/pickle.py", line 770, in save_list
  120.     self._batch_appends(obj)
  121.   File "/usr/lib/python3.5/pickle.py", line 797, in _batch_appends
  122.     save(tmp[0])
  123.   File "/usr/lib/python3.5/pickle.py", line 520, in save
  124.     self.save_reduce(obj=obj, *rv)
  125.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 780, in save_reduce
  126.     save(cls)
  127.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  128.     f(self, obj) # Call unbound method with explicit self
  129.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 639, in save_global
  130.     self.save_dynamic_class(obj)
  131.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 476, in save_dynamic_class
  132.     save(clsdict)
  133.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  134.     f(self, obj) # Call unbound method with explicit self
  135.   File "/usr/lib/python3.5/pickle.py", line 810, in save_dict
  136.     self._batch_setitems(obj.items())
  137.   File "/usr/lib/python3.5/pickle.py", line 836, in _batch_setitems
  138.     save(v)
  139.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  140.     f(self, obj) # Call unbound method with explicit self
  141.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 372, in save_function
  142.     self.save_function_tuple(obj)
  143.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 525, in save_function_tuple
  144.     save(f_globals)
  145.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  146.     f(self, obj) # Call unbound method with explicit self
  147.   File "/usr/lib/python3.5/pickle.py", line 810, in save_dict
  148.     self._batch_setitems(obj.items())
  149.   File "/usr/lib/python3.5/pickle.py", line 841, in _batch_setitems
  150.     save(v)
  151.   File "/usr/lib/python3.5/pickle.py", line 520, in save
  152.     self.save_reduce(obj=obj, *rv)
  153.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 804, in save_reduce
  154.     save(state)
  155.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  156.     f(self, obj) # Call unbound method with explicit self
  157.   File "/usr/lib/python3.5/pickle.py", line 810, in save_dict
  158.     self._batch_setitems(obj.items())
  159.   File "/usr/lib/python3.5/pickle.py", line 836, in _batch_setitems
  160.     save(v)
  161.   File "/usr/lib/python3.5/pickle.py", line 520, in save
  162.     self.save_reduce(obj=obj, *rv)
  163.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 804, in save_reduce
  164.     save(state)
  165.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  166.     f(self, obj) # Call unbound method with explicit self
  167.   File "/usr/lib/python3.5/pickle.py", line 810, in save_dict
  168.     self._batch_setitems(obj.items())
  169.   File "/usr/lib/python3.5/pickle.py", line 836, in _batch_setitems
  170.     save(v)
  171.   File "/usr/lib/python3.5/pickle.py", line 495, in save
  172.     rv = reduce(self.proto)
  173.   File "/usr/lib/python3.5/socket.py", line 175, in __getstate__
  174.     raise TypeError("Cannot serialize socket object")
  175. TypeError: Cannot serialize socket object
  176. Traceback (most recent call last):
  177.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 235, in dump
  178.     return Pickler.dump(self, obj)
  179.   File "/usr/lib/python3.5/pickle.py", line 408, in dump
  180.     self.save(obj)
  181.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  182.     f(self, obj) # Call unbound method with explicit self
  183.   File "/usr/lib/python3.5/pickle.py", line 740, in save_tuple
  184.     save(element)
  185.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  186.     f(self, obj) # Call unbound method with explicit self
  187.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 378, in save_function
  188.     self.save_function_tuple(obj)
  189.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 529, in save_function_tuple
  190.     save(closure_values)
  191.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  192.     f(self, obj) # Call unbound method with explicit self
  193.   File "/usr/lib/python3.5/pickle.py", line 770, in save_list
  194.     self._batch_appends(obj)
  195.   File "/usr/lib/python3.5/pickle.py", line 794, in _batch_appends
  196.     save(x)
  197.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  198.     f(self, obj) # Call unbound method with explicit self
  199.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 378, in save_function
  200.     self.save_function_tuple(obj)
  201.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 529, in save_function_tuple
  202.     save(closure_values)
  203.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  204.     f(self, obj) # Call unbound method with explicit self
  205.   File "/usr/lib/python3.5/pickle.py", line 770, in save_list
  206.     self._batch_appends(obj)
  207.   File "/usr/lib/python3.5/pickle.py", line 794, in _batch_appends
  208.     save(x)
  209.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  210.     f(self, obj) # Call unbound method with explicit self
  211.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 378, in save_function
  212.     self.save_function_tuple(obj)
  213.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 529, in save_function_tuple
  214.     save(closure_values)
  215.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  216.     f(self, obj) # Call unbound method with explicit self
  217.   File "/usr/lib/python3.5/pickle.py", line 770, in save_list
  218.     self._batch_appends(obj)
  219.   File "/usr/lib/python3.5/pickle.py", line 794, in _batch_appends
  220.     save(x)
  221.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  222.     f(self, obj) # Call unbound method with explicit self
  223.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 378, in save_function
  224.     self.save_function_tuple(obj)
  225.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 529, in save_function_tuple
  226.     save(closure_values)
  227.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  228.     f(self, obj) # Call unbound method with explicit self
  229.   File "/usr/lib/python3.5/pickle.py", line 770, in save_list
  230.     self._batch_appends(obj)
  231.   File "/usr/lib/python3.5/pickle.py", line 797, in _batch_appends
  232.     save(tmp[0])
  233.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  234.     f(self, obj) # Call unbound method with explicit self
  235.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 378, in save_function
  236.     self.save_function_tuple(obj)
  237.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 529, in save_function_tuple
  238.     save(closure_values)
  239.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  240.     f(self, obj) # Call unbound method with explicit self
  241.   File "/usr/lib/python3.5/pickle.py", line 770, in save_list
  242.     self._batch_appends(obj)
  243.   File "/usr/lib/python3.5/pickle.py", line 797, in _batch_appends
  244.     save(tmp[0])
  245.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  246.     f(self, obj) # Call unbound method with explicit self
  247.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 372, in save_function
  248.     self.save_function_tuple(obj)
  249.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 529, in save_function_tuple
  250.     save(closure_values)
  251.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  252.     f(self, obj) # Call unbound method with explicit self
  253.   File "/usr/lib/python3.5/pickle.py", line 770, in save_list
  254.     self._batch_appends(obj)
  255.   File "/usr/lib/python3.5/pickle.py", line 797, in _batch_appends
  256.     save(tmp[0])
  257.   File "/usr/lib/python3.5/pickle.py", line 520, in save
  258.     self.save_reduce(obj=obj, *rv)
  259.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 780, in save_reduce
  260.     save(cls)
  261.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  262.     f(self, obj) # Call unbound method with explicit self
  263.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 639, in save_global
  264.     self.save_dynamic_class(obj)
  265.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 476, in save_dynamic_class
  266.     save(clsdict)
  267.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  268.     f(self, obj) # Call unbound method with explicit self
  269.   File "/usr/lib/python3.5/pickle.py", line 810, in save_dict
  270.     self._batch_setitems(obj.items())
  271.   File "/usr/lib/python3.5/pickle.py", line 836, in _batch_setitems
  272.     save(v)
  273.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  274.     f(self, obj) # Call unbound method with explicit self
  275.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 372, in save_function
  276.     self.save_function_tuple(obj)
  277.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 525, in save_function_tuple
  278.     save(f_globals)
  279.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  280.     f(self, obj) # Call unbound method with explicit self
  281.   File "/usr/lib/python3.5/pickle.py", line 810, in save_dict
  282.     self._batch_setitems(obj.items())
  283.   File "/usr/lib/python3.5/pickle.py", line 841, in _batch_setitems
  284.     save(v)
  285.   File "/usr/lib/python3.5/pickle.py", line 520, in save
  286.     self.save_reduce(obj=obj, *rv)
  287.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 804, in save_reduce
  288.     save(state)
  289.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  290.     f(self, obj) # Call unbound method with explicit self
  291.   File "/usr/lib/python3.5/pickle.py", line 810, in save_dict
  292.     self._batch_setitems(obj.items())
  293.   File "/usr/lib/python3.5/pickle.py", line 836, in _batch_setitems
  294.     save(v)
  295.   File "/usr/lib/python3.5/pickle.py", line 520, in save
  296.     self.save_reduce(obj=obj, *rv)
  297.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 804, in save_reduce
  298.     save(state)
  299.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  300.     f(self, obj) # Call unbound method with explicit self
  301.   File "/usr/lib/python3.5/pickle.py", line 810, in save_dict
  302.     self._batch_setitems(obj.items())
  303.   File "/usr/lib/python3.5/pickle.py", line 836, in _batch_setitems
  304.     save(v)
  305.   File "/usr/lib/python3.5/pickle.py", line 495, in save
  306.     rv = reduce(self.proto)
  307.   File "/usr/lib/python3.5/socket.py", line 175, in __getstate__
  308.     raise TypeError("Cannot serialize socket object")
  309. TypeError: Cannot serialize socket object
  310.  
  311. During handling of the above exception, another exception occurred:
  312.  
  313. Traceback (most recent call last):
  314.   File "/tmp/load.py", line 406, in <module>
  315.     lt.load(mi, v_client)
  316.   File "/tmp/load.py", line 124, in load
  317.     df.rdd.foreachPartition(bulk_writing)
  318.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/rdd.py", line 814, in foreachPartition
  319.     self.mapPartitions(func).count()  # Force evaluation
  320.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/rdd.py", line 1056, in count
  321.     return self.mapPartitions(lambda i: [sum(1 for _ in i)]).sum()
  322.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/rdd.py", line 1047, in sum
  323.     return self.mapPartitions(lambda x: [sum(x)]).fold(0, operator.add)
  324.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/rdd.py", line 921, in fold
  325.     vals = self.mapPartitions(func).collect()
  326.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/rdd.py", line 824, in collect
  327.     port = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd())
  328.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/rdd.py", line 2470, in _jrdd
  329.     self._jrdd_deserializer, profiler)
  330.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/rdd.py", line 2403, in _wrap_function
  331.     pickled_command, broadcast_vars, env, includes = _prepare_for_python_RDD(sc, command)
  332.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/rdd.py", line 2389, in _prepare_for_python_RDD
  333.     pickled_command = ser.dumps(command)
  334.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/serializers.py", line 568, in dumps
  335.     return cloudpickle.dumps(obj, 2)
  336.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 918, in dumps
  337.     cp.dump(obj)
  338.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 249, in dump
  339.     raise pickle.PicklingError(msg)
  340. _pickle.PicklingError: Could not serialize object: TypeError: Cannot serialize socket object
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!