Guest User

Untitled

a guest
May 12th, 2018
173
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
  1. 2018-05-12 14:14:46 INFO  DAGScheduler:54 - ResultStage 0 (csv at NativeMethodAccessorImpl.java:0) finished in 0.326 s
  2. 2018-05-12 14:14:46 INFO  DAGScheduler:54 - Job 0 finished: csv at NativeMethodAccessorImpl.java:0, took 0.375615 s
  3. 2018-05-12 14:14:46 INFO  FileSourceStrategy:54 - Pruning directories with:
  4. 2018-05-12 14:14:46 INFO  FileSourceStrategy:54 - Post-Scan Filters:
  5. 2018-05-12 14:14:46 INFO  FileSourceStrategy:54 - Output Data Schema: struct<value: string>
  6. 2018-05-12 14:14:46 INFO  FileSourceScanExec:54 - Pushed Filters:
  7. 2018-05-12 14:14:46 INFO  CodeGenerator:54 - Code generated in 7.4815 ms
  8. 2018-05-12 14:14:46 INFO  MemoryStore:54 - Block broadcast_2 stored as values in memory (estimated size 277.6 KB, free 4.1 GB)
  9. 2018-05-12 14:14:46 INFO  MemoryStore:54 - Block broadcast_2_piece0 stored as bytes in memory (estimated size 23.4 KB, free 4.1 GB)
  10. 2018-05-12 14:14:46 INFO  BlockManagerInfo:54 - Added broadcast_2_piece0 in memory on a77d7f063d74:36956 (size: 23.4 KB, free: 4.1 GB)
  11. 2018-05-12 14:14:46 INFO  SparkContext:54 - Created broadcast 2 from csv at NativeMethodAccessorImpl.java:0
  12. 2018-05-12 14:14:46 INFO  FileSourceScanExec:54 - Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes.
  13. 2018-05-12 14:14:46 INFO  SparkContext:54 - Starting job: csv at NativeMethodAccessorImpl.java:0
  14. 2018-05-12 14:14:46 INFO  DAGScheduler:54 - Got job 1 (csv at NativeMethodAccessorImpl.java:0) with 1 output partitions
  15. 2018-05-12 14:14:46 INFO  DAGScheduler:54 - Final stage: ResultStage 1 (csv at NativeMethodAccessorImpl.java:0)
  16. 2018-05-12 14:14:46 INFO  DAGScheduler:54 - Parents of final stage: List()
  17. 2018-05-12 14:14:46 INFO  DAGScheduler:54 - Missing parents: List()
  18. 2018-05-12 14:14:46 INFO  DAGScheduler:54 - Submitting ResultStage 1 (MapPartitionsRDD[8] at csv at NativeMethodAccessorImpl.java:0), which has no missing parents
  19. 2018-05-12 14:14:46 INFO  MemoryStore:54 - Block broadcast_3 stored as values in memory (estimated size 13.0 KB, free 4.1 GB)
  20. 2018-05-12 14:14:46 INFO  MemoryStore:54 - Block broadcast_3_piece0 stored as bytes in memory (estimated size 7.0 KB, free 4.1 GB)
  21. 2018-05-12 14:14:46 INFO  BlockManagerInfo:54 - Added broadcast_3_piece0 in memory on a77d7f063d74:36956 (size: 7.0 KB, free: 4.1 GB)
  22. 2018-05-12 14:14:46 INFO  SparkContext:54 - Created broadcast 3 from broadcast at DAGScheduler.scala:1039
  23. 2018-05-12 14:14:46 INFO  DAGScheduler:54 - Submitting 1 missing tasks from ResultStage 1 (MapPartitionsRDD[8] at csv at NativeMethodAccessorImpl.java:0) (first 15 tasks are for partitions Vector(0))
  24. 2018-05-12 14:14:46 INFO  TaskSchedulerImpl:54 - Adding task set 1.0 with 1 tasks
  25. 2018-05-12 14:14:46 INFO  TaskSetManager:54 - Starting task 0.0 in stage 1.0 (TID 1, localhost, executor driver, partition 0, PROCESS_LOCAL, 8313 bytes)
  26. 2018-05-12 14:14:46 INFO  Executor:54 - Running task 0.0 in stage 1.0 (TID 1)
  27. 2018-05-12 14:14:46 INFO  FileScanRDD:54 - Reading File path: file:///tmp/src/input/csv/lasertriangulation.csv, range: 0-1726, partition values: [empty row]
  28. 2018-05-12 14:14:46 INFO  Executor:54 - Finished task 0.0 in stage 1.0 (TID 1). 1761 bytes result sent to driver
  29. 2018-05-12 14:14:46 INFO  TaskSetManager:54 - Finished task 0.0 in stage 1.0 (TID 1) in 51 ms on localhost (executor driver) (1/1)
  30. 2018-05-12 14:14:46 INFO  DAGScheduler:54 - ResultStage 1 (csv at NativeMethodAccessorImpl.java:0) finished in 0.074 s
  31. 2018-05-12 14:14:46 INFO  TaskSchedulerImpl:54 - Removed TaskSet 1.0, whose tasks have all completed, from pool
  32. 2018-05-12 14:14:46 INFO  DAGScheduler:54 - Job 1 finished: csv at NativeMethodAccessorImpl.java:0, took 0.079536 s
  33. 2018-05-12 14:14:46 INFO  FileSourceStrategy:54 - Pruning directories with:
  34. 2018-05-12 14:14:46 INFO  FileSourceStrategy:54 - Post-Scan Filters:
  35. 2018-05-12 14:14:46 INFO  FileSourceStrategy:54 - Output Data Schema: struct<InspectionType: string, Altitude_UM: double, Floor_UM: double, SurfaceFlatness_UM: double, Height_UM: double ... 18 more fields>
  36. 2018-05-12 14:14:46 INFO  FileSourceScanExec:54 - Pushed Filters:
  37. 2018-05-12 14:14:46 INFO  MemoryStore:54 - Block broadcast_4 stored as values in memory (estimated size 277.9 KB, free 4.1 GB)
  38. 2018-05-12 14:14:46 INFO  MemoryStore:54 - Block broadcast_4_piece0 stored as bytes in memory (estimated size 23.4 KB, free 4.1 GB)
  39. 2018-05-12 14:14:46 INFO  BlockManagerInfo:54 - Added broadcast_4_piece0 in memory on a77d7f063d74:36956 (size: 23.4 KB, free: 4.1 GB)
  40. 2018-05-12 14:14:46 INFO  SparkContext:54 - Created broadcast 4 from javaToPython at NativeMethodAccessorImpl.java:0
  41. 2018-05-12 14:14:47 INFO  FileSourceScanExec:54 - Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes.
  42. Traceback (most recent call last):
  43.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 235, in dump
  44.     return Pickler.dump(self, obj)
  45.   File "/usr/lib/python3.5/pickle.py", line 408, in dump
  46.     self.save(obj)
  47.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  48.     f(self, obj) # Call unbound method with explicit self
  49.   File "/usr/lib/python3.5/pickle.py", line 740, in save_tuple
  50.     save(element)
  51.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  52.     f(self, obj) # Call unbound method with explicit self
  53.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 378, in save_function
  54.     self.save_function_tuple(obj)
  55.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 529, in save_function_tuple
  56.     save(closure_values)
  57.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  58.     f(self, obj) # Call unbound method with explicit self
  59.   File "/usr/lib/python3.5/pickle.py", line 770, in save_list
  60.     self._batch_appends(obj)
  61.   File "/usr/lib/python3.5/pickle.py", line 794, in _batch_appends
  62.     save(x)
  63.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  64.     f(self, obj) # Call unbound method with explicit self
  65.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 378, in save_function
  66.     self.save_function_tuple(obj)
  67.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 529, in save_function_tuple
  68.     save(closure_values)
  69.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  70.     f(self, obj) # Call unbound method with explicit self
  71.   File "/usr/lib/python3.5/pickle.py", line 770, in save_list
  72.     self._batch_appends(obj)
  73.   File "/usr/lib/python3.5/pickle.py", line 794, in _batch_appends
  74.     save(x)
  75.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  76.     f(self, obj) # Call unbound method with explicit self
  77.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 378, in save_function
  78.     self.save_function_tuple(obj)
  79.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 529, in save_function_tuple
  80.     save(closure_values)
  81.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  82.     f(self, obj) # Call unbound method with explicit self
  83.   File "/usr/lib/python3.5/pickle.py", line 770, in save_list
  84.     self._batch_appends(obj)
  85.   File "/usr/lib/python3.5/pickle.py", line 794, in _batch_appends
  86.     save(x)
  87.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  88.     f(self, obj) # Call unbound method with explicit self
  89.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 378, in save_function
  90.     self.save_function_tuple(obj)
  91.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 529, in save_function_tuple
  92.     save(closure_values)
  93.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  94.     f(self, obj) # Call unbound method with explicit self
  95.   File "/usr/lib/python3.5/pickle.py", line 770, in save_list
  96.     self._batch_appends(obj)
  97.   File "/usr/lib/python3.5/pickle.py", line 797, in _batch_appends
  98.     save(tmp[0])
  99.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  100.     f(self, obj) # Call unbound method with explicit self
  101.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 378, in save_function
  102.     self.save_function_tuple(obj)
  103.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 529, in save_function_tuple
  104.     save(closure_values)
  105.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  106.     f(self, obj) # Call unbound method with explicit self
  107.   File "/usr/lib/python3.5/pickle.py", line 770, in save_list
  108.     self._batch_appends(obj)
  109.   File "/usr/lib/python3.5/pickle.py", line 797, in _batch_appends
  110.     save(tmp[0])
  111.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  112.     f(self, obj) # Call unbound method with explicit self
  113.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 372, in save_function
  114.     self.save_function_tuple(obj)
  115.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 529, in save_function_tuple
  116.     save(closure_values)
  117.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  118.     f(self, obj) # Call unbound method with explicit self
  119.   File "/usr/lib/python3.5/pickle.py", line 770, in save_list
  120.     self._batch_appends(obj)
  121.   File "/usr/lib/python3.5/pickle.py", line 797, in _batch_appends
  122.     save(tmp[0])
  123.   File "/usr/lib/python3.5/pickle.py", line 520, in save
  124.     self.save_reduce(obj=obj, *rv)
  125.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 780, in save_reduce
  126.     save(cls)
  127.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  128.     f(self, obj) # Call unbound method with explicit self
  129.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 639, in save_global
  130.     self.save_dynamic_class(obj)
  131.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 476, in save_dynamic_class
  132.     save(clsdict)
  133.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  134.     f(self, obj) # Call unbound method with explicit self
  135.   File "/usr/lib/python3.5/pickle.py", line 810, in save_dict
  136.     self._batch_setitems(obj.items())
  137.   File "/usr/lib/python3.5/pickle.py", line 836, in _batch_setitems
  138.     save(v)
  139.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  140.     f(self, obj) # Call unbound method with explicit self
  141.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 372, in save_function
  142.     self.save_function_tuple(obj)
  143.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 525, in save_function_tuple
  144.     save(f_globals)
  145.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  146.     f(self, obj) # Call unbound method with explicit self
  147.   File "/usr/lib/python3.5/pickle.py", line 810, in save_dict
  148.     self._batch_setitems(obj.items())
  149.   File "/usr/lib/python3.5/pickle.py", line 841, in _batch_setitems
  150.     save(v)
  151.   File "/usr/lib/python3.5/pickle.py", line 520, in save
  152.     self.save_reduce(obj=obj, *rv)
  153.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 804, in save_reduce
  154.     save(state)
  155.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  156.     f(self, obj) # Call unbound method with explicit self
  157.   File "/usr/lib/python3.5/pickle.py", line 810, in save_dict
  158.     self._batch_setitems(obj.items())
  159.   File "/usr/lib/python3.5/pickle.py", line 836, in _batch_setitems
  160.     save(v)
  161.   File "/usr/lib/python3.5/pickle.py", line 520, in save
  162.     self.save_reduce(obj=obj, *rv)
  163.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 804, in save_reduce
  164.     save(state)
  165.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  166.     f(self, obj) # Call unbound method with explicit self
  167.   File "/usr/lib/python3.5/pickle.py", line 810, in save_dict
  168.     self._batch_setitems(obj.items())
  169.   File "/usr/lib/python3.5/pickle.py", line 836, in _batch_setitems
  170.     save(v)
  171.   File "/usr/lib/python3.5/pickle.py", line 495, in save
  172.     rv = reduce(self.proto)
  173.   File "/usr/lib/python3.5/socket.py", line 175, in __getstate__
  174.     raise TypeError("Cannot serialize socket object")
  175. TypeError: Cannot serialize socket object
  176. Traceback (most recent call last):
  177.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 235, in dump
  178.     return Pickler.dump(self, obj)
  179.   File "/usr/lib/python3.5/pickle.py", line 408, in dump
  180.     self.save(obj)
  181.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  182.     f(self, obj) # Call unbound method with explicit self
  183.   File "/usr/lib/python3.5/pickle.py", line 740, in save_tuple
  184.     save(element)
  185.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  186.     f(self, obj) # Call unbound method with explicit self
  187.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 378, in save_function
  188.     self.save_function_tuple(obj)
  189.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 529, in save_function_tuple
  190.     save(closure_values)
  191.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  192.     f(self, obj) # Call unbound method with explicit self
  193.   File "/usr/lib/python3.5/pickle.py", line 770, in save_list
  194.     self._batch_appends(obj)
  195.   File "/usr/lib/python3.5/pickle.py", line 794, in _batch_appends
  196.     save(x)
  197.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  198.     f(self, obj) # Call unbound method with explicit self
  199.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 378, in save_function
  200.     self.save_function_tuple(obj)
  201.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 529, in save_function_tuple
  202.     save(closure_values)
  203.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  204.     f(self, obj) # Call unbound method with explicit self
  205.   File "/usr/lib/python3.5/pickle.py", line 770, in save_list
  206.     self._batch_appends(obj)
  207.   File "/usr/lib/python3.5/pickle.py", line 794, in _batch_appends
  208.     save(x)
  209.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  210.     f(self, obj) # Call unbound method with explicit self
  211.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 378, in save_function
  212.     self.save_function_tuple(obj)
  213.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 529, in save_function_tuple
  214.     save(closure_values)
  215.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  216.     f(self, obj) # Call unbound method with explicit self
  217.   File "/usr/lib/python3.5/pickle.py", line 770, in save_list
  218.     self._batch_appends(obj)
  219.   File "/usr/lib/python3.5/pickle.py", line 794, in _batch_appends
  220.     save(x)
  221.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  222.     f(self, obj) # Call unbound method with explicit self
  223.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 378, in save_function
  224.     self.save_function_tuple(obj)
  225.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 529, in save_function_tuple
  226.     save(closure_values)
  227.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  228.     f(self, obj) # Call unbound method with explicit self
  229.   File "/usr/lib/python3.5/pickle.py", line 770, in save_list
  230.     self._batch_appends(obj)
  231.   File "/usr/lib/python3.5/pickle.py", line 797, in _batch_appends
  232.     save(tmp[0])
  233.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  234.     f(self, obj) # Call unbound method with explicit self
  235.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 378, in save_function
  236.     self.save_function_tuple(obj)
  237.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 529, in save_function_tuple
  238.     save(closure_values)
  239.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  240.     f(self, obj) # Call unbound method with explicit self
  241.   File "/usr/lib/python3.5/pickle.py", line 770, in save_list
  242.     self._batch_appends(obj)
  243.   File "/usr/lib/python3.5/pickle.py", line 797, in _batch_appends
  244.     save(tmp[0])
  245.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  246.     f(self, obj) # Call unbound method with explicit self
  247.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 372, in save_function
  248.     self.save_function_tuple(obj)
  249.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 529, in save_function_tuple
  250.     save(closure_values)
  251.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  252.     f(self, obj) # Call unbound method with explicit self
  253.   File "/usr/lib/python3.5/pickle.py", line 770, in save_list
  254.     self._batch_appends(obj)
  255.   File "/usr/lib/python3.5/pickle.py", line 797, in _batch_appends
  256.     save(tmp[0])
  257.   File "/usr/lib/python3.5/pickle.py", line 520, in save
  258.     self.save_reduce(obj=obj, *rv)
  259.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 780, in save_reduce
  260.     save(cls)
  261.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  262.     f(self, obj) # Call unbound method with explicit self
  263.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 639, in save_global
  264.     self.save_dynamic_class(obj)
  265.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 476, in save_dynamic_class
  266.     save(clsdict)
  267.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  268.     f(self, obj) # Call unbound method with explicit self
  269.   File "/usr/lib/python3.5/pickle.py", line 810, in save_dict
  270.     self._batch_setitems(obj.items())
  271.   File "/usr/lib/python3.5/pickle.py", line 836, in _batch_setitems
  272.     save(v)
  273.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  274.     f(self, obj) # Call unbound method with explicit self
  275.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 372, in save_function
  276.     self.save_function_tuple(obj)
  277.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 525, in save_function_tuple
  278.     save(f_globals)
  279.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  280.     f(self, obj) # Call unbound method with explicit self
  281.   File "/usr/lib/python3.5/pickle.py", line 810, in save_dict
  282.     self._batch_setitems(obj.items())
  283.   File "/usr/lib/python3.5/pickle.py", line 841, in _batch_setitems
  284.     save(v)
  285.   File "/usr/lib/python3.5/pickle.py", line 520, in save
  286.     self.save_reduce(obj=obj, *rv)
  287.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 804, in save_reduce
  288.     save(state)
  289.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  290.     f(self, obj) # Call unbound method with explicit self
  291.   File "/usr/lib/python3.5/pickle.py", line 810, in save_dict
  292.     self._batch_setitems(obj.items())
  293.   File "/usr/lib/python3.5/pickle.py", line 836, in _batch_setitems
  294.     save(v)
  295.   File "/usr/lib/python3.5/pickle.py", line 520, in save
  296.     self.save_reduce(obj=obj, *rv)
  297.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 804, in save_reduce
  298.     save(state)
  299.   File "/usr/lib/python3.5/pickle.py", line 475, in save
  300.     f(self, obj) # Call unbound method with explicit self
  301.   File "/usr/lib/python3.5/pickle.py", line 810, in save_dict
  302.     self._batch_setitems(obj.items())
  303.   File "/usr/lib/python3.5/pickle.py", line 836, in _batch_setitems
  304.     save(v)
  305.   File "/usr/lib/python3.5/pickle.py", line 495, in save
  306.     rv = reduce(self.proto)
  307.   File "/usr/lib/python3.5/socket.py", line 175, in __getstate__
  308.     raise TypeError("Cannot serialize socket object")
  309. TypeError: Cannot serialize socket object
  310.  
  311. During handling of the above exception, another exception occurred:
  312.  
  313. Traceback (most recent call last):
  314.   File "/tmp/load.py", line 406, in <module>
  315.     lt.load(mi, v_client)
  316.   File "/tmp/load.py", line 124, in load
  317.     df.rdd.foreachPartition(bulk_writing)
  318.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/rdd.py", line 814, in foreachPartition
  319.     self.mapPartitions(func).count()  # Force evaluation
  320.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/rdd.py", line 1056, in count
  321.     return self.mapPartitions(lambda i: [sum(1 for _ in i)]).sum()
  322.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/rdd.py", line 1047, in sum
  323.     return self.mapPartitions(lambda x: [sum(x)]).fold(0, operator.add)
  324.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/rdd.py", line 921, in fold
  325.     vals = self.mapPartitions(func).collect()
  326.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/rdd.py", line 824, in collect
  327.     port = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd())
  328.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/rdd.py", line 2470, in _jrdd
  329.     self._jrdd_deserializer, profiler)
  330.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/rdd.py", line 2403, in _wrap_function
  331.     pickled_command, broadcast_vars, env, includes = _prepare_for_python_RDD(sc, command)
  332.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/rdd.py", line 2389, in _prepare_for_python_RDD
  333.     pickled_command = ser.dumps(command)
  334.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/serializers.py", line 568, in dumps
  335.     return cloudpickle.dumps(obj, 2)
  336.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 918, in dumps
  337.     cp.dump(obj)
  338.   File "/usr/local/lib/python3.5/dist-packages/pyspark-2.3.0-py3.5.egg/pyspark/cloudpickle.py", line 249, in dump
  339.     raise pickle.PicklingError(msg)
  340. _pickle.PicklingError: Could not serialize object: TypeError: Cannot serialize socket object
RAW Paste Data