API tools faq
paste
Guest User

Untitled

a guest
May 14th, 2018
220
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 8.08 KB | None | 0 0
raw download clone embed print report
  1. myDf.show(30)
  2. myDf.write.mode(SaveMode.Append)
  3. .jdbc("jdbc:postgresql://<connectionString>", "apptest", props)
  4.  
  5. +---------+------+-----------+-----+
  6. |countryid|kindid|publisherid| rank|
  7. +---------+------+-----------+-----+
  8. | 2| 5| 1440370637| 143|
  9. | 2| 5| 2114144780| 1558|
  10. | 2| 8| 199559784| 3684|
  11. | 2| 5| 1522258372| 1016|
  12. | 2| 9| 918480276| 1051|
  13. | 2| 5| 8788| 619|
  14. | 2| 5| 2760330013| 612|
  15. | 2| 5| 1733022508| 3427|
  16. | 2| 10| 1733020347| 2010|
  17. | 2| 8| 11867| 9431|
  18. | 2| 7| 512462197| 5089|
  19. | 2| 8| 1219783067| 83|
  20. | 2| 5| 1459628648| 4990|
  21. | 2| 5| 2447786955| 155|
  22. | 2| 5| 1985613413| 651|
  23. | 2| 5| 1560140870| 404|
  24. | 2| 5| 1445788622| 4737|
  25. | 2| 7| 2352155047|19288|
  26. | 2| 5| 1667560844| 87|
  27. | 2| 5| 1932266564| 764|
  28. | 2| 5| 1470914641| 31|
  29. | 2| 8| 2601082735| 341|
  30. | 2| 5| 1985796127| 21|
  31. | 2| 8| 510919182| 994|
  32. | 2| 5| 338483845| 3292|
  33. | 2| 5| 2303279333| 809|
  34. | 2| 5| 2454300331| 620|
  35. | 2| 8| 379213322| 570|
  36. | 2| 7| 260492638|20219|
  37. | 2| 5| 1435774759| 78|
  38. +---------+------+-----------+-----+
  39.  
  40. 18/05/14 16:04:05 INFO MemoryStore: Block broadcast_9 stored as values in memory (estimated size 276.3 KB, free 363.7 MB)
  41. 18/05/14 16:04:05 INFO MemoryStore: Block broadcast_9_piece0 stored as bytes in memory (estimated size 23.2 KB, free 363.7 MB)
  42. 18/05/14 16:04:05 INFO BlockManagerInfo: Added broadcast_9_piece0 in memory on 192.168.0.17:62045 (size: 23.2 KB, free: 366.1 MB)
  43. 18/05/14 16:04:05 INFO SparkContext: Created broadcast 9 from newAPIHadoopFile at RedshiftRelation.scala:146
  44. 18/05/14 16:04:05 INFO MemoryStore: Block broadcast_10 stored as values in memory (estimated size 276.3 KB, free 363.4 MB)
  45. 18/05/14 16:04:06 INFO MemoryStore: Block broadcast_10_piece0 stored as bytes in memory (estimated size 23.2 KB, free 363.4 MB)
  46. 18/05/14 16:04:06 INFO BlockManagerInfo: Added broadcast_10_piece0 in memory on 192.168.0.17:62045 (size: 23.2 KB, free: 366.1 MB)
  47. 18/05/14 16:04:06 INFO SparkContext: Created broadcast 10 from newAPIHadoopFile at RedshiftRelation.scala:146
  48. 18/05/14 16:04:06 INFO MemoryStore: Block broadcast_11 stored as values in memory (estimated size 276.3 KB, free 363.1 MB)
  49. 18/05/14 16:04:06 INFO MemoryStore: Block broadcast_11_piece0 stored as bytes in memory (estimated size 23.2 KB, free 363.1 MB)
  50. 18/05/14 16:04:06 INFO BlockManagerInfo: Added broadcast_11_piece0 in memory on 192.168.0.17:62045 (size: 23.2 KB, free: 366.0 MB)
  51. 18/05/14 16:04:06 INFO SparkContext: Created broadcast 11 from newAPIHadoopFile at RedshiftRelation.scala:146
  52. 18/05/14 16:04:06 INFO MemoryStore: Block broadcast_12 stored as values in memory (estimated size 276.3 KB, free 362.8 MB)
  53. 18/05/14 16:04:06 INFO MemoryStore: Block broadcast_12_piece0 stored as bytes in memory (estimated size 23.2 KB, free 362.8 MB)
  54. 18/05/14 16:04:06 INFO BlockManagerInfo: Added broadcast_12_piece0 in memory on 192.168.0.17:62045 (size: 23.2 KB, free: 366.0 MB)
  55. 18/05/14 16:04:06 INFO SparkContext: Created broadcast 12 from newAPIHadoopFile at RedshiftRelation.scala:146
  56. 18/05/14 16:04:06 INFO FileInputFormat: Total input paths to process : 1
  57. 18/05/14 16:04:06 INFO FileInputFormat: Total input paths to process : 1
  58. 18/05/14 16:04:06 INFO FileInputFormat: Total input paths to process : 1
  59. 18/05/14 16:04:06 INFO FileInputFormat: Total input paths to process : 1
  60. 18/05/14 16:04:06 INFO SparkContext: Starting job: jdbc at MyAppWriter.scala:29
  61. 18/05/14 16:04:06 INFO DAGScheduler: Got job 1 (jdbc at MyAppWriter.scala:29) with 4 output partitions
  62. 18/05/14 16:04:06 INFO DAGScheduler: Final stage: ResultStage 1 (jdbc at MyAppWriter.scala:29)
  63. 18/05/14 16:04:06 INFO DAGScheduler: Parents of final stage: List()
  64. 18/05/14 16:04:06 INFO DAGScheduler: Missing parents: List()
  65. 18/05/14 16:04:06 INFO DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[28] at jdbc at MyAppWriter.scala:29), which has no missing parents
  66. 18/05/14 16:04:06 INFO MemoryStore: Block broadcast_13 stored as values in memory (estimated size 13.2 KB, free 362.8 MB)
  67. 18/05/14 16:04:06 INFO MemoryStore: Block broadcast_13_piece0 stored as bytes in memory (estimated size 6.8 KB, free 362.8 MB)
  68. 18/05/14 16:04:06 INFO BlockManagerInfo: Added broadcast_13_piece0 in memory on 192.168.0.17:62045 (size: 6.8 KB, free: 366.0 MB)
  69. 18/05/14 16:04:06 INFO SparkContext: Created broadcast 13 from broadcast at DAGScheduler.scala:1006
  70. 18/05/14 16:04:06 INFO DAGScheduler: Submitting 4 missing tasks from ResultStage 1 (MapPartitionsRDD[28] at jdbc at MyAppWriter.scala:29) (first 15 tasks are for partitions Vector(0, 1, 2, 3))
  71. 18/05/14 16:04:06 INFO TaskSchedulerImpl: Adding task set 1.0 with 4 tasks
  72. 18/05/14 16:04:06 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, localhost, executor driver, partition 0, PROCESS_LOCAL, 5072 bytes)
  73. 18/05/14 16:04:06 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 2, localhost, executor driver, partition 1, PROCESS_LOCAL, 5072 bytes)
  74. 18/05/14 16:04:06 INFO TaskSetManager: Starting task 2.0 in stage 1.0 (TID 3, localhost, executor driver, partition 2, PROCESS_LOCAL, 5072 bytes)
  75. 18/05/14 16:04:06 INFO TaskSetManager: Starting task 3.0 in stage 1.0 (TID 4, localhost, executor driver, partition 3, PROCESS_LOCAL, 5072 bytes)
  76. 18/05/14 16:04:06 INFO Executor: Running task 0.0 in stage 1.0 (TID 1)
  77. 18/05/14 16:04:06 INFO Executor: Running task 1.0 in stage 1.0 (TID 2)
  78. 18/05/14 16:04:06 INFO Executor: Running task 2.0 in stage 1.0 (TID 3)
  79. 18/05/14 16:04:06 INFO Executor: Running task 3.0 in stage 1.0 (TID 4)
  80. 18/05/14 16:04:06 INFO NewHadoopRDD: Input split: s3n://mybucket-example/mydata/spark/temp/38c92e24-8f9d-486a-b84a-cfdb5964cb15/0000_part_00:0+55124
  81. 18/05/14 16:04:06 INFO NewHadoopRDD: Input split: s3n://mybucket-example/mydata/spark/temp/38c92e24-8f9d-486a-b84a-cfdb5964cb15/0002_part_00:0+54686
  82. 18/05/14 16:04:06 INFO NewHadoopRDD: Input split: s3n://mybucket-example/mydata/spark/temp/38c92e24-8f9d-486a-b84a-cfdb5964cb15/0003_part_00:0+53651
  83. 18/05/14 16:04:06 INFO NewHadoopRDD: Input split: s3n://mybucket-example/mydata/spark/temp/38c92e24-8f9d-486a-b84a-cfdb5964cb15/0001_part_00:0+53475
  84. 18/05/14 16:04:06 INFO NativeS3FileSystem: Opening 's3n://mybucket-example/mydata/spark/temp/38c92e24-8f9d-486a-b84a-cfdb5964cb15/0000_part_00' for reading
  85. 18/05/14 16:04:06 INFO NativeS3FileSystem: Opening 's3n://mybucket-example/mydata/spark/temp/38c92e24-8f9d-486a-b84a-cfdb5964cb15/0001_part_00' for reading
  86. 18/05/14 16:04:06 INFO NativeS3FileSystem: Opening 's3n://mybucket-example/mydata/spark/temp/38c92e24-8f9d-486a-b84a-cfdb5964cb15/0002_part_00' for reading
  87. 18/05/14 16:04:06 INFO NativeS3FileSystem: Opening 's3n://mybucket-example/mydata/spark/temp/38c92e24-8f9d-486a-b84a-cfdb5964cb15/0003_part_00' for reading
  88. 18/05/14 16:07:47 INFO Executor: Finished task 0.0 in stage 1.0 (TID 1). 1236 bytes result sent to driver
  89. 18/05/14 16:07:47 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 220999 ms on localhost (executor driver) (1/4)
  90. 18/05/14 16:10:40 INFO Executor: Finished task 2.0 in stage 1.0 (TID 3). 1193 bytes result sent to driver
  91. 18/05/14 16:10:40 INFO TaskSetManager: Finished task 2.0 in stage 1.0 (TID 3) in 393573 ms on localhost (executor driver) (2/4)
  92. 18/05/14 16:13:35 INFO Executor: Finished task 1.0 in stage 1.0 (TID 2). 1193 bytes result sent to driver
  93. 18/05/14 16:13:35 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 2) in 569130 ms on localhost (executor driver) (3/4)
  94. 18/05/14 16:16:30 INFO Executor: Finished task 3.0 in stage 1.0 (TID 4). 1193 bytes result sent to driver
  95. 18/05/14 16:16:30 INFO TaskSetManager: Finished task 3.0 in stage 1.0 (TID 4) in 744012 ms on localhost (executor driver) (4/4)
  96. 18/05/14 16:16:30 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool
  97. 18/05/14 16:16:30 INFO DAGScheduler: ResultStage 1 (jdbc at MyAppWriter.scala:29) finished in 744.014 s
  98. 18/05/14 16:16:30 INFO DAGScheduler: Job 1 finished: jdbc at MyAppWriter.scala:29, took 744.064359 s
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!