API tools faq
paste
lamiastella

Untitled

lamiastella
Oct 29th, 2015
267
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 10.96 KB | None | 0 0
raw download clone embed print report
  1. ubuntu@group-3-vm1:~/software/spark-1.5.0-bin-hadoop2.6/python$ python mona_tfidf.py
  2. Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
  3. 15/10/30 02:01:36 INFO SparkContext: Running Spark version 1.5.0
  4. 15/10/30 02:01:36 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  5. 15/10/30 02:01:36 INFO SecurityManager: Changing view acls to: ubuntu
  6. 15/10/30 02:01:36 INFO SecurityManager: Changing modify acls to: ubuntu
  7. 15/10/30 02:01:36 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(ubuntu); users with modify permissions: Set(ubuntu)
  8. 15/10/30 02:01:37 INFO Slf4jLogger: Slf4jLogger started
  9. 15/10/30 02:01:37 INFO Remoting: Starting remoting
  10. 15/10/30 02:01:38 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://[email protected]:55723]
  11. 15/10/30 02:01:38 INFO Utils: Successfully started service 'sparkDriver' on port 55723.
  12. 15/10/30 02:01:38 INFO SparkEnv: Registering MapOutputTracker
  13. 15/10/30 02:01:38 INFO SparkEnv: Registering BlockManagerMaster
  14. 15/10/30 02:01:38 INFO DiskBlockManager: Created local directory at /home/ubuntu/storage/data/spark/rdds_shuffle/blockmgr-50f40e46-46f4-496f-8dc9-e33bfe8dacad
  15. 15/10/30 02:01:38 INFO MemoryStore: MemoryStore started with capacity 530.3 MB
  16. 15/10/30 02:01:38 INFO HttpFileServer: HTTP File server directory is /home/ubuntu/storage/data/spark/rdds_shuffle/spark-c0d962b3-f86a-4726-8767-f3a8bbfb5b6c/httpd-b3945536-c2a4-4da6-8538-724e7ccb403e
  17. 15/10/30 02:01:38 INFO HttpServer: Starting HTTP Server
  18. 15/10/30 02:01:38 INFO Utils: Successfully started service 'HTTP file server' on port 33066.
  19. 15/10/30 02:01:38 INFO SparkEnv: Registering OutputCommitCoordinator
  20. 15/10/30 02:01:38 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
  21. 15/10/30 02:01:38 WARN Utils: Service 'SparkUI' could not bind on port 4041. Attempting port 4042.
  22. 15/10/30 02:01:38 WARN Utils: Service 'SparkUI' could not bind on port 4042. Attempting port 4043.
  23. 15/10/30 02:01:38 WARN Utils: Service 'SparkUI' could not bind on port 4043. Attempting port 4044.
  24. 15/10/30 02:01:38 INFO Utils: Successfully started service 'SparkUI' on port 4044.
  25. 15/10/30 02:01:38 INFO SparkUI: Started SparkUI at http://10.0.1.31:4044
  26. 15/10/30 02:01:38 WARN MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set.
  27. 15/10/30 02:01:38 INFO AppClient$ClientEndpoint: Connecting to master spark://10.0.1.31:7077...
  28. 15/10/30 02:01:39 INFO SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20151030020139-0158
  29. 15/10/30 02:01:39 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 55225.
  30. 15/10/30 02:01:39 INFO NettyBlockTransferService: Server created on 55225
  31. 15/10/30 02:01:39 INFO BlockManagerMaster: Trying to register BlockManager
  32. 15/10/30 02:01:39 INFO BlockManagerMasterEndpoint: Registering block manager 10.0.1.31:55225 with 530.3 MB RAM, BlockManagerId(driver, 10.0.1.31, 55225)
  33. 15/10/30 02:01:39 INFO BlockManagerMaster: Registered BlockManager
  34. 15/10/30 02:01:39 INFO EventLoggingListener: Logging events to file:/home/ubuntu/storage/logs/app-20151030020139-0158
  35. 15/10/30 02:01:39 INFO SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
  36. 15/10/30 02:01:40 INFO MemoryStore: ensureFreeSpace(238288) called with curMem=0, maxMem=556038881
  37. 15/10/30 02:01:40 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 232.7 KB, free 530.1 MB)
  38. 15/10/30 02:01:40 INFO MemoryStore: ensureFreeSpace(20622) called with curMem=238288, maxMem=556038881
  39. 15/10/30 02:01:40 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 20.1 KB, free 530.0 MB)
  40. 15/10/30 02:01:40 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.0.1.31:55225 (size: 20.1 KB, free: 530.3 MB)
  41. 15/10/30 02:01:40 INFO SparkContext: Created broadcast 0 from textFile at NativeMethodAccessorImpl.java:-2
  42. PythonRDD[2] at RDD at PythonRDD.scala:43
  43. 15/10/30 02:01:41 INFO FileInputFormat: Total input paths to process : 1
  44. 15/10/30 02:01:41 INFO SparkContext: Starting job: treeAggregate at IDF.scala:56
  45. 15/10/30 02:01:41 INFO DAGScheduler: Got job 0 (treeAggregate at IDF.scala:56) with 2 output partitions
  46. 15/10/30 02:01:41 INFO DAGScheduler: Final stage: ResultStage 0(treeAggregate at IDF.scala:56)
  47. 15/10/30 02:01:41 INFO DAGScheduler: Parents of final stage: List()
  48. 15/10/30 02:01:41 INFO DAGScheduler: Missing parents: List()
  49. 15/10/30 02:01:41 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[6] at treeAggregate at IDF.scala:56), which has no missing parents
  50. 15/10/30 02:01:41 INFO MemoryStore: ensureFreeSpace(8136) called with curMem=258910, maxMem=556038881
  51. 15/10/30 02:01:41 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 7.9 KB, free 530.0 MB)
  52. 15/10/30 02:01:41 INFO MemoryStore: ensureFreeSpace(4664) called with curMem=267046, maxMem=556038881
  53. 15/10/30 02:01:41 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 4.6 KB, free 530.0 MB)
  54. 15/10/30 02:01:41 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 10.0.1.31:55225 (size: 4.6 KB, free: 530.3 MB)
  55. 15/10/30 02:01:41 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:861
  56. 15/10/30 02:01:41 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 0 (MapPartitionsRDD[6] at treeAggregate at IDF.scala:56)
  57. 15/10/30 02:01:41 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
  58. 15/10/30 02:01:56 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
  59. 15/10/30 02:02:11 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
  60. 15/10/30 02:02:26 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
  61. 15/10/30 02:02:41 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
  62. 15/10/30 02:02:56 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
  63. 15/10/30 02:03:11 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
  64. 15/10/30 02:03:26 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
  65. 15/10/30 02:03:41 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
  66. 15/10/30 02:03:56 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
  67. 15/10/30 02:04:11 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
  68. 15/10/30 02:04:26 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
  69. 15/10/30 02:04:41 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
  70. 15/10/30 02:04:56 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
  71. 15/10/30 02:05:11 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
  72. 15/10/30 02:05:26 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
  73. 15/10/30 02:05:41 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
  74. 15/10/30 02:05:56 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
  75. 15/10/30 02:06:11 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
  76. 15/10/30 02:06:26 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
  77. 15/10/30 02:06:41 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
  78. 15/10/30 02:06:56 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
  79. 15/10/30 02:07:11 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
  80. 15/10/30 02:07:26 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
  81. 15/10/30 02:07:41 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
  82. 15/10/30 02:07:56 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
  83. 15/10/30 02:08:11 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
  84. 15/10/30 02:08:26 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
  85. 15/10/30 02:08:41 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
  86. 15/10/30 02:08:56 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
  87. 15/10/30 02:09:11 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
  88. 15/10/30 02:09:26 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!