API tools faq
paste
Guest User

Spark / Beam job stopping

a guest
Jun 12th, 2020
159
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
Apache Log 53.31 KB | None | 0 0
raw download clone embed print report
  1.  
  2. ++ id -u
  3. + myuid=0
  4. ++ id -g
  5. + mygid=0
  6. + set +e
  7. ++ getent passwd 0
  8. + uidentry=root:x:0:0:root:/root:/bin/bash
  9. + set -e
  10. + '[' -z root:x:0:0:root:/root:/bin/bash ']'
  11. + SPARK_K8S_CMD=driver
  12. + case "$SPARK_K8S_CMD" in
  13. + shift 1
  14. + SPARK_CLASSPATH=':/opt/spark/jars/*'
  15. + env
  16. + grep SPARK_JAVA_OPT_
  17. + sort -t_ -k4 -n
  18. + sed 's/[^=]*=\(.*\)/\1/g'
  19. + readarray -t SPARK_EXECUTOR_JAVA_OPTS
  20. + '[' -n '' ']'
  21. + '[' -n '' ']'
  22. + PYSPARK_ARGS=
  23. + '[' -n '' ']'
  24. + R_ARGS=
  25. + '[' -n '' ']'
  26. + '[' '' == 2 ']'
  27. + '[' '' == 3 ']'
  28. + case "$SPARK_K8S_CMD" in
  29. + CMD=("$SPARK_HOME/bin/spark-submit" --conf "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS" --deploy-mode client "$@")
  30. + exec /usr/bin/tini -s -- /opt/spark/bin/spark-submit --conf spark.driver.bindAddress=10.28.138.26 --deploy-mode client
  31. --properties-file /opt/spark/conf/spark.properties --class org.company.group.beam.items.itemsPipeline spark-internal --runner=SparkRunner --brokerUrl=http://group-kafka-kafka-bootstrap:9091,http://group-kafka-kafka-bootstrap:9092,http://group-kafka-kafka-bootstrap:9093 --schemaRegistryUrl=http://group-schema-registry:8081
  32. 20/06/12 14:59:06 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applgroupble
  33. log4j:WARN No appenders could be found for logger (org.apache.beam.sdk.options.PipelineOptionsFactory).
  34. log4j:WARN Please initialize the log4j system properly.
  35. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
  36. Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
  37. 20/06/12 14:59:09 WARN Checkpoint: Checkpoint directory /tmp/itemspipeline-root-0612145909-7d5ea37e/spark-checkpoint does not exist
  38. 20/06/12 14:59:09 INFO SparkRunnerStreamingContextFactory: Creating a new Spark Streaming Context
  39. 20/06/12 14:59:09 INFO SparkRunnerStreamingContextFactory: Setting Spark streaming batchDuration to 500 msec
  40. 20/06/12 14:59:09 INFO SparkContextFactory: Creating a brand new Spark Context.
  41. 20/06/12 14:59:09 INFO SparkContext: Running Spark version 2.4.5
  42. 20/06/12 14:59:09 INFO SparkContext: Submitted applgrouption: itemsPipeline
  43. 20/06/12 14:59:09 INFO SecurityManager: Changing view acls to: root
  44. 20/06/12 14:59:09 INFO SecurityManager: Changing modify acls to: root
  45. 20/06/12 14:59:09 INFO SecurityManager: Changing view acls groups to:
  46. 20/06/12 14:59:09 INFO SecurityManager: Changing modify acls groups to:
  47. 20/06/12 14:59:09 INFO SecurityManager: SecurityManager: authentgrouption disabled; ui acls disabled; items  with view permissions: Set(root); groups with view permissions: Set(); items  with modify permissions: Set(root); groups with modify permissions: Set()
  48. 20/06/12 14:59:09 INFO Utils: Successfully started service 'sparkDriver' on port 7078.
  49. 20/06/12 14:59:09 INFO SparkEnv: Registering MapOutputTracker
  50. 20/06/12 14:59:09 INFO SparkEnv: Registering BlockManagerMaster
  51. 20/06/12 14:59:09 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
  52. 20/06/12 14:59:09 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
  53. 20/06/12 14:59:09 INFO DiskBlockManager: Created local directory at /var/data/spark-0bef3d66-1392-412a-8428-c2a1104471d9/blockmgr-1d4713a4-046d-40be-b858-9dbcf6f5d88d
  54. 20/06/12 14:59:09 INFO MemoryStore: MemoryStore started with capacity 413.9 MB
  55. 20/06/12 14:59:10 INFO SparkEnv: Registering OutputCommitCoordinator
  56. 20/06/12 14:59:10 INFO Utils: Successfully started service 'SparkUI' on port 4040.
  57. 20/06/12 14:59:10 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://group-beam-items-1591973941775-driver-svc.default.svc:4040
  58. 20/06/12 14:59:10 INFO SparkContext: Added JAR file:///opt/spark/items-standalone.jar at spark://group-beam-items-1591973941775-driver-svc.default.svc:7078/jars/items-standalone.jar with timestamp 1591973950319
  59. 20/06/12 14:59:11 INFO ExecutorPodsAllocator: Going to request 1 executors from Kubernetes.
  60. 20/06/12 14:59:11 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on
  61. port 7079.
  62. 20/06/12 14:59:11 INFO NettyBlockTransferService: Server created on group-beam-items-1591973941775-driver-svc.default.svc:7079
  63. 20/06/12 14:59:11 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplgrouptionPolicy for block replgrouption policy
  64. 20/06/12 14:59:11 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, group-beam-items-1591973941775-driver-svc.default.svc, 7079, None)
  65. 20/06/12 14:59:11 INFO BlockManagerMasterEndpoint: Registering block manager group-beam-items-1591973941775-driver-svc.default.svc:7079 with 413.9 MB RAM, BlockManagerId(driver, group-beam-items-1591973941775-driver-svc.default.svc, 7079, None)  
  66. 20/06/12 14:59:11 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, group-beam-items-1591973941775-driver-svc.default.svc, 7079, None)
  67. 20/06/12 14:59:11 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, group-beam-items-1591973941775-driver-svc.default.svc, 7079, None)
  68. 20/06/12 14:59:15 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.28.139.19:36602) with ID 1
  69. 20/06/12 14:59:15 INFO KubernetesClusterSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
  70. 20/06/12 14:59:15 WARN Checkpoint$CheckpointDir: The specified checkpoint dir /tmp/itemspipeline-root-0612145909-7d5ea37e does not match a reliable filesystem so in case of failures this job may not recover properly or even at all.
  71. 20/06/12 14:59:15 INFO Checkpoint$CheckpointDir: Checkpoint dir set to: /tmp/itemspipeline-root-0612145909-7d5ea37e
  72. 20/06/12 14:59:15 INFO BlockManagerMasterEndpoint: Registering block manager 10.28.139.19:44225 with 2.1 GB RAM, BlockManagerId(1, 10.28.139.19, 44225, None)
  73. 20/06/12 14:59:15 INFO MetricsAccumulator: No metrics checkpoint found.
  74. 20/06/12 14:59:15 INFO MetricsAccumulator: Instantiated metrics accumulator: MetricQueryResults()
  75. 20/06/12 14:59:15 WARN Checkpoint$CheckpointDir: The specified checkpoint dir /tmp/itemspipeline-root-0612145909-7d5ea37e does not match a reliable filesystem so in case of failures this job may not recover properly or even at all.
  76. 20/06/12 14:59:15 INFO Checkpoint$CheckpointDir: Checkpoint dir set to: /tmp/itemspipeline-root-0612145909-7d5ea37e      
  77. 20/06/12 14:59:15 INFO AggregatorsAccumulator: No accumulator checkpoint found.
  78. 20/06/12 14:59:15 INFO AggregatorsAccumulator: Instantiated aggregators accumulator:
  79. 20/06/12 14:59:15 INFO SparkRunner$Evaluator: Evaluating Read(KafkaUnboundedSource)
  80. 20/06/12 14:59:15 INFO PIDRateEstimator: Created PIDRateEstimator with proportional = 1.0, integral = 0.2, derivative = 0.0, min rate = 100.0
  81. 20/06/12 14:59:15 INFO SourceDStream: Read duration set to: PT0.200S
  82. 20/06/12 14:59:15 INFO SourceDStream: Max records per batch has not been limited by neither configuration nor the rate controller, and will remain unlimited for the current batch (9223372036854775807).
  83. 20/06/12 14:59:16 INFO ConsumerConfig: ConsumerConfig values:
  84.        allow.auto.create.topics = true
  85.        auto.commit.interval.ms = 5000
  86.        auto.offset.reset = earliest
  87.        bootstrap.servers = [http://group-kafka-kafka-bootstrap:9091, http://group-kafka-kafka-bootstrap:9092, http://group-kafka-kafka-bootstrap:9093]
  88.        check.crcs = true
  89.        client.dns.lookup = default
  90.        client.id =
  91.        client.rack =
  92.        connections.max.idle.ms = 540000
  93.        default.api.timeout.ms = 60000
  94.        enable.auto.commit = false
  95.        exclude.internal.topics = true
  96.        fetch.max.bytes = 52428800
  97.        fetch.max.wait.ms = 500
  98.        fetch.min.bytes = 1
  99.        group.id = pipeline-items
  100.        group.instance.id = null
  101.        heartbeat.interval.ms = 3000
  102.        interceptor.classes = []
  103.        internal.leave.group.on.close = true
  104.        isolation.level = read_uncommitted
  105.        key.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer
  106.        max.partition.fetch.bytes = 1048576
  107.        max.poll.interval.ms = 300000
  108.        max.poll.records = 500
  109.        metadata.max.age.ms = 300000
  110.        metric.reporters = []
  111.        metrics.num.samples = 2
  112.        metrics.recording.level = INFO
  113.        metrics.sample.window.ms = 30000
  114.        partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
  115.        receive.buffer.bytes = 524288
  116.        reconnect.backoff.max.ms = 1000
  117.        reconnect.backoff.ms = 50
  118.        request.timeout.ms = 30000
  119.        retry.backoff.ms = 100
  120.        sasl.client.callback.handler.class = null
  121.        sasl.jaas.config = null
  122.        sasl.kerberos.kinit.cmd = /usr/bin/kinit
  123.        sasl.kerberos.min.time.before.relogin = 60000
  124.        sasl.kerberos.service.name = null
  125.        sasl.kerberos.ticket.renew.jitter = 0.05
  126.        sasl.kerberos.ticket.renew.window.factor = 0.8
  127.        sasl.login.callback.handler.class = null
  128.        sasl.login.class = null
  129.        sasl.login.refresh.buffer.seconds = 300
  130.        sasl.login.refresh.min.period.seconds = 60
  131.        sasl.login.refresh.window.factor = 0.8
  132.        sasl.login.refresh.window.jitter = 0.05
  133.        sasl.mechanism = GSSAPI
  134.        security.protocol = PLAINTEXT
  135.        security.providers = null
  136.        send.buffer.bytes = 131072
  137.        session.timeout.ms = 10000
  138.        ssl.cipher.suites = null
  139.        ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
  140.        ssl.endpoint.identifgrouption.algorithm = https
  141.        ssl.key.password = null
  142.        ssl.keymanager.algorithm = SunX509
  143.        ssl.keystore.location = null
  144.        ssl.keystore.password = null
  145.        ssl.keystore.type = JKS
  146.        ssl.protocol = TLS
  147.        ssl.provider = null
  148.        ssl.secure.random.implementation = null
  149.        ssl.trustmanager.algorithm = PKIX
  150.        ssl.truststore.location = null
  151.        ssl.truststore.password = null
  152.        ssl.truststore.type = JKS
  153.        value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer
  154.  
  155. 20/06/12 14:59:16 WARN ConsumerConfig: The configuration 'schema.registry.url' was supplied but isn't a known config.
  156. 20/06/12 14:59:16 INFO AppInfoParser: Kafka version: 2.4.1
  157. 20/06/12 14:59:16 INFO AppInfoParser: Kafka commitId: c57222ae8cd7866b
  158. 20/06/12 14:59:16 INFO AppInfoParser: Kafka startTimeMs: 1591973956168
  159. 20/06/12 14:59:17 WARN NetworkClient: [Consumer clientId=consumer-pipeline-items-1, groupId=pipeline-items] Bootstrap broker group-kafka-kafka-bootstrap:9091 (id: -1 rack: null) disconnected
  160. 20/06/12 14:59:17 INFO Metadata: [Consumer clientId=consumer-pipeline-items-1, groupId=pipeline-items] Cluster ID: 3x0FYahAQNaWVFgjDPdduQ
  161. 20/06/12 14:59:17 INFO KafkaUnboundedSource: Partitions assigned to split 0 (total 5): source.mongo.elastgroupco.tenant_items-0,source.mongo.elastgroupco.tenant_items-2,source.mongo.elastgroupco.tenant_items-4,source.mongo.elastgroupco.tenant_items-6,source.mongo.elastgroupco.tenant_items-8
  162. 20/06/12 14:59:17 INFO KafkaUnboundedSource: Partitions assigned to split 1 (total 5): source.mongo.elastgroupco.tenant_items-1,source.mongo.elastgroupco.tenant_items-3,source.mongo.elastgroupco.tenant_items-5,source.mongo.elastgroupco.tenant_items-7,source.mongo.elastgroupco.tenant_items-9
  163. 20/06/12 14:59:17 INFO SparkRunner$Evaluator: Evaluating org.apache.beam.sdk.io.kafka.KafkaIO$TypedWithoutMetadata$1@1aa6e3c0
  164. 20/06/12 14:59:17 INFO SparkRunner$Evaluator: Evaluating org.company.group.beam.items.transforms.items$ExtractUserAndPortalUser@6ba7383d
  165. 20/06/12 14:59:17 INFO SparkRunner$Evaluator: Evaluating org.apache.beam.sdk.transforms.MapElements$1@1fde0371
  166. 20/06/12 14:59:17 INFO SparkRunner$Evaluator: Evaluating org.apache.beam.sdk.io.kafka.KafkaWriter@2042ccce
  167. 20/06/12 14:59:17 INFO SparkRunner$Evaluator: Evaluating org.apache.beam.sdk.transforms.MapElements$1@2af46afd
  168. 20/06/12 14:59:17 INFO SparkRunner$Evaluator: Evaluating org.apache.beam.sdk.io.kafka.KafkaWriter@6ff6efdc
  169. 20/06/12 14:59:17 WARN SparkContext: Spark is not running in local mode, therefore the checkpoint directory must not be on the local filesystem. Directory 'file:/tmp/itemspipeline-root-0612145909-7d5ea37e/spark-checkpoint''schema.registry.url' was supplied but isn't a known config.    
  170. 20/06/12 14:59:18 INFO AppInfoParser: Kafka version: 2.4.1
  171. 20/06/12 14:59:18 INFO AppInfoParser: Kafka commitId: c57222ae8cd7866b
  172. 20/06/12 14:59:18 INFO AppInfoParser: Kafka startTimeMs: 1591973958163
  173. 20/06/12 14:59:18 INFO Metadata: [Consumer clientId=consumer-pipeline-items-2, groupId=pipeline-items] Cluster ID: 3x0FYahAQNaWVFgjDPdduQ
  174. 20/06/12 14:59:18 INFO KafkaUnboundedSource: Partitions assigned to split 0 (total 5): source.mongo.elastgroupco.tenant_items-0,source.mongo.elastgroupco.tenant_items-2,source.mongo.elastgroupco.tenant_items-4,source.mongo.elastgroupco.tenant_items-6,source.mongo.elastgroupco.tenant_items-8
  175. 20/06/12 14:59:18 INFO KafkaUnboundedSource: Partitions assigned to split 1 (total 5): source.mongo.elastgroupco.tenant_items-1,source.mongo.elastgroupco.tenant_items-3,source.mongo.elastgroupco.tenant_items-5,source.mongo.elastgroupco.tenant_items-7,source.mongo.elastgroupco.tenant_items-9
  176. 20/06/12 14:59:18 INFO DAGScheduler: Got job 0 (DStream at SparkUnboundedSource.java:171) with 2 output partitions      
  177. 20/06/12 14:59:18 INFO DAGScheduler: Final stage: ResultStage 2 (DStream at SparkUnboundedSource.java:171)
  178. 20/06/12 14:59:18 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 0, ShuffleMapStage 1)
  179. 20/06/12 14:59:18 INFO DAGScheduler: Missing parents: List()
  180. 20/06/12 14:59:18 INFO DAGScheduler: Submitting ResultStage 2 (MapPartitionsRDD[9] at map at SparkUnboundedSource.java:113), which has no missing parents
  181. 20/06/12 14:59:18 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 14.9 KB, free 413.9 MB)
  182. 20/06/12 14:59:18 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 7.3 KB, free 413.9 MB)
  183. 20/06/12 14:59:18 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on group-beam-items-1591973941775-driver-svc.default.svc:7079 (size: 7.3 KB, free: 413.9 MB)
  184. 20/06/12 14:59:18 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1163
  185. 20/06/12 14:59:18 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 2 (MapPartitionsRDD[9] at map at SparkUnboundedSource.java:113) (first 15 tasks are for partitions Vector(0, 1))
  186. 20/06/12 14:59:18 INFO TaskSchedulerImpl: Adding task set 2.0 with 2 tasks
  187. 20/06/12 14:59:18 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 0, 10.28.139.19, executor 1, partition 0, PROCESS_LOCAL, 15751 bytes)
  188. 20/06/12 14:59:21 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.28.139.19:44225 (size: 7.3 KB, free: 2.1 GB)
  189. 20/06/12 14:59:22 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 10.28.139.19:36602
  190. 20/06/12 14:59:22 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.28.139.19:36602
  191. 20/06/12 14:59:25 INFO TaskSetManager: Starting task 1.0 in stage 2.0 (TID 1, 10.28.139.19, executor 1, partition 1, PROCESS_LOCAL, 15751 bytes)
  192. 20/06/12 14:59:25 WARN TaskSetManager: Lost task 0.0 in stage 2.0 (TID 0, 10.28.139.19, executor 1): org.apache.kafka.common.errors.SerializationException: Unknown magic byte!
  193.  
  194. 20/06/12 14:59:26 INFO TaskSetManager: Starting task 0.1 in stage 2.0 (TID 2, 10.28.139.19, executor 1, partition 0, PROCESS_LOCAL, 15751 bytes)
  195. 20/06/12 14:59:26 INFO TaskSetManager: Lost task 1.0 in stage 2.0 (TID 1) on 10.28.139.19, executor 1: org.apache.kafka.common.errors.SerializationException (Unknown magic byte!) [duplgroupte 1]
  196. 20/06/12 14:59:26 INFO TaskSetManager: Starting task 1.1 in stage 2.0 (TID 3, 10.28.139.19, executor 1, partition 1, PROCESS_LOCAL, 15751 bytes)
  197. 20/06/12 14:59:26 INFO TaskSetManager: Lost task 0.1 in stage 2.0 (TID 2) on 10.28.139.19, executor 1: org.apache.kafka.common.errors.SerializationException (Unknown magic byte!) [duplgroupte 2]
  198. 20/06/12 14:59:26 INFO TaskSetManager: Starting task 0.2 in stage 2.0 (TID 4, 10.28.139.19, executor 1, partition 0, PROCESS_LOCAL, 15751 bytes)
  199. 20/06/12 14:59:26 INFO TaskSetManager: Lost task 1.1 in stage 2.0 (TID 3) on 10.28.139.19, executor 1: org.apache.kafka.common.errors.SerializationException (Unknown magic byte!) [duplgroupte 3]
  200. 20/06/12 14:59:26 INFO TaskSetManager: Starting task 1.2 in stage 2.0 (TID 5, 10.28.139.19, executor 1, partition 1, PROCESS_LOCAL, 15751 bytes)
  201. 20/06/12 14:59:26 INFO TaskSetManager: Lost task 0.2 in stage 2.0 (TID 4) on 10.28.139.19, executor 1: org.apache.kafka.common.errors.SerializationException (Unknown magic byte!) [duplgroupte 4]
  202. 20/06/12 14:59:26 INFO TaskSetManager: Starting task 0.3 in stage 2.0 (TID 6, 10.28.139.19, executor 1, partition 0, PROCESS_LOCAL, 15751 bytes)
  203. 20/06/12 14:59:26 INFO TaskSetManager: Lost task 1.2 in stage 2.0 (TID 5) on 10.28.139.19, executor 1: org.apache.kafka.common.errors.SerializationException (Unknown magic byte!) [duplgroupte 5]
  204. 20/06/12 14:59:27 INFO BlockManagerInfo: Added rdd_7_0 in memory on 10.28.139.19:44225 (size: 24.8 KB, free: 2.1 GB)
  205. 20/06/12 14:59:27 INFO TaskSetManager: Starting task 1.3 in stage 2.0 (TID 7, 10.28.139.19, executor 1, partition 1, PROCESS_LOCAL, 15751 bytes)
  206. 20/06/12 14:59:27 INFO TaskSetManager: Finished task 0.3 in stage 2.0 (TID 6) in 382 ms on 10.28.139.19 (executor 1) (1/2)
  207. 20/06/12 14:59:27 INFO BlockManagerInfo: Added rdd_7_1 in memory on 10.28.139.19:44225 (size: 24.8 KB, free: 2.1 GB)
  208. 20/06/12 14:59:27 INFO TaskSetManager: Finished task 1.3 in stage 2.0 (TID 7) in 250 ms on 10.28.139.19 (executor 1) (2/2)
  209. 20/06/12 14:59:27 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool
  210. 20/06/12 14:59:27 INFO DAGScheduler: ResultStage 2 (DStream at SparkUnboundedSource.java:171) finished in 9.315 s        
  211. 20/06/12 14:59:27 INFO DAGScheduler: Job 0 finished: DStream at SparkUnboundedSource.java:171, took 9.460734 s
  212. 20/06/12 14:59:27 INFO JobScheduler: Added jobs for time 1591973958000 ms
  213. 20/06/12 14:59:27 INFO CheckpointWriter: CheckpointWriter executor terminated? true, waited for 0 ms.
  214. 20/06/12 14:59:27 INFO JobGenerator: Stopped JobGenerator
  215. 20/06/12 14:59:27 INFO JobScheduler: Starting job streaming job 1591973958000 ms.0 from job set of time 1591973958000 ms
  216. 20/06/12 14:59:27 INFO SparkContext: Starting job: foreach at UnboundedDataset.java:79
  217. 20/06/12 14:59:27 INFO DAGScheduler: Got job 1 (foreach at UnboundedDataset.java:79) with 2 output partitions
  218. 20/06/12 14:59:27 INFO DAGScheduler: Final stage: ResultStage 5 (foreach at UnboundedDataset.java:79)
  219. 20/06/12 14:59:27 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 3, ShuffleMapStage 4)
  220. 20/06/12 14:59:27 INFO DAGScheduler: Missing parents: List()
  221. 20/06/12 14:59:27 INFO DAGScheduler: Submitting ResultStage 5 (MapPartitionsRDD[25] at map at TranslationUtils.java:134), which has no missing parents
  222. 20/06/12 14:59:27 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 32.5 KB, free 413.9 MB)
  223. 20/06/12 14:59:27 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 13.3 KB, free 413.9 MB)
  224. 20/06/12 14:59:27 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on group-beam-items-1591973941775-driver-svc.default.svc:7079 (size: 13.3 KB, free: 413.9 MB)
  225. 20/06/12 14:59:27 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1163
  226. 20/06/12 14:59:27 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 5 (MapPartitionsRDD[25] at map at TranslationUtils.java:134) (first 15 tasks are for partitions Vector(0, 1))
  227. 20/06/12 14:59:27 INFO TaskSchedulerImpl: Adding task set 5.0 with 2 tasks
  228. 20/06/12 14:59:27 INFO TaskSetManager: Starting task 0.0 in stage 5.0 (TID 8, 10.28.139.19, executor 1, partition 0, PROCESS_LOCAL, 15751 bytes)
  229. 20/06/12 14:59:27 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 10.28.139.19:44225 (size: 13.3 KB, free: 2.1 GB)
  230. 20/06/12 14:59:28 INFO BlockManagerInfo: Added rdd_16_0 in memory on 10.28.139.19:44225 (size: 4.0 B, free: 2.1 GB)
  231. 20/06/12 14:59:28 INFO TaskSetManager: Starting task 1.0 in stage 5.0 (TID 9, 10.28.139.19, executor 1, partition 1, PROCESS_LOCAL, 15751 bytes)
  232. 20/06/12 14:59:28 INFO TaskSetManager: Finished task 0.0 in stage 5.0 (TID 8) in 782 ms on 10.28.139.19 (executor 1) (1/2)
  233. 20/06/12 14:59:28 INFO BlockManagerInfo: Added rdd_16_1 in memory on 10.28.139.19:44225 (size: 4.0 B, free: 2.1 GB)
  234. 20/06/12 14:59:28 INFO TaskSetManager: Finished task 1.0 in stage 5.0 (TID 9) in 32 ms on 10.28.139.19 (executor 1) (2/2)20/06/12 14:59:28 INFO TaskSchedulerImpl: Removed TaskSet 5.0, whose tasks have all completed, from pool
  235. 20/06/12 14:59:28 INFO DAGScheduler: ResultStage 5 (foreach at UnboundedDataset.java:79) finished in 0.833 s
  236. 20/06/12 14:59:28 INFO DAGScheduler: Job 1 finished: foreach at UnboundedDataset.java:79, took 0.853407 s
  237. 20/06/12 14:59:28 INFO JobScheduler: Finished job streaming job 1591973958000 ms.0 from job set of time 1591973958000 ms
  238. 20/06/12 14:59:28 INFO JobScheduler: Starting job streaming job 1591973958000 ms.1 from job set of time 1591973958000 ms
  239. 20/06/12 14:59:28 INFO SparkContext: Starting job: foreach at UnboundedDataset.java:79
  240. 20/06/12 14:59:28 INFO DAGScheduler: Got job 2 (foreach at UnboundedDataset.java:79) with 2 output partitions
  241. 20/06/12 14:59:28 INFO DAGScheduler: Final stage: ResultStage 8 (foreach at UnboundedDataset.java:79)
  242. 20/06/12 14:59:28 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 6, ShuffleMapStage 7)
  243. 20/06/12 14:59:28 INFO DAGScheduler: Missing parents: List()
  244. 20/06/12 14:59:28 INFO DAGScheduler: Submitting ResultStage 8 (MapPartitionsRDD[33] at map at TranslationUtils.java:134), which has no missing parents
  245. 20/06/12 14:59:28 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 32.6 KB, free 413.8 MB)
  246. 20/06/12 14:59:28 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 13.3 KB, free 413.8 MB)
  247. 20/06/12 14:59:28 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on group-beam-items-1591973941775-driver-svc.default.svc:7079 (size: 13.3 KB, free: 413.9 MB)
  248. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 12
  249. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 2
  250. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 48
  251. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 36
  252. 20/06/12 14:59:28 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1163
  253. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 14
  254. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 23
  255. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 43
  256. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 24
  257. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 29
  258. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 39
  259. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 5
  260. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 47
  261. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 21
  262. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 37
  263. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 9
  264. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 50
  265. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 7
  266. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 4
  267. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 3
  268. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 13
  269. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 18
  270. 20/06/12 14:59:28 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 8 (MapPartitionsRDD[33] at map at TranslationUtils.java:134) (first 15 tasks are for partitions Vector(0, 1))
  271. 20/06/12 14:59:28 INFO TaskSchedulerImpl: Adding task set 8.0 with 2 tasks
  272. 20/06/12 14:59:28 INFO TaskSetManager: Starting task 0.0 in stage 8.0 (TID 10, 10.28.139.19, executor 1, partition 0, PROCESS_LOCAL, 15751 bytes)
  273. 20/06/12 14:59:28 INFO BlockManagerInfo: Removed broadcast_0_piece0 on group-beam-items-1591973941775-driver-svc.default.svc:7079 in memory (size: 7.3 KB, free: 413.9 MB)
  274. 20/06/12 14:59:28 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 10.28.139.19:44225 (size: 13.3 KB, free: 2.1 GB)
  275. 20/06/12 14:59:28 INFO BlockManagerInfo: Removed broadcast_0_piece0 on 10.28.139.19:44225 in memory (size: 7.3 KB, free:
  276. 2.1 GB)
  277. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 11
  278. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 30
  279. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 8
  280. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 20
  281. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 34
  282. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 6
  283. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 32
  284. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 17
  285. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 51
  286. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 40
  287. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 27
  288. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 45
  289. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 49
  290. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 31
  291. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 42
  292. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 15
  293. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 25
  294. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 28
  295. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 10
  296. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 26
  297. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 38
  298. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 46
  299. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 35
  300. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 41
  301. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 19
  302. 20/06/12 14:59:28 INFO BlockManagerInfo: Removed broadcast_1_piece0 on group-beam-items-1591973941775-driver-svc.default.svc:7079 in memory (size: 13.3 KB, free: 413.9 MB)
  303. 20/06/12 14:59:28 INFO TaskSetManager: Starting task 1.0 in stage 8.0 (TID 11, 10.28.139.19, executor 1, partition 1, PROCESS_LOCAL, 15751 bytes)
  304. 20/06/12 14:59:28 INFO TaskSetManager: Finished task 0.0 in stage 8.0 (TID 10) in 55 ms on 10.28.139.19 (executor 1) (1/2)
  305. 20/06/12 14:59:28 INFO BlockManagerInfo: Removed broadcast_1_piece0 on 10.28.139.19:44225 in memory (size: 13.3 KB, free: 2.1 GB)
  306. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 22
  307. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 16
  308. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 33
  309. 20/06/12 14:59:28 INFO ContextCleaner: Cleaned accumulator 44
  310. 20/06/12 14:59:28 INFO TaskSetManager: Finished task 1.0 in stage 8.0 (TID 11) in 31 ms on 10.28.139.19 (executor 1) (2/2)
  311. 20/06/12 14:59:28 INFO TaskSchedulerImpl: Removed TaskSet 8.0, whose tasks have all completed, from pool
  312. 20/06/12 14:59:28 INFO DAGScheduler: ResultStage 8 (foreach at UnboundedDataset.java:79) finished in 0.105 s
  313. 20/06/12 14:59:28 INFO DAGScheduler: Job 2 finished: foreach at UnboundedDataset.java:79, took 0.115234 s
  314. 20/06/12 14:59:28 INFO JobScheduler: Finished job streaming job 1591973958000 ms.1 from job set of time 1591973958000 ms
  315. 20/06/12 14:59:28 INFO JobScheduler: Total delay: 10.725 s for time 1591973958000 ms (execution: 0.997 s)
  316. 20/06/12 14:59:28 INFO JobScheduler: Stopped JobScheduler
  317. 20/06/12 14:59:28 INFO StreamingContext: StreamingContext stopped successfully
  318. 20/06/12 14:59:28 INFO SparkContext: Invoking stop() from shutdown hook
  319. 20/06/12 14:59:28 INFO SparkUI: Stopped Spark web UI at http://group-beam-items-1591973941775-driver-svc.default.svc:4040
  320. 20/06/12 14:59:28 INFO KubernetesClusterSchedulerBackend: Shutting down all executors
  321. 20/06/12 14:59:28 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asking each executor to shut down
  322. 20/06/12 14:59:28 WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed (this is expected if the applgrouption is shutting down.)
  323. 20/06/12 14:59:28 INFO MemoryStore: Block broadcast_0_WATERMARKS stored as values in memory (estimated size 72.0 B, free
  324. 413.9 MB)
  325. 20/06/12 14:59:28 INFO BlockManagerInfo: Added broadcast_0_WATERMARKS in memory on group-beam-items-1591973941775-driver-svc.default.svc:7079 (size: 72.0 B, free: 413.9 MB)
  326. 20/06/12 14:59:28 INFO BlockManagerInfo: Removed broadcast_0_WATERMARKS on group-beam-items-1591973941775-driver-svc.default.svc:7079 in memory (size: 72.0 B, free: 413.9 MB)
  327. 20/06/12 14:59:28 INFO MemoryStore: Block broadcast_0_WATERMARKS stored as values in memory (estimated size 296.0 B, free 413.9 MB)
  328. 20/06/12 14:59:28 INFO BlockManagerInfo: Added broadcast_0_WATERMARKS in memory on group-beam-items-1591973941775-driver-svc.default.svc:7079 (size: 296.0 B, free: 413.9 MB)
  329. 20/06/12 14:59:28 INFO GlobalWatermarkHolder: Put new watermark block: {0=SparkWatermarks{lowWatermark=-290308-12-21T19:59:05.225Z, highWatermark=2020-06-12T14:59:27.508Z, synchronizedProcessingTime=2020-06-12T14:59:18.000Z}}
  330. 20/06/12 14:59:28 INFO GlobalWatermarkHolder$WatermarkAdvancingStreamingListener: Batch with timestamp: 1591973958000 has completed, watermarks have been updated.
  331. 20/06/12 14:59:28 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
  332. 20/06/12 14:59:28 INFO MemoryStore: MemoryStore cleared
  333. 20/06/12 14:59:28 INFO BlockManager: BlockManager stopped
  334. 20/06/12 14:59:28 INFO BlockManagerMaster: BlockManagerMaster stopped
  335. 20/06/12 14:59:28 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
  336. 20/06/12 14:59:28 INFO SparkContext: Successfully stopped SparkContext
  337. 20/06/12 14:59:28 INFO ShutdownHookManager: Shutdown hook called
  338. 20/06/12 14:59:28 INFO ShutdownHookManager: Deleting directory /var/data/spark-0bef3d66-1392-412a-8428-c2a1104471d9/spark-a0a360ab-a25e-400e-90fa-bd3ae22ea0e6
  339. 20/06/12 14:59:28 INFO ShutdownHookManager: Deleting directory /tmp/spark-9cfd84c3-0155-47e5-af77-b6e071c87b47
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!