API tools faq
paste
Advertisement
Guest User

big scope error

a guest
Apr 6th, 2018
311
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 35.85 KB | None | 0 0
raw download clone embed print report
  1. An error was encountered:
  2. An error occurred while calling o96.toJSON.
  3. : org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 4.0 failed 4 times, most recent failure: Lost task 0.3 in stage 4.0 (TID 7, wn2-MDMstr.zxmmgisclg5udfemnv0v3qva3e.ax.internal.cloudapp.net, executor 1): java.lang.IllegalStateException: com.microsoft.azure.documentdb.DocumentClientException: Message: {"Errors":["Request rate is large"]}
  4. ActivityId: 859bd9a2-eaef-4c32-acb8-43ce021c28c5, Request URI: /apps/e01795ed-8f69-47b3-ac45-e1a66f2ec14f/services/568a3c5f-3e58-4f21-b744-8f784b91fc61/partitions/197c818e-4a33-4fa0-8031-6c665e25a592/replicas/131662668080760616p, RequestStats:
  5. ResponseTime: 2018-04-06T16:04:01.3903033Z, StoreReadResult: StorePhysicalAddress: rntbd://10.0.0.104:16700/apps/e01795ed-8f69-47b3-ac45-e1a66f2ec14f/services/568a3c5f-3e58-4f21-b744-8f784b91fc61/partitions/197c818e-4a33-4fa0-8031-6c665e25a592/replicas/131662668080760616p, LSN: 465, GlobalCommittedLsn: 465, PartitionKeyRangeId: , IsValid: True, StatusCode: 0, IsGone: False, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0.38, ItemLSN: -1, ResourceType: Document, OperationType: ReadFeed
  6. , SDK: Microsoft.Azure.Documents.Common/1.21.0.0, StatusCode: TooManyRequests
  7. at com.microsoft.azure.documentdb.internal.query.DefaultQueryExecutionContext.next(DefaultQueryExecutionContext.java:86)
  8. at com.microsoft.azure.documentdb.internal.query.DefaultQueryExecutionContext.next(DefaultQueryExecutionContext.java:33)
  9. at com.microsoft.azure.documentdb.internal.query.ProxyQueryExecutionContext.<init>(ProxyQueryExecutionContext.java:68)
  10. at com.microsoft.azure.documentdb.internal.query.QueryExecutionContextFactory.createQueryExecutionContext(QueryExecutionContextFactory.java:23)
  11. at com.microsoft.azure.documentdb.QueryIterable.createQueryExecutionContext(QueryIterable.java:70)
  12. at com.microsoft.azure.documentdb.QueryIterable.reset(QueryIterable.java:115)
  13. at com.microsoft.azure.documentdb.QueryIterable.<init>(QueryIterable.java:57)
  14. at com.microsoft.azure.documentdb.QueryIterable.<init>(QueryIterable.java:31)
  15. at com.microsoft.azure.documentdb.DocumentClient.readDocuments(DocumentClient.java:1002)
  16. at com.microsoft.azure.cosmosdb.spark.CosmosDBConnection.readDocuments(CosmosDBConnection.scala:205)
  17. at com.microsoft.azure.cosmosdb.spark.rdd.CosmosDBRDDIterator.queryDocuments$1(CosmosDBRDDIterator.scala:192)
  18. at com.microsoft.azure.cosmosdb.spark.rdd.CosmosDBRDDIterator.reader$lzycompute(CosmosDBRDDIterator.scala:321)
  19. at com.microsoft.azure.cosmosdb.spark.rdd.CosmosDBRDDIterator.reader(CosmosDBRDDIterator.scala:137)
  20. at com.microsoft.azure.cosmosdb.spark.rdd.CosmosDBRDDIterator.hasNext(CosmosDBRDDIterator.scala:334)
  21. at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
  22. at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
  23. at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
  24. at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
  25. at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:395)
  26. at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
  27. at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
  28. at org.apache.spark.util.random.SamplingUtils$.reservoirSampleAndCount(SamplingUtils.scala:41)
  29. at org.apache.spark.RangePartitioner$$anonfun$9.apply(Partitioner.scala:263)
  30. at org.apache.spark.RangePartitioner$$anonfun$9.apply(Partitioner.scala:261)
  31. at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:844)
  32. at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:844)
  33. at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
  34. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  35. at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  36. at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
  37. at org.apache.spark.scheduler.Task.run(Task.scala:108)
  38. at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338)
  39. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  40. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  41. at java.lang.Thread.run(Thread.java:748)
  42. Caused by: com.microsoft.azure.documentdb.DocumentClientException: Message: {"Errors":["Request rate is large"]}
  43. ActivityId: 859bd9a2-eaef-4c32-acb8-43ce021c28c5, Request URI: /apps/e01795ed-8f69-47b3-ac45-e1a66f2ec14f/services/568a3c5f-3e58-4f21-b744-8f784b91fc61/partitions/197c818e-4a33-4fa0-8031-6c665e25a592/replicas/131662668080760616p, RequestStats:
  44. ResponseTime: 2018-04-06T16:04:01.3903033Z, StoreReadResult: StorePhysicalAddress: rntbd://10.0.0.104:16700/apps/e01795ed-8f69-47b3-ac45-e1a66f2ec14f/services/568a3c5f-3e58-4f21-b744-8f784b91fc61/partitions/197c818e-4a33-4fa0-8031-6c665e25a592/replicas/131662668080760616p, LSN: 465, GlobalCommittedLsn: 465, PartitionKeyRangeId: , IsValid: True, StatusCode: 0, IsGone: False, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0.38, ItemLSN: -1, ResourceType: Document, OperationType: ReadFeed
  45. , SDK: Microsoft.Azure.Documents.Common/1.21.0.0, StatusCode: TooManyRequests
  46. at com.microsoft.azure.documentdb.internal.ErrorUtils.maybeThrowException(ErrorUtils.java:69)
  47. at com.microsoft.azure.documentdb.internal.GatewayProxy.performGetRequest(GatewayProxy.java:245)
  48. at com.microsoft.azure.documentdb.internal.GatewayProxy.doReadFeed(GatewayProxy.java:111)
  49. at com.microsoft.azure.documentdb.internal.GatewayProxy.processMessage(GatewayProxy.java:346)
  50. at com.microsoft.azure.documentdb.DocumentClient$9.apply(DocumentClient.java:3011)
  51. at com.microsoft.azure.documentdb.internal.RetryUtility.executeDocumentClientRequest(RetryUtility.java:58)
  52. at com.microsoft.azure.documentdb.DocumentClient.doReadFeed(DocumentClient.java:3021)
  53. at com.microsoft.azure.documentdb.DocumentQueryClientInternal.doReadFeed(DocumentQueryClientInternal.java:36)
  54. at com.microsoft.azure.documentdb.internal.query.AbstractQueryExecutionContext.executeRequest(AbstractQueryExecutionContext.java:215)
  55. at com.microsoft.azure.documentdb.internal.query.DefaultQueryExecutionContext.executeOnce(DefaultQueryExecutionContext.java:137)
  56. at com.microsoft.azure.documentdb.internal.query.DefaultQueryExecutionContext.fillBuffer(DefaultQueryExecutionContext.java:101)
  57. at com.microsoft.azure.documentdb.internal.query.DefaultQueryExecutionContext.next(DefaultQueryExecutionContext.java:84)
  58. ... 34 more
  59.  
  60. Driver stacktrace:
  61. at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1517)
  62. at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1505)
  63. at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1504)
  64. at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
  65. at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
  66. at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1504)
  67. at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:814)
  68. at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:814)
  69. at scala.Option.foreach(Option.scala:257)
  70. at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:814)
  71. at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1732)
  72. at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1687)
  73. at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1676)
  74. at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
  75. at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:630)
  76. at org.apache.spark.SparkContext.runJob(SparkContext.scala:2029)
  77. at org.apache.spark.SparkContext.runJob(SparkContext.scala:2050)
  78. at org.apache.spark.SparkContext.runJob(SparkContext.scala:2069)
  79. at org.apache.spark.SparkContext.runJob(SparkContext.scala:2094)
  80. at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:936)
  81. at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  82. at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
  83. at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
  84. at org.apache.spark.rdd.RDD.collect(RDD.scala:935)
  85. at org.apache.spark.RangePartitioner$.sketch(Partitioner.scala:266)
  86. at org.apache.spark.RangePartitioner.<init>(Partitioner.scala:128)
  87. at org.apache.spark.sql.execution.exchange.ShuffleExchange$.prepareShuffleDependency(ShuffleExchange.scala:221)
  88. at org.apache.spark.sql.execution.exchange.ShuffleExchange.prepareShuffleDependency(ShuffleExchange.scala:87)
  89. at org.apache.spark.sql.execution.exchange.ShuffleExchange$$anonfun$doExecute$1.apply(ShuffleExchange.scala:124)
  90. at org.apache.spark.sql.execution.exchange.ShuffleExchange$$anonfun$doExecute$1.apply(ShuffleExchange.scala:115)
  91. at org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:52)
  92. at org.apache.spark.sql.execution.exchange.ShuffleExchange.doExecute(ShuffleExchange.scala:115)
  93. at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
  94. at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
  95. at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:138)
  96. at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  97. at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:135)
  98. at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:116)
  99. at org.apache.spark.sql.execution.InputAdapter.inputRDDs(WholeStageCodegenExec.scala:252)
  100. at org.apache.spark.sql.execution.SortExec.inputRDDs(SortExec.scala:121)
  101. at org.apache.spark.sql.execution.WholeStageCodegenExec.doExecute(WholeStageCodegenExec.scala:386)
  102. at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
  103. at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
  104. at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:138)
  105. at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  106. at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:135)
  107. at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:116)
  108. at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:92)
  109. at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:92)
  110. at org.apache.spark.sql.Dataset.toJSON(Dataset.scala:2743)
  111. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  112. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  113. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  114. at java.lang.reflect.Method.invoke(Method.java:498)
  115. at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
  116. at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
  117. at py4j.Gateway.invoke(Gateway.java:280)
  118. at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
  119. at py4j.commands.CallCommand.execute(CallCommand.java:79)
  120. at py4j.GatewayConnection.run(GatewayConnection.java:214)
  121. at java.lang.Thread.run(Thread.java:748)
  122. Caused by: java.lang.IllegalStateException: com.microsoft.azure.documentdb.DocumentClientException: Message: {"Errors":["Request rate is large"]}
  123. ActivityId: 859bd9a2-eaef-4c32-acb8-43ce021c28c5, Request URI: /apps/e01795ed-8f69-47b3-ac45-e1a66f2ec14f/services/568a3c5f-3e58-4f21-b744-8f784b91fc61/partitions/197c818e-4a33-4fa0-8031-6c665e25a592/replicas/131662668080760616p, RequestStats:
  124. ResponseTime: 2018-04-06T16:04:01.3903033Z, StoreReadResult: StorePhysicalAddress: rntbd://10.0.0.104:16700/apps/e01795ed-8f69-47b3-ac45-e1a66f2ec14f/services/568a3c5f-3e58-4f21-b744-8f784b91fc61/partitions/197c818e-4a33-4fa0-8031-6c665e25a592/replicas/131662668080760616p, LSN: 465, GlobalCommittedLsn: 465, PartitionKeyRangeId: , IsValid: True, StatusCode: 0, IsGone: False, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0.38, ItemLSN: -1, ResourceType: Document, OperationType: ReadFeed
  125. , SDK: Microsoft.Azure.Documents.Common/1.21.0.0, StatusCode: TooManyRequests
  126. at com.microsoft.azure.documentdb.internal.query.DefaultQueryExecutionContext.next(DefaultQueryExecutionContext.java:86)
  127. at com.microsoft.azure.documentdb.internal.query.DefaultQueryExecutionContext.next(DefaultQueryExecutionContext.java:33)
  128. at com.microsoft.azure.documentdb.internal.query.ProxyQueryExecutionContext.<init>(ProxyQueryExecutionContext.java:68)
  129. at com.microsoft.azure.documentdb.internal.query.QueryExecutionContextFactory.createQueryExecutionContext(QueryExecutionContextFactory.java:23)
  130. at com.microsoft.azure.documentdb.QueryIterable.createQueryExecutionContext(QueryIterable.java:70)
  131. at com.microsoft.azure.documentdb.QueryIterable.reset(QueryIterable.java:115)
  132. at com.microsoft.azure.documentdb.QueryIterable.<init>(QueryIterable.java:57)
  133. at com.microsoft.azure.documentdb.QueryIterable.<init>(QueryIterable.java:31)
  134. at com.microsoft.azure.documentdb.DocumentClient.readDocuments(DocumentClient.java:1002)
  135. at com.microsoft.azure.cosmosdb.spark.CosmosDBConnection.readDocuments(CosmosDBConnection.scala:205)
  136. at com.microsoft.azure.cosmosdb.spark.rdd.CosmosDBRDDIterator.queryDocuments$1(CosmosDBRDDIterator.scala:192)
  137. at com.microsoft.azure.cosmosdb.spark.rdd.CosmosDBRDDIterator.reader$lzycompute(CosmosDBRDDIterator.scala:321)
  138. at com.microsoft.azure.cosmosdb.spark.rdd.CosmosDBRDDIterator.reader(CosmosDBRDDIterator.scala:137)
  139. at com.microsoft.azure.cosmosdb.spark.rdd.CosmosDBRDDIterator.hasNext(CosmosDBRDDIterator.scala:334)
  140. at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
  141. at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
  142. at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
  143. at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
  144. at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:395)
  145. at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
  146. at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
  147. at org.apache.spark.util.random.SamplingUtils$.reservoirSampleAndCount(SamplingUtils.scala:41)
  148. at org.apache.spark.RangePartitioner$$anonfun$9.apply(Partitioner.scala:263)
  149. at org.apache.spark.RangePartitioner$$anonfun$9.apply(Partitioner.scala:261)
  150. at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:844)
  151. at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:844)
  152. at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
  153. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  154. at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  155. at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
  156. at org.apache.spark.scheduler.Task.run(Task.scala:108)
  157. at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338)
  158. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  159. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  160. ... 1 more
  161. Caused by: com.microsoft.azure.documentdb.DocumentClientException: Message: {"Errors":["Request rate is large"]}
  162. ActivityId: 859bd9a2-eaef-4c32-acb8-43ce021c28c5, Request URI: /apps/e01795ed-8f69-47b3-ac45-e1a66f2ec14f/services/568a3c5f-3e58-4f21-b744-8f784b91fc61/partitions/197c818e-4a33-4fa0-8031-6c665e25a592/replicas/131662668080760616p, RequestStats:
  163. ResponseTime: 2018-04-06T16:04:01.3903033Z, StoreReadResult: StorePhysicalAddress: rntbd://10.0.0.104:16700/apps/e01795ed-8f69-47b3-ac45-e1a66f2ec14f/services/568a3c5f-3e58-4f21-b744-8f784b91fc61/partitions/197c818e-4a33-4fa0-8031-6c665e25a592/replicas/131662668080760616p, LSN: 465, GlobalCommittedLsn: 465, PartitionKeyRangeId: , IsValid: True, StatusCode: 0, IsGone: False, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0.38, ItemLSN: -1, ResourceType: Document, OperationType: ReadFeed
  164. , SDK: Microsoft.Azure.Documents.Common/1.21.0.0, StatusCode: TooManyRequests
  165. at com.microsoft.azure.documentdb.internal.ErrorUtils.maybeThrowException(ErrorUtils.java:69)
  166. at com.microsoft.azure.documentdb.internal.GatewayProxy.performGetRequest(GatewayProxy.java:245)
  167. at com.microsoft.azure.documentdb.internal.GatewayProxy.doReadFeed(GatewayProxy.java:111)
  168. at com.microsoft.azure.documentdb.internal.GatewayProxy.processMessage(GatewayProxy.java:346)
  169. at com.microsoft.azure.documentdb.DocumentClient$9.apply(DocumentClient.java:3011)
  170. at com.microsoft.azure.documentdb.internal.RetryUtility.executeDocumentClientRequest(RetryUtility.java:58)
  171. at com.microsoft.azure.documentdb.DocumentClient.doReadFeed(DocumentClient.java:3021)
  172. at com.microsoft.azure.documentdb.DocumentQueryClientInternal.doReadFeed(DocumentQueryClientInternal.java:36)
  173. at com.microsoft.azure.documentdb.internal.query.AbstractQueryExecutionContext.executeRequest(AbstractQueryExecutionContext.java:215)
  174. at com.microsoft.azure.documentdb.internal.query.DefaultQueryExecutionContext.executeOnce(DefaultQueryExecutionContext.java:137)
  175. at com.microsoft.azure.documentdb.internal.query.DefaultQueryExecutionContext.fillBuffer(DefaultQueryExecutionContext.java:101)
  176. at com.microsoft.azure.documentdb.internal.query.DefaultQueryExecutionContext.next(DefaultQueryExecutionContext.java:84)
  177. ... 34 more
  178.  
  179. Traceback (most recent call last):
  180. File "/usr/hdp/current/spark2-client/python/pyspark/sql/dataframe.py", line 113, in toJSON
  181. rdd = self._jdf.toJSON()
  182. File "/usr/hdp/current/spark2-client/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__
  183. answer, self.gateway_client, self.target_id, self.name)
  184. File "/usr/hdp/current/spark2-client/python/pyspark/sql/utils.py", line 63, in deco
  185. return f(*a, **kw)
  186. File "/usr/hdp/current/spark2-client/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py", line 319, in get_return_value
  187. format(target_id, ".", name), value)
  188. py4j.protocol.Py4JJavaError: An error occurred while calling o96.toJSON.
  189. : org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 4.0 failed 4 times, most recent failure: Lost task 0.3 in stage 4.0 (TID 7, wn2-MDMstr.zxmmgisclg5udfemnv0v3qva3e.ax.internal.cloudapp.net, executor 1): java.lang.IllegalStateException: com.microsoft.azure.documentdb.DocumentClientException: Message: {"Errors":["Request rate is large"]}
  190. ActivityId: 859bd9a2-eaef-4c32-acb8-43ce021c28c5, Request URI: /apps/e01795ed-8f69-47b3-ac45-e1a66f2ec14f/services/568a3c5f-3e58-4f21-b744-8f784b91fc61/partitions/197c818e-4a33-4fa0-8031-6c665e25a592/replicas/131662668080760616p, RequestStats:
  191. ResponseTime: 2018-04-06T16:04:01.3903033Z, StoreReadResult: StorePhysicalAddress: rntbd://10.0.0.104:16700/apps/e01795ed-8f69-47b3-ac45-e1a66f2ec14f/services/568a3c5f-3e58-4f21-b744-8f784b91fc61/partitions/197c818e-4a33-4fa0-8031-6c665e25a592/replicas/131662668080760616p, LSN: 465, GlobalCommittedLsn: 465, PartitionKeyRangeId: , IsValid: True, StatusCode: 0, IsGone: False, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0.38, ItemLSN: -1, ResourceType: Document, OperationType: ReadFeed
  192. , SDK: Microsoft.Azure.Documents.Common/1.21.0.0, StatusCode: TooManyRequests
  193. at com.microsoft.azure.documentdb.internal.query.DefaultQueryExecutionContext.next(DefaultQueryExecutionContext.java:86)
  194. at com.microsoft.azure.documentdb.internal.query.DefaultQueryExecutionContext.next(DefaultQueryExecutionContext.java:33)
  195. at com.microsoft.azure.documentdb.internal.query.ProxyQueryExecutionContext.<init>(ProxyQueryExecutionContext.java:68)
  196. at com.microsoft.azure.documentdb.internal.query.QueryExecutionContextFactory.createQueryExecutionContext(QueryExecutionContextFactory.java:23)
  197. at com.microsoft.azure.documentdb.QueryIterable.createQueryExecutionContext(QueryIterable.java:70)
  198. at com.microsoft.azure.documentdb.QueryIterable.reset(QueryIterable.java:115)
  199. at com.microsoft.azure.documentdb.QueryIterable.<init>(QueryIterable.java:57)
  200. at com.microsoft.azure.documentdb.QueryIterable.<init>(QueryIterable.java:31)
  201. at com.microsoft.azure.documentdb.DocumentClient.readDocuments(DocumentClient.java:1002)
  202. at com.microsoft.azure.cosmosdb.spark.CosmosDBConnection.readDocuments(CosmosDBConnection.scala:205)
  203. at com.microsoft.azure.cosmosdb.spark.rdd.CosmosDBRDDIterator.queryDocuments$1(CosmosDBRDDIterator.scala:192)
  204. at com.microsoft.azure.cosmosdb.spark.rdd.CosmosDBRDDIterator.reader$lzycompute(CosmosDBRDDIterator.scala:321)
  205. at com.microsoft.azure.cosmosdb.spark.rdd.CosmosDBRDDIterator.reader(CosmosDBRDDIterator.scala:137)
  206. at com.microsoft.azure.cosmosdb.spark.rdd.CosmosDBRDDIterator.hasNext(CosmosDBRDDIterator.scala:334)
  207. at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
  208. at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
  209. at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
  210. at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
  211. at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:395)
  212. at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
  213. at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
  214. at org.apache.spark.util.random.SamplingUtils$.reservoirSampleAndCount(SamplingUtils.scala:41)
  215. at org.apache.spark.RangePartitioner$$anonfun$9.apply(Partitioner.scala:263)
  216. at org.apache.spark.RangePartitioner$$anonfun$9.apply(Partitioner.scala:261)
  217. at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:844)
  218. at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:844)
  219. at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
  220. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  221. at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  222. at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
  223. at org.apache.spark.scheduler.Task.run(Task.scala:108)
  224. at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338)
  225. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  226. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  227. at java.lang.Thread.run(Thread.java:748)
  228. Caused by: com.microsoft.azure.documentdb.DocumentClientException: Message: {"Errors":["Request rate is large"]}
  229. ActivityId: 859bd9a2-eaef-4c32-acb8-43ce021c28c5, Request URI: /apps/e01795ed-8f69-47b3-ac45-e1a66f2ec14f/services/568a3c5f-3e58-4f21-b744-8f784b91fc61/partitions/197c818e-4a33-4fa0-8031-6c665e25a592/replicas/131662668080760616p, RequestStats:
  230. ResponseTime: 2018-04-06T16:04:01.3903033Z, StoreReadResult: StorePhysicalAddress: rntbd://10.0.0.104:16700/apps/e01795ed-8f69-47b3-ac45-e1a66f2ec14f/services/568a3c5f-3e58-4f21-b744-8f784b91fc61/partitions/197c818e-4a33-4fa0-8031-6c665e25a592/replicas/131662668080760616p, LSN: 465, GlobalCommittedLsn: 465, PartitionKeyRangeId: , IsValid: True, StatusCode: 0, IsGone: False, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0.38, ItemLSN: -1, ResourceType: Document, OperationType: ReadFeed
  231. , SDK: Microsoft.Azure.Documents.Common/1.21.0.0, StatusCode: TooManyRequests
  232. at com.microsoft.azure.documentdb.internal.ErrorUtils.maybeThrowException(ErrorUtils.java:69)
  233. at com.microsoft.azure.documentdb.internal.GatewayProxy.performGetRequest(GatewayProxy.java:245)
  234. at com.microsoft.azure.documentdb.internal.GatewayProxy.doReadFeed(GatewayProxy.java:111)
  235. at com.microsoft.azure.documentdb.internal.GatewayProxy.processMessage(GatewayProxy.java:346)
  236. at com.microsoft.azure.documentdb.DocumentClient$9.apply(DocumentClient.java:3011)
  237. at com.microsoft.azure.documentdb.internal.RetryUtility.executeDocumentClientRequest(RetryUtility.java:58)
  238. at com.microsoft.azure.documentdb.DocumentClient.doReadFeed(DocumentClient.java:3021)
  239. at com.microsoft.azure.documentdb.DocumentQueryClientInternal.doReadFeed(DocumentQueryClientInternal.java:36)
  240. at com.microsoft.azure.documentdb.internal.query.AbstractQueryExecutionContext.executeRequest(AbstractQueryExecutionContext.java:215)
  241. at com.microsoft.azure.documentdb.internal.query.DefaultQueryExecutionContext.executeOnce(DefaultQueryExecutionContext.java:137)
  242. at com.microsoft.azure.documentdb.internal.query.DefaultQueryExecutionContext.fillBuffer(DefaultQueryExecutionContext.java:101)
  243. at com.microsoft.azure.documentdb.internal.query.DefaultQueryExecutionContext.next(DefaultQueryExecutionContext.java:84)
  244. ... 34 more
  245.  
  246. Driver stacktrace:
  247. at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1517)
  248. at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1505)
  249. at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1504)
  250. at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
  251. at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
  252. at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1504)
  253. at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:814)
  254. at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:814)
  255. at scala.Option.foreach(Option.scala:257)
  256. at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:814)
  257. at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1732)
  258. at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1687)
  259. at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1676)
  260. at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
  261. at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:630)
  262. at org.apache.spark.SparkContext.runJob(SparkContext.scala:2029)
  263. at org.apache.spark.SparkContext.runJob(SparkContext.scala:2050)
  264. at org.apache.spark.SparkContext.runJob(SparkContext.scala:2069)
  265. at org.apache.spark.SparkContext.runJob(SparkContext.scala:2094)
  266. at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:936)
  267. at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  268. at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
  269. at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
  270. at org.apache.spark.rdd.RDD.collect(RDD.scala:935)
  271. at org.apache.spark.RangePartitioner$.sketch(Partitioner.scala:266)
  272. at org.apache.spark.RangePartitioner.<init>(Partitioner.scala:128)
  273. at org.apache.spark.sql.execution.exchange.ShuffleExchange$.prepareShuffleDependency(ShuffleExchange.scala:221)
  274. at org.apache.spark.sql.execution.exchange.ShuffleExchange.prepareShuffleDependency(ShuffleExchange.scala:87)
  275. at org.apache.spark.sql.execution.exchange.ShuffleExchange$$anonfun$doExecute$1.apply(ShuffleExchange.scala:124)
  276. at org.apache.spark.sql.execution.exchange.ShuffleExchange$$anonfun$doExecute$1.apply(ShuffleExchange.scala:115)
  277. at org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:52)
  278. at org.apache.spark.sql.execution.exchange.ShuffleExchange.doExecute(ShuffleExchange.scala:115)
  279. at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
  280. at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
  281. at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:138)
  282. at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  283. at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:135)
  284. at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:116)
  285. at org.apache.spark.sql.execution.InputAdapter.inputRDDs(WholeStageCodegenExec.scala:252)
  286. at org.apache.spark.sql.execution.SortExec.inputRDDs(SortExec.scala:121)
  287. at org.apache.spark.sql.execution.WholeStageCodegenExec.doExecute(WholeStageCodegenExec.scala:386)
  288. at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
  289. at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
  290. at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:138)
  291. at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  292. at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:135)
  293. at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:116)
  294. at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:92)
  295. at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:92)
  296. at org.apache.spark.sql.Dataset.toJSON(Dataset.scala:2743)
  297. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  298. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  299. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  300. at java.lang.reflect.Method.invoke(Method.java:498)
  301. at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
  302. at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
  303. at py4j.Gateway.invoke(Gateway.java:280)
  304. at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
  305. at py4j.commands.CallCommand.execute(CallCommand.java:79)
  306. at py4j.GatewayConnection.run(GatewayConnection.java:214)
  307. at java.lang.Thread.run(Thread.java:748)
  308. Caused by: java.lang.IllegalStateException: com.microsoft.azure.documentdb.DocumentClientException: Message: {"Errors":["Request rate is large"]}
  309. ActivityId: 859bd9a2-eaef-4c32-acb8-43ce021c28c5, Request URI: /apps/e01795ed-8f69-47b3-ac45-e1a66f2ec14f/services/568a3c5f-3e58-4f21-b744-8f784b91fc61/partitions/197c818e-4a33-4fa0-8031-6c665e25a592/replicas/131662668080760616p, RequestStats:
  310. ResponseTime: 2018-04-06T16:04:01.3903033Z, StoreReadResult: StorePhysicalAddress: rntbd://10.0.0.104:16700/apps/e01795ed-8f69-47b3-ac45-e1a66f2ec14f/services/568a3c5f-3e58-4f21-b744-8f784b91fc61/partitions/197c818e-4a33-4fa0-8031-6c665e25a592/replicas/131662668080760616p, LSN: 465, GlobalCommittedLsn: 465, PartitionKeyRangeId: , IsValid: True, StatusCode: 0, IsGone: False, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0.38, ItemLSN: -1, ResourceType: Document, OperationType: ReadFeed
  311. , SDK: Microsoft.Azure.Documents.Common/1.21.0.0, StatusCode: TooManyRequests
  312. at com.microsoft.azure.documentdb.internal.query.DefaultQueryExecutionContext.next(DefaultQueryExecutionContext.java:86)
  313. at com.microsoft.azure.documentdb.internal.query.DefaultQueryExecutionContext.next(DefaultQueryExecutionContext.java:33)
  314. at com.microsoft.azure.documentdb.internal.query.ProxyQueryExecutionContext.<init>(ProxyQueryExecutionContext.java:68)
  315. at com.microsoft.azure.documentdb.internal.query.QueryExecutionContextFactory.createQueryExecutionContext(QueryExecutionContextFactory.java:23)
  316. at com.microsoft.azure.documentdb.QueryIterable.createQueryExecutionContext(QueryIterable.java:70)
  317. at com.microsoft.azure.documentdb.QueryIterable.reset(QueryIterable.java:115)
  318. at com.microsoft.azure.documentdb.QueryIterable.<init>(QueryIterable.java:57)
  319. at com.microsoft.azure.documentdb.QueryIterable.<init>(QueryIterable.java:31)
  320. at com.microsoft.azure.documentdb.DocumentClient.readDocuments(DocumentClient.java:1002)
  321. at com.microsoft.azure.cosmosdb.spark.CosmosDBConnection.readDocuments(CosmosDBConnection.scala:205)
  322. at com.microsoft.azure.cosmosdb.spark.rdd.CosmosDBRDDIterator.queryDocuments$1(CosmosDBRDDIterator.scala:192)
  323. at com.microsoft.azure.cosmosdb.spark.rdd.CosmosDBRDDIterator.reader$lzycompute(CosmosDBRDDIterator.scala:321)
  324. at com.microsoft.azure.cosmosdb.spark.rdd.CosmosDBRDDIterator.reader(CosmosDBRDDIterator.scala:137)
  325. at com.microsoft.azure.cosmosdb.spark.rdd.CosmosDBRDDIterator.hasNext(CosmosDBRDDIterator.scala:334)
  326. at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
  327. at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
  328. at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
  329. at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
  330. at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:395)
  331. at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
  332. at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
  333. at org.apache.spark.util.random.SamplingUtils$.reservoirSampleAndCount(SamplingUtils.scala:41)
  334. at org.apache.spark.RangePartitioner$$anonfun$9.apply(Partitioner.scala:263)
  335. at org.apache.spark.RangePartitioner$$anonfun$9.apply(Partitioner.scala:261)
  336. at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:844)
  337. at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:844)
  338. at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
  339. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  340. at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  341. at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
  342. at org.apache.spark.scheduler.Task.run(Task.scala:108)
  343. at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338)
  344. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  345. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  346. ... 1 more
  347. Caused by: com.microsoft.azure.documentdb.DocumentClientException: Message: {"Errors":["Request rate is large"]}
  348. ActivityId: 859bd9a2-eaef-4c32-acb8-43ce021c28c5, Request URI: /apps/e01795ed-8f69-47b3-ac45-e1a66f2ec14f/services/568a3c5f-3e58-4f21-b744-8f784b91fc61/partitions/197c818e-4a33-4fa0-8031-6c665e25a592/replicas/131662668080760616p, RequestStats:
  349. ResponseTime: 2018-04-06T16:04:01.3903033Z, StoreReadResult: StorePhysicalAddress: rntbd://10.0.0.104:16700/apps/e01795ed-8f69-47b3-ac45-e1a66f2ec14f/services/568a3c5f-3e58-4f21-b744-8f784b91fc61/partitions/197c818e-4a33-4fa0-8031-6c665e25a592/replicas/131662668080760616p, LSN: 465, GlobalCommittedLsn: 465, PartitionKeyRangeId: , IsValid: True, StatusCode: 0, IsGone: False, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0.38, ItemLSN: -1, ResourceType: Document, OperationType: ReadFeed
  350. , SDK: Microsoft.Azure.Documents.Common/1.21.0.0, StatusCode: TooManyRequests
  351. at com.microsoft.azure.documentdb.internal.ErrorUtils.maybeThrowException(ErrorUtils.java:69)
  352. at com.microsoft.azure.documentdb.internal.GatewayProxy.performGetRequest(GatewayProxy.java:245)
  353. at com.microsoft.azure.documentdb.internal.GatewayProxy.doReadFeed(GatewayProxy.java:111)
  354. at com.microsoft.azure.documentdb.internal.GatewayProxy.processMessage(GatewayProxy.java:346)
  355. at com.microsoft.azure.documentdb.DocumentClient$9.apply(DocumentClient.java:3011)
  356. at com.microsoft.azure.documentdb.internal.RetryUtility.executeDocumentClientRequest(RetryUtility.java:58)
  357. at com.microsoft.azure.documentdb.DocumentClient.doReadFeed(DocumentClient.java:3021)
  358. at com.microsoft.azure.documentdb.DocumentQueryClientInternal.doReadFeed(DocumentQueryClientInternal.java:36)
  359. at com.microsoft.azure.documentdb.internal.query.AbstractQueryExecutionContext.executeRequest(AbstractQueryExecutionContext.java:215)
  360. at com.microsoft.azure.documentdb.internal.query.DefaultQueryExecutionContext.executeOnce(DefaultQueryExecutionContext.java:137)
  361. at com.microsoft.azure.documentdb.internal.query.DefaultQueryExecutionContext.fillBuffer(DefaultQueryExecutionContext.java:101)
  362. at com.microsoft.azure.documentdb.internal.query.DefaultQueryExecutionContext.next(DefaultQueryExecutionContext.java:84)
  363. ... 34 more
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!