API tools faq
paste
Advertisement
Guest User

Untitled

a guest
Dec 5th, 2017
99
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 71.76 KB | None | 0 0
raw download clone embed print report
  1. george@...:~/adam$ bin/adam-submit --jars=$GOOGLE_CLOUD_NIO_JAR -- reads2coverage gs://genomics-public-data/test-data/dna/wgs/hiseq2500/NA12878/H06JUADXX130110.1.ATCACGAT.20k_reads.bam /users/george/out.coverage4.bam
  2. Using ADAM_MAIN=org.bdgenomics.adam.cli.ADAMMain
  3. Using spark-submit=/usr/lib/spark/bin/spark-submit
  4. 17/12/05 07:48:46 INFO org.bdgenomics.adam.cli.ADAMMain: ADAM invoked with args: "reads2coverage" "gs://genomics-public-data/test-data/dna/wgs/hiseq2500/NA12878/H06JUADXX130110.1.ATCACGAT.20k_reads.bam" "/users/george/out.coverage4.bam"
  5. 17/12/05 07:48:49 INFO org.spark_project.jetty.util.log: Logging initialized @4390ms
  6. 17/12/05 07:48:49 INFO org.spark_project.jetty.server.Server: jetty-9.3.z-SNAPSHOT
  7. 17/12/05 07:48:49 INFO org.spark_project.jetty.server.Server: Started @4496ms
  8. 17/12/05 07:48:49 INFO org.spark_project.jetty.server.AbstractConnector: Started ServerConnector@340fbf35{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
  9. 17/12/05 07:48:50 INFO com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase: GHFS version: 1.6.1-hadoop2
  10. 17/12/05 07:48:52 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at mango-2-m/10.142.0.7:8032
  11. 17/12/05 07:48:56 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl: Submitted application application_1512456264283_0011
  12. 17/12/05 07:49:03 INFO org.bdgenomics.adam.rdd.ADAMContext: Loading gs://genomics-public-data/test-data/dna/wgs/hiseq2500/NA12878/H06JUADXX130110.1.ATCACGAT.20k_reads.bam as BAM/CRAM/SAM and converting to AlignmentRecords.
  13. 17/12/05 07:49:05 INFO org.bdgenomics.adam.rdd.ADAMContext: Loaded header from gs://genomics-public-data/test-data/dna/wgs/hiseq2500/NA12878/H06JUADXX130110.1.ATCACGAT.20k_reads.bam
  14. 17/12/05 07:49:07 INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat: Total input files to process : 1
  15. 17/12/05 07:49:08 INFO org.bdgenomics.adam.rdd.feature.RDDBoundFeatureRDD: Saving data in ADAM format
  16. 17/12/05 07:49:26 WARN org.apache.spark.scheduler.TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, mango-2-w-1.c.mango-bdgenomics.internal, executor 1): com.google.cloud.storage.StorageException: Error code 404 trying to get security access token from Compute Engine metadata for the default service account. This may be because the virtual machine instance does not have permission scopes specified.
  17. at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:189)
  18. at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:340)
  19. at com.google.cloud.storage.StorageImpl$5.call(StorageImpl.java:197)
  20. at com.google.cloud.storage.StorageImpl$5.call(StorageImpl.java:194)
  21. at shaded.cloud_nio.com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:89)
  22. at com.google.cloud.RetryHelper.run(RetryHelper.java:74)
  23. at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:51)
  24. at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:194)
  25. at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:208)
  26. at com.google.cloud.storage.contrib.nio.CloudStorageFileSystemProvider.readAttributes(CloudStorageFileSystemProvider.java:647)
  27. at java.nio.file.Files.readAttributes(Files.java:1737)
  28. at java.nio.file.Files.isRegularFile(Files.java:2229)
  29. at htsjdk.samtools.SamFiles.lookForIndex(SamFiles.java:72)
  30. at htsjdk.samtools.SamFiles.findIndex(SamFiles.java:39)
  31. at org.seqdoop.hadoop_bam.BAMRecordReader.initialize(BAMRecordReader.java:143)
  32. at org.seqdoop.hadoop_bam.BAMInputFormat.createRecordReader(BAMInputFormat.java:226)
  33. at org.seqdoop.hadoop_bam.AnySAMInputFormat.createRecordReader(AnySAMInputFormat.java:190)
  34. at org.apache.spark.rdd.NewHadoopRDD$$anon$1.liftedTree1$1(NewHadoopRDD.scala:180)
  35. at org.apache.spark.rdd.NewHadoopRDD$$anon$1.<init>(NewHadoopRDD.scala:179)
  36. at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:134)
  37. at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:69)
  38. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  39. at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  40. at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
  41. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  42. at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  43. at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
  44. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  45. at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  46. at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
  47. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  48. at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  49. at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
  50. at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
  51. at org.apache.spark.scheduler.Task.run(Task.scala:108)
  52. at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
  53. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  54. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  55. at java.lang.Thread.run(Thread.java:748)
  56. Caused by: java.io.IOException: Error code 404 trying to get security access token from Compute Engine metadata for the default service account. This may be because the virtual machine instance does not have permission scopes specified.
  57. at shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials.refreshAccessToken(ComputeEngineCredentials.java:152)
  58. at shaded.cloud_nio.com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:175)
  59. at shaded.cloud_nio.com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:161)
  60. at shaded.cloud_nio.com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:96)
  61. at com.google.cloud.http.HttpTransportOptions$1.initialize(HttpTransportOptions.java:157)
  62. at shaded.cloud_nio.com.google.api.client.http.HttpRequestFactory.buildRequest(HttpRequestFactory.java:93)
  63. at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.buildHttpRequest(AbstractGoogleClientRequest.java:300)
  64. at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
  65. at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
  66. at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
  67. at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:338)
  68. ... 37 more
  69.  
  70. 17/12/05 07:49:27 ERROR org.apache.spark.scheduler.TaskSetManager: Task 0 in stage 0.0 failed 4 times; aborting job
  71. 17/12/05 07:49:27 ERROR org.apache.spark.internal.io.SparkHadoopMapReduceWriter: Aborting job job_20171205074908_0007.
  72. org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, mango-2-w-1.c.mango-bdgenomics.internal, executor 1): com.google.cloud.storage.StorageException: Error code 404 trying to get security access token from Compute Engine metadata for the default service account. This may be because the virtual machine instance does not have permission scopes specified.
  73. at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:189)
  74. at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:340)
  75. at com.google.cloud.storage.StorageImpl$5.call(StorageImpl.java:197)
  76. at com.google.cloud.storage.StorageImpl$5.call(StorageImpl.java:194)
  77. at shaded.cloud_nio.com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:89)
  78. at com.google.cloud.RetryHelper.run(RetryHelper.java:74)
  79. at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:51)
  80. at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:194)
  81. at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:208)
  82. at com.google.cloud.storage.contrib.nio.CloudStorageFileSystemProvider.readAttributes(CloudStorageFileSystemProvider.java:647)
  83. at java.nio.file.Files.readAttributes(Files.java:1737)
  84. at java.nio.file.Files.isRegularFile(Files.java:2229)
  85. at htsjdk.samtools.SamFiles.lookForIndex(SamFiles.java:72)
  86. at htsjdk.samtools.SamFiles.findIndex(SamFiles.java:39)
  87. at org.seqdoop.hadoop_bam.BAMRecordReader.initialize(BAMRecordReader.java:143)
  88. at org.seqdoop.hadoop_bam.BAMInputFormat.createRecordReader(BAMInputFormat.java:226)
  89. at org.seqdoop.hadoop_bam.AnySAMInputFormat.createRecordReader(AnySAMInputFormat.java:190)
  90. at org.apache.spark.rdd.NewHadoopRDD$$anon$1.liftedTree1$1(NewHadoopRDD.scala:180)
  91. at org.apache.spark.rdd.NewHadoopRDD$$anon$1.<init>(NewHadoopRDD.scala:179)
  92. at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:134)
  93. at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:69)
  94. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  95. at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  96. at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
  97. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  98. at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  99. at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
  100. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  101. at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  102. at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
  103. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  104. at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  105. at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
  106. at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
  107. at org.apache.spark.scheduler.Task.run(Task.scala:108)
  108. at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
  109. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  110. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  111. at java.lang.Thread.run(Thread.java:748)
  112. Caused by: java.io.IOException: Error code 404 trying to get security access token from Compute Engine metadata for the default service account. This may be because the virtual machine instance does not have permission scopes specified.
  113. at shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials.refreshAccessToken(ComputeEngineCredentials.java:152)
  114. at shaded.cloud_nio.com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:175)
  115. at shaded.cloud_nio.com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:161)
  116. at shaded.cloud_nio.com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:96)
  117. at com.google.cloud.http.HttpTransportOptions$1.initialize(HttpTransportOptions.java:157)
  118. at shaded.cloud_nio.com.google.api.client.http.HttpRequestFactory.buildRequest(HttpRequestFactory.java:93)
  119. at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.buildHttpRequest(AbstractGoogleClientRequest.java:300)
  120. at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
  121. at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
  122. at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
  123. at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:338)
  124. ... 37 more
  125.  
  126. Driver stacktrace:
  127. at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1499)
  128. at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1487)
  129. at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1486)
  130. at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
  131. at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
  132. at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1486)
  133. at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:814)
  134. at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:814)
  135. at scala.Option.foreach(Option.scala:257)
  136. at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:814)
  137. at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1714)
  138. at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1669)
  139. at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1658)
  140. at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
  141. at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:630)
  142. at org.apache.spark.SparkContext.runJob(SparkContext.scala:2022)
  143. at org.apache.spark.SparkContext.runJob(SparkContext.scala:2043)
  144. at org.apache.spark.SparkContext.runJob(SparkContext.scala:2075)
  145. at org.apache.spark.internal.io.SparkHadoopMapReduceWriter$.write(SparkHadoopMapReduceWriter.scala:88)
  146. at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1085)
  147. at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1085)
  148. at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1085)
  149. at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  150. at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
  151. at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
  152. at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:1084)
  153. at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply$mcV$sp(PairRDDFunctions.scala:1003)
  154. at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:994)
  155. at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:994)
  156. at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  157. at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
  158. at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
  159. at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:994)
  160. at org.apache.spark.rdd.InstrumentedPairRDDFunctions.saveAsNewAPIHadoopFile(InstrumentedPairRDDFunctions.scala:477)
  161. at org.bdgenomics.adam.rdd.ADAMRDDFunctions$$anonfun$saveRddAsParquet$1.apply$mcV$sp(ADAMRDDFunctions.scala:165)
  162. at org.bdgenomics.adam.rdd.ADAMRDDFunctions$$anonfun$saveRddAsParquet$1.apply(ADAMRDDFunctions.scala:149)
  163. at org.bdgenomics.adam.rdd.ADAMRDDFunctions$$anonfun$saveRddAsParquet$1.apply(ADAMRDDFunctions.scala:149)
  164. at scala.Option.fold(Option.scala:158)
  165. at org.apache.spark.rdd.Timer.time(Timer.scala:48)
  166. at org.bdgenomics.adam.rdd.ADAMRDDFunctions.saveRddAsParquet(ADAMRDDFunctions.scala:149)
  167. at org.bdgenomics.adam.rdd.AvroGenomicRDD.saveAsParquet(GenomicRDD.scala:2481)
  168. at org.bdgenomics.adam.rdd.AvroGenomicRDD.saveAsParquet(GenomicRDD.scala:2456)
  169. at org.bdgenomics.adam.rdd.feature.FeatureRDD.save(FeatureRDD.scala:411)
  170. at org.bdgenomics.adam.rdd.feature.CoverageRDD.save(CoverageRDD.scala:209)
  171. at org.bdgenomics.adam.cli.Reads2Coverage.run(Reads2Coverage.scala:102)
  172. at org.bdgenomics.utils.cli.BDGSparkCommand$class.run(BDGCommand.scala:55)
  173. at org.bdgenomics.adam.cli.Reads2Coverage.run(Reads2Coverage.scala:65)
  174. at org.bdgenomics.adam.cli.ADAMMain.apply(ADAMMain.scala:126)
  175. at org.bdgenomics.adam.cli.ADAMMain$.main(ADAMMain.scala:65)
  176. at org.bdgenomics.adam.cli.ADAMMain.main(ADAMMain.scala)
  177. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  178. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  179. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  180. at java.lang.reflect.Method.invoke(Method.java:498)
  181. at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:755)
  182. at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
  183. at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
  184. at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
  185. at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  186. Caused by: com.google.cloud.storage.StorageException: Error code 404 trying to get security access token from Compute Engine metadata for the default service account. This may be because the virtual machine instance does not have permission scopes specified.
  187. at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:189)
  188. at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:340)
  189. at com.google.cloud.storage.StorageImpl$5.call(StorageImpl.java:197)
  190. at com.google.cloud.storage.StorageImpl$5.call(StorageImpl.java:194)
  191. at shaded.cloud_nio.com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:89)
  192. at com.google.cloud.RetryHelper.run(RetryHelper.java:74)
  193. at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:51)
  194. at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:194)
  195. at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:208)
  196. at com.google.cloud.storage.contrib.nio.CloudStorageFileSystemProvider.readAttributes(CloudStorageFileSystemProvider.java:647)
  197. at java.nio.file.Files.readAttributes(Files.java:1737)
  198. at java.nio.file.Files.isRegularFile(Files.java:2229)
  199. at htsjdk.samtools.SamFiles.lookForIndex(SamFiles.java:72)
  200. at htsjdk.samtools.SamFiles.findIndex(SamFiles.java:39)
  201. at org.seqdoop.hadoop_bam.BAMRecordReader.initialize(BAMRecordReader.java:143)
  202. at org.seqdoop.hadoop_bam.BAMInputFormat.createRecordReader(BAMInputFormat.java:226)
  203. at org.seqdoop.hadoop_bam.AnySAMInputFormat.createRecordReader(AnySAMInputFormat.java:190)
  204. at org.apache.spark.rdd.NewHadoopRDD$$anon$1.liftedTree1$1(NewHadoopRDD.scala:180)
  205. at org.apache.spark.rdd.NewHadoopRDD$$anon$1.<init>(NewHadoopRDD.scala:179)
  206. at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:134)
  207. at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:69)
  208. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  209. at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  210. at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
  211. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  212. at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  213. at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
  214. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  215. at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  216. at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
  217. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  218. at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  219. at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
  220. at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
  221. at org.apache.spark.scheduler.Task.run(Task.scala:108)
  222. at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
  223. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  224. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  225. at java.lang.Thread.run(Thread.java:748)
  226. Caused by: java.io.IOException: Error code 404 trying to get security access token from Compute Engine metadata for the default service account. This may be because the virtual machine instance does not have permission scopes specified.
  227. at shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials.refreshAccessToken(ComputeEngineCredentials.java:152)
  228. at shaded.cloud_nio.com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:175)
  229. at shaded.cloud_nio.com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:161)
  230. at shaded.cloud_nio.com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:96)
  231. at com.google.cloud.http.HttpTransportOptions$1.initialize(HttpTransportOptions.java:157)
  232. at shaded.cloud_nio.com.google.api.client.http.HttpRequestFactory.buildRequest(HttpRequestFactory.java:93)
  233. at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.buildHttpRequest(AbstractGoogleClientRequest.java:300)
  234. at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
  235. at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
  236. at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
  237. at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:338)
  238. ... 37 more
  239. Command body threw exception:
  240. org.apache.spark.SparkException: Job aborted.
  241. 17/12/05 07:49:27 INFO org.bdgenomics.adam.cli.Reads2Coverage: Overall Duration: 40.56 secs
  242. Exception in thread "main" org.apache.spark.SparkException: Job aborted.
  243. at org.apache.spark.internal.io.SparkHadoopMapReduceWriter$.write(SparkHadoopMapReduceWriter.scala:107)
  244. at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1085)
  245. at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1085)
  246. at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1085)
  247. at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  248. at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
  249. at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
  250. at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:1084)
  251. at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply$mcV$sp(PairRDDFunctions.scala:1003)
  252. at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:994)
  253. at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:994)
  254. at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  255. at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
  256. at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
  257. at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:994)
  258. at org.apache.spark.rdd.InstrumentedPairRDDFunctions.saveAsNewAPIHadoopFile(InstrumentedPairRDDFunctions.scala:477)
  259. at org.bdgenomics.adam.rdd.ADAMRDDFunctions$$anonfun$saveRddAsParquet$1.apply$mcV$sp(ADAMRDDFunctions.scala:165)
  260. at org.bdgenomics.adam.rdd.ADAMRDDFunctions$$anonfun$saveRddAsParquet$1.apply(ADAMRDDFunctions.scala:149)
  261. at org.bdgenomics.adam.rdd.ADAMRDDFunctions$$anonfun$saveRddAsParquet$1.apply(ADAMRDDFunctions.scala:149)
  262. at scala.Option.fold(Option.scala:158)
  263. at org.apache.spark.rdd.Timer.time(Timer.scala:48)
  264. at org.bdgenomics.adam.rdd.ADAMRDDFunctions.saveRddAsParquet(ADAMRDDFunctions.scala:149)
  265. at org.bdgenomics.adam.rdd.AvroGenomicRDD.saveAsParquet(GenomicRDD.scala:2481)
  266. at org.bdgenomics.adam.rdd.AvroGenomicRDD.saveAsParquet(GenomicRDD.scala:2456)
  267. at org.bdgenomics.adam.rdd.feature.FeatureRDD.save(FeatureRDD.scala:411)
  268. at org.bdgenomics.adam.rdd.feature.CoverageRDD.save(CoverageRDD.scala:209)
  269. at org.bdgenomics.adam.cli.Reads2Coverage.run(Reads2Coverage.scala:102)
  270. at org.bdgenomics.utils.cli.BDGSparkCommand$class.run(BDGCommand.scala:55)
  271. at org.bdgenomics.adam.cli.Reads2Coverage.run(Reads2Coverage.scala:65)
  272. at org.bdgenomics.adam.cli.ADAMMain.apply(ADAMMain.scala:126)
  273. at org.bdgenomics.adam.cli.ADAMMain$.main(ADAMMain.scala:65)
  274. at org.bdgenomics.adam.cli.ADAMMain.main(ADAMMain.scala)
  275. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  276. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  277. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  278. at java.lang.reflect.Method.invoke(Method.java:498)
  279. at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:755)
  280. at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
  281. at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
  282. at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
  283. at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  284. Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, mango-2-w-1.c.mango-bdgenomics.internal, executor 1): com.google.cloud.storage.StorageException: Error code 404 trying to get security access token from Compute Engine metadata for the default service account. This may be because the virtual machine instance does not have permission scopes specified.
  285. at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:189)
  286. at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:340)
  287. at com.google.cloud.storage.StorageImpl$5.call(StorageImpl.java:197)
  288. at com.google.cloud.storage.StorageImpl$5.call(StorageImpl.java:194)
  289. at shaded.cloud_nio.com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:89)
  290. at com.google.cloud.RetryHelper.run(RetryHelper.java:74)
  291. at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:51)
  292. at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:194)
  293. at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:208)
  294. at com.google.cloud.storage.contrib.nio.CloudStorageFileSystemProvider.readAttributes(CloudStorageFileSystemProvider.java:647)
  295. at java.nio.file.Files.readAttributes(Files.java:1737)
  296. at java.nio.file.Files.isRegularFile(Files.java:2229)
  297. at htsjdk.samtools.SamFiles.lookForIndex(SamFiles.java:72)
  298. at htsjdk.samtools.SamFiles.findIndex(SamFiles.java:39)
  299. at org.seqdoop.hadoop_bam.BAMRecordReader.initialize(BAMRecordReader.java:143)
  300. at org.seqdoop.hadoop_bam.BAMInputFormat.createRecordReader(BAMInputFormat.java:226)
  301. at org.seqdoop.hadoop_bam.AnySAMInputFormat.createRecordReader(AnySAMInputFormat.java:190)
  302. at org.apache.spark.rdd.NewHadoopRDD$$anon$1.liftedTree1$1(NewHadoopRDD.scala:180)
  303. at org.apache.spark.rdd.NewHadoopRDD$$anon$1.<init>(NewHadoopRDD.scala:179)
  304. at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:134)
  305. at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:69)
  306. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  307. at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  308. at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
  309. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  310. at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  311. at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
  312. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  313. at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  314. at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
  315. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  316. at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  317. at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
  318. at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
  319. at org.apache.spark.scheduler.Task.run(Task.scala:108)
  320. at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
  321. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  322. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  323. at java.lang.Thread.run(Thread.java:748)
  324. Caused by: java.io.IOException: Error code 404 trying to get security access token from Compute Engine metadata for the default service account. This may be because the virtual machine instance does not have permission scopes specified.
  325. at shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials.refreshAccessToken(ComputeEngineCredentials.java:152)
  326. at shaded.cloud_nio.com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:175)
  327. at shaded.cloud_nio.com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:161)
  328. at shaded.cloud_nio.com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:96)
  329. at com.google.cloud.http.HttpTransportOptions$1.initialize(HttpTransportOptions.java:157)
  330. at shaded.cloud_nio.com.google.api.client.http.HttpRequestFactory.buildRequest(HttpRequestFactory.java:93)
  331. at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.buildHttpRequest(AbstractGoogleClientRequest.java:300)
  332. at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
  333. at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
  334. at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
  335. at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:338)
  336. ... 37 more
  337.  
  338. Driver stacktrace:
  339. at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1499)
  340. at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1487)
  341. at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1486)
  342. at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
  343. at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
  344. at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1486)
  345. at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:814)
  346. at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:814)
  347. at scala.Option.foreach(Option.scala:257)
  348. at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:814)
  349. at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1714)
  350. at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1669)
  351. at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1658)
  352. at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
  353. at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:630)
  354. at org.apache.spark.SparkContext.runJob(SparkContext.scala:2022)
  355. at org.apache.spark.SparkContext.runJob(SparkContext.scala:2043)
  356. at org.apache.spark.SparkContext.runJob(SparkContext.scala:2075)
  357. at org.apache.spark.internal.io.SparkHadoopMapReduceWriter$.write(SparkHadoopMapReduceWriter.scala:88)
  358. ... 40 more
  359. Caused by: com.google.cloud.storage.StorageException: Error code 404 trying to get security access token from Compute Engine metadata for the default service account. This may be because the virtual machine instance does not have permission scopes specified.
  360. at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:189)
  361. at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:340)
  362. at com.google.cloud.storage.StorageImpl$5.call(StorageImpl.java:197)
  363. at com.google.cloud.storage.StorageImpl$5.call(StorageImpl.java:194)
  364. at shaded.cloud_nio.com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:89)
  365. at com.google.cloud.RetryHelper.run(RetryHelper.java:74)
  366. at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:51)
  367. at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:194)
  368. at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:208)
  369. at com.google.cloud.storage.contrib.nio.CloudStorageFileSystemProvider.readAttributes(CloudStorageFileSystemProvider.java:647)
  370. at java.nio.file.Files.readAttributes(Files.java:1737)
  371. at java.nio.file.Files.isRegularFile(Files.java:2229)
  372. at htsjdk.samtools.SamFiles.lookForIndex(SamFiles.java:72)
  373. at htsjdk.samtools.SamFiles.findIndex(SamFiles.java:39)
  374. at org.seqdoop.hadoop_bam.BAMRecordReader.initialize(BAMRecordReader.java:143)
  375. at org.seqdoop.hadoop_bam.BAMInputFormat.createRecordReader(BAMInputFormat.java:226)
  376. at org.seqdoop.hadoop_bam.AnySAMInputFormat.createRecordReader(AnySAMInputFormat.java:190)
  377. at org.apache.spark.rdd.NewHadoopRDD$$anon$1.liftedTree1$1(NewHadoopRDD.scala:180)
  378. at org.apache.spark.rdd.NewHadoopRDD$$anon$1.<init>(NewHadoopRDD.scala:179)
  379. at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:134)
  380. at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:69)
  381. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  382. at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  383. at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
  384. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  385. at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  386. at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
  387. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  388. at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  389. at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
  390. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  391. at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  392. at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
  393. at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
  394. at org.apache.spark.scheduler.Task.run(Task.scala:108)
  395. at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
  396. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  397. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  398. at java.lang.Thread.run(Thread.java:748)
  399. Caused by: java.io.IOException: Error code 404 trying to get security access token from Compute Engine metadata for the default service account. This may be because the virtual machine instance does not have permission scopes specified.
  400. at shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials.refreshAccessToken(ComputeEngineCredentials.java:152)
  401. at shaded.cloud_nio.com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:175)
  402. at shaded.cloud_nio.com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:161)
  403. at shaded.cloud_nio.com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:96)
  404. at com.google.cloud.http.HttpTransportOptions$1.initialize(HttpTransportOptions.java:157)
  405. at shaded.cloud_nio.com.google.api.client.http.HttpRequestFactory.buildRequest(HttpRequestFactory.java:93)
  406. at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.buildHttpRequest(AbstractGoogleClientRequest.java:300)
  407. at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
  408. at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
  409. at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
  410. at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:338)
  411. ... 37 more
  412. 17/12/05 07:49:27 INFO org.spark_project.jetty.server.AbstractConnector: Stopped Spark@340fbf35{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
  413. george@mango-2-m:~/adam$ bin/adam-submit --jars=$GOOGLE_CLOUD_NIO_JAR -- reads2coverage gs://gen^Cics-public-data/test-data/dna/wgs/hiseq2500/NA12878/H06JUADXX130110.1.ATCACGAT.20k_reads.bam /users/george/out.coverage4.bam
  414. george@mango-2-m:~/adam$ sudo bin/adam-submit --jars=$GOOGLE_CLOUD_NIO_JAR -- reads2coverage gs://genomics-public-data/test-data/dna/wgs/hiseq2500/NA12878/H06JUADXX130110.1.ATCACGAT.20k_reads.bam /users/george/out.coverage5.bam
  415. Using ADAM_MAIN=org.bdgenomics.adam.cli.ADAMMain
  416. Using spark-submit=/usr/bin/spark-submit
  417. 17/12/05 07:54:43 INFO org.bdgenomics.adam.cli.ADAMMain: ADAM invoked with args: "reads2coverage" "gs://genomics-public-data/test-data/dna/wgs/hiseq2500/NA12878/H06JUADXX130110.1.ATCACGAT.20k_reads.bam" "/users/george/out.coverage5.bam"
  418. 17/12/05 07:54:46 INFO org.spark_project.jetty.util.log: Logging initialized @4802ms
  419. 17/12/05 07:54:46 INFO org.spark_project.jetty.server.Server: jetty-9.3.z-SNAPSHOT
  420. 17/12/05 07:54:46 INFO org.spark_project.jetty.server.Server: Started @5031ms
  421. 17/12/05 07:54:46 INFO org.spark_project.jetty.server.AbstractConnector: Started ServerConnector@2dc74087{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
  422. 17/12/05 07:54:47 INFO com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase: GHFS version: 1.6.1-hadoop2
  423. 17/12/05 07:54:49 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at mango-2-m/10.142.0.7:8032
  424. 17/12/05 07:54:54 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl: Submitted application application_1512456264283_0012
  425. 17/12/05 07:55:03 INFO org.bdgenomics.adam.rdd.ADAMContext: Loading gs://genomics-public-data/test-data/dna/wgs/hiseq2500/NA12878/H06JUADXX130110.1.ATCACGAT.20k_reads.bam as BAM/CRAM/SAM and converting to AlignmentRecords.
  426. 17/12/05 07:55:05 INFO org.bdgenomics.adam.rdd.ADAMContext: Loaded header from gs://genomics-public-data/test-data/dna/wgs/hiseq2500/NA12878/H06JUADXX130110.1.ATCACGAT.20k_reads.bam
  427. 17/12/05 07:55:07 INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat: Total input files to process : 1
  428. 17/12/05 07:55:08 INFO org.bdgenomics.adam.rdd.feature.RDDBoundFeatureRDD: Saving data in ADAM format
  429. 17/12/05 07:55:26 WARN org.apache.spark.scheduler.TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, mango-2-w-1.c.mango-bdgenomics.internal, executor 1): com.google.cloud.storage.StorageException: Error code 404 trying to get security access token from Compute Engine metadata for the default service account. This may be because the virtual machine instance does not have permission scopes specified.
  430. at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:189)
  431. at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:340)
  432. at com.google.cloud.storage.StorageImpl$5.call(StorageImpl.java:197)
  433. at com.google.cloud.storage.StorageImpl$5.call(StorageImpl.java:194)
  434. at shaded.cloud_nio.com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:89)
  435. at com.google.cloud.RetryHelper.run(RetryHelper.java:74)
  436. at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:51)
  437. at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:194)
  438. at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:208)
  439. at com.google.cloud.storage.contrib.nio.CloudStorageFileSystemProvider.readAttributes(CloudStorageFileSystemProvider.java:647)
  440. at java.nio.file.Files.readAttributes(Files.java:1737)
  441. at java.nio.file.Files.isRegularFile(Files.java:2229)
  442. at htsjdk.samtools.SamFiles.lookForIndex(SamFiles.java:72)
  443. at htsjdk.samtools.SamFiles.findIndex(SamFiles.java:39)
  444. at org.seqdoop.hadoop_bam.BAMRecordReader.initialize(BAMRecordReader.java:143)
  445. at org.seqdoop.hadoop_bam.BAMInputFormat.createRecordReader(BAMInputFormat.java:226)
  446. at org.seqdoop.hadoop_bam.AnySAMInputFormat.createRecordReader(AnySAMInputFormat.java:190)
  447. at org.apache.spark.rdd.NewHadoopRDD$$anon$1.liftedTree1$1(NewHadoopRDD.scala:180)
  448. at org.apache.spark.rdd.NewHadoopRDD$$anon$1.<init>(NewHadoopRDD.scala:179)
  449. at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:134)
  450. at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:69)
  451. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  452. at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  453. at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
  454. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  455. at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  456. at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
  457. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  458. at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  459. at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
  460. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  461. at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  462. at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
  463. at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
  464. at org.apache.spark.scheduler.Task.run(Task.scala:108)
  465. at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
  466. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  467. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  468. at java.lang.Thread.run(Thread.java:748)
  469. Caused by: java.io.IOException: Error code 404 trying to get security access token from Compute Engine metadata for the default service account. This may be because the virtual machine instance does not have permission scopes specified.
  470. at shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials.refreshAccessToken(ComputeEngineCredentials.java:152)
  471. at shaded.cloud_nio.com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:175)
  472. at shaded.cloud_nio.com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:161)
  473. at shaded.cloud_nio.com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:96)
  474. at com.google.cloud.http.HttpTransportOptions$1.initialize(HttpTransportOptions.java:157)
  475. at shaded.cloud_nio.com.google.api.client.http.HttpRequestFactory.buildRequest(HttpRequestFactory.java:93)
  476. at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.buildHttpRequest(AbstractGoogleClientRequest.java:300)
  477. at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
  478. at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
  479. at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
  480. at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:338)
  481. ... 37 more
  482.  
  483. 17/12/05 07:55:27 ERROR org.apache.spark.scheduler.TaskSetManager: Task 0 in stage 0.0 failed 4 times; aborting job
  484. 17/12/05 07:55:27 ERROR org.apache.spark.internal.io.SparkHadoopMapReduceWriter: Aborting job job_20171205075508_0007.
  485. org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, mango-2-w-1.c.mango-bdgenomics.internal, executor 1): com.google.cloud.storage.StorageException: Error code 404 trying to get security access token from Compute Engine metadata for the default service account. This may be because the virtual machine instance does not have permission scopes specified.
  486. at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:189)
  487. at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:340)
  488. at com.google.cloud.storage.StorageImpl$5.call(StorageImpl.java:197)
  489. at com.google.cloud.storage.StorageImpl$5.call(StorageImpl.java:194)
  490. at shaded.cloud_nio.com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:89)
  491. at com.google.cloud.RetryHelper.run(RetryHelper.java:74)
  492. at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:51)
  493. at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:194)
  494. at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:208)
  495. at com.google.cloud.storage.contrib.nio.CloudStorageFileSystemProvider.readAttributes(CloudStorageFileSystemProvider.java:647)
  496. at java.nio.file.Files.readAttributes(Files.java:1737)
  497. at java.nio.file.Files.isRegularFile(Files.java:2229)
  498. at htsjdk.samtools.SamFiles.lookForIndex(SamFiles.java:72)
  499. at htsjdk.samtools.SamFiles.findIndex(SamFiles.java:39)
  500. at org.seqdoop.hadoop_bam.BAMRecordReader.initialize(BAMRecordReader.java:143)
  501. at org.seqdoop.hadoop_bam.BAMInputFormat.createRecordReader(BAMInputFormat.java:226)
  502. at org.seqdoop.hadoop_bam.AnySAMInputFormat.createRecordReader(AnySAMInputFormat.java:190)
  503. at org.apache.spark.rdd.NewHadoopRDD$$anon$1.liftedTree1$1(NewHadoopRDD.scala:180)
  504. at org.apache.spark.rdd.NewHadoopRDD$$anon$1.<init>(NewHadoopRDD.scala:179)
  505. at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:134)
  506. at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:69)
  507. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  508. at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  509. at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
  510. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  511. at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  512. at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
  513. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  514. at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  515. at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
  516. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  517. at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  518. at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
  519. at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
  520. at org.apache.spark.scheduler.Task.run(Task.scala:108)
  521. at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
  522. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  523. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  524. at java.lang.Thread.run(Thread.java:748)
  525. Caused by: java.io.IOException: Error code 404 trying to get security access token from Compute Engine metadata for the default service account. This may be because the virtual machine instance does not have permission scopes specified.
  526. at shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials.refreshAccessToken(ComputeEngineCredentials.java:152)
  527. at shaded.cloud_nio.com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:175)
  528. at shaded.cloud_nio.com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:161)
  529. at shaded.cloud_nio.com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:96)
  530. at com.google.cloud.http.HttpTransportOptions$1.initialize(HttpTransportOptions.java:157)
  531. at shaded.cloud_nio.com.google.api.client.http.HttpRequestFactory.buildRequest(HttpRequestFactory.java:93)
  532. at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.buildHttpRequest(AbstractGoogleClientRequest.java:300)
  533. at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
  534. at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
  535. at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
  536. at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:338)
  537. ... 37 more
  538.  
  539. Driver stacktrace:
  540. at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1499)
  541. at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1487)
  542. at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1486)
  543. at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
  544. at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
  545. at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1486)
  546. at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:814)
  547. at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:814)
  548. at scala.Option.foreach(Option.scala:257)
  549. at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:814)
  550. at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1714)
  551. at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1669)
  552. at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1658)
  553. at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
  554. at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:630)
  555. at org.apache.spark.SparkContext.runJob(SparkContext.scala:2022)
  556. at org.apache.spark.SparkContext.runJob(SparkContext.scala:2043)
  557. at org.apache.spark.SparkContext.runJob(SparkContext.scala:2075)
  558. at org.apache.spark.internal.io.SparkHadoopMapReduceWriter$.write(SparkHadoopMapReduceWriter.scala:88)
  559. at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1085)
  560. at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1085)
  561. at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1085)
  562. at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  563. at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
  564. at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
  565. at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:1084)
  566. at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply$mcV$sp(PairRDDFunctions.scala:1003)
  567. at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:994)
  568. at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:994)
  569. at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  570. at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
  571. at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
  572. at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:994)
  573. at org.apache.spark.rdd.InstrumentedPairRDDFunctions.saveAsNewAPIHadoopFile(InstrumentedPairRDDFunctions.scala:477)
  574. at org.bdgenomics.adam.rdd.ADAMRDDFunctions$$anonfun$saveRddAsParquet$1.apply$mcV$sp(ADAMRDDFunctions.scala:165)
  575. at org.bdgenomics.adam.rdd.ADAMRDDFunctions$$anonfun$saveRddAsParquet$1.apply(ADAMRDDFunctions.scala:149)
  576. at org.bdgenomics.adam.rdd.ADAMRDDFunctions$$anonfun$saveRddAsParquet$1.apply(ADAMRDDFunctions.scala:149)
  577. at scala.Option.fold(Option.scala:158)
  578. at org.apache.spark.rdd.Timer.time(Timer.scala:48)
  579. at org.bdgenomics.adam.rdd.ADAMRDDFunctions.saveRddAsParquet(ADAMRDDFunctions.scala:149)
  580. at org.bdgenomics.adam.rdd.AvroGenomicRDD.saveAsParquet(GenomicRDD.scala:2481)
  581. at org.bdgenomics.adam.rdd.AvroGenomicRDD.saveAsParquet(GenomicRDD.scala:2456)
  582. at org.bdgenomics.adam.rdd.feature.FeatureRDD.save(FeatureRDD.scala:411)
  583. at org.bdgenomics.adam.rdd.feature.CoverageRDD.save(CoverageRDD.scala:209)
  584. at org.bdgenomics.adam.cli.Reads2Coverage.run(Reads2Coverage.scala:102)
  585. at org.bdgenomics.utils.cli.BDGSparkCommand$class.run(BDGCommand.scala:55)
  586. at org.bdgenomics.adam.cli.Reads2Coverage.run(Reads2Coverage.scala:65)
  587. at org.bdgenomics.adam.cli.ADAMMain.apply(ADAMMain.scala:126)
  588. at org.bdgenomics.adam.cli.ADAMMain$.main(ADAMMain.scala:65)
  589. at org.bdgenomics.adam.cli.ADAMMain.main(ADAMMain.scala)
  590. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  591. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  592. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  593. at java.lang.reflect.Method.invoke(Method.java:498)
  594. at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:755)
  595. at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
  596. at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
  597. at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
  598. at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  599. Caused by: com.google.cloud.storage.StorageException: Error code 404 trying to get security access token from Compute Engine metadata for the default service account. This may be because the virtual machine instance does not have permission scopes specified.
  600. at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:189)
  601. at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:340)
  602. at com.google.cloud.storage.StorageImpl$5.call(StorageImpl.java:197)
  603. at com.google.cloud.storage.StorageImpl$5.call(StorageImpl.java:194)
  604. at shaded.cloud_nio.com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:89)
  605. at com.google.cloud.RetryHelper.run(RetryHelper.java:74)
  606. at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:51)
  607. at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:194)
  608. at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:208)
  609. at com.google.cloud.storage.contrib.nio.CloudStorageFileSystemProvider.readAttributes(CloudStorageFileSystemProvider.java:647)
  610. at java.nio.file.Files.readAttributes(Files.java:1737)
  611. at java.nio.file.Files.isRegularFile(Files.java:2229)
  612. at htsjdk.samtools.SamFiles.lookForIndex(SamFiles.java:72)
  613. at htsjdk.samtools.SamFiles.findIndex(SamFiles.java:39)
  614. at org.seqdoop.hadoop_bam.BAMRecordReader.initialize(BAMRecordReader.java:143)
  615. at org.seqdoop.hadoop_bam.BAMInputFormat.createRecordReader(BAMInputFormat.java:226)
  616. at org.seqdoop.hadoop_bam.AnySAMInputFormat.createRecordReader(AnySAMInputFormat.java:190)
  617. at org.apache.spark.rdd.NewHadoopRDD$$anon$1.liftedTree1$1(NewHadoopRDD.scala:180)
  618. at org.apache.spark.rdd.NewHadoopRDD$$anon$1.<init>(NewHadoopRDD.scala:179)
  619. at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:134)
  620. at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:69)
  621. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  622. at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  623. at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
  624. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  625. at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  626. at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
  627. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  628. at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  629. at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
  630. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  631. at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  632. at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
  633. at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
  634. at org.apache.spark.scheduler.Task.run(Task.scala:108)
  635. at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
  636. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  637. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  638. at java.lang.Thread.run(Thread.java:748)
  639. Caused by: java.io.IOException: Error code 404 trying to get security access token from Compute Engine metadata for the default service account. This may be because the virtual machine instance does not have permission scopes specified.
  640. at shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials.refreshAccessToken(ComputeEngineCredentials.java:152)
  641. at shaded.cloud_nio.com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:175)
  642. at shaded.cloud_nio.com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:161)
  643. at shaded.cloud_nio.com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:96)
  644. at com.google.cloud.http.HttpTransportOptions$1.initialize(HttpTransportOptions.java:157)
  645. at shaded.cloud_nio.com.google.api.client.http.HttpRequestFactory.buildRequest(HttpRequestFactory.java:93)
  646. at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.buildHttpRequest(AbstractGoogleClientRequest.java:300)
  647. at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
  648. at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
  649. at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
  650. at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:338)
  651. ... 37 more
  652. Command body threw exception:
  653. org.apache.spark.SparkException: Job aborted.
  654. 17/12/05 07:55:27 INFO org.bdgenomics.adam.cli.Reads2Coverage: Overall Duration: 43.69 secs
  655. Exception in thread "main" org.apache.spark.SparkException: Job aborted.
  656. at org.apache.spark.internal.io.SparkHadoopMapReduceWriter$.write(SparkHadoopMapReduceWriter.scala:107)
  657. at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1085)
  658. at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1085)
  659. at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1085)
  660. at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  661. at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
  662. at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
  663. at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:1084)
  664. at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply$mcV$sp(PairRDDFunctions.scala:1003)
  665. at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:994)
  666. at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:994)
  667. at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  668. at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
  669. at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
  670. at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:994)
  671. at org.apache.spark.rdd.InstrumentedPairRDDFunctions.saveAsNewAPIHadoopFile(InstrumentedPairRDDFunctions.scala:477)
  672. at org.bdgenomics.adam.rdd.ADAMRDDFunctions$$anonfun$saveRddAsParquet$1.apply$mcV$sp(ADAMRDDFunctions.scala:165)
  673. at org.bdgenomics.adam.rdd.ADAMRDDFunctions$$anonfun$saveRddAsParquet$1.apply(ADAMRDDFunctions.scala:149)
  674. at org.bdgenomics.adam.rdd.ADAMRDDFunctions$$anonfun$saveRddAsParquet$1.apply(ADAMRDDFunctions.scala:149)
  675. at scala.Option.fold(Option.scala:158)
  676. at org.apache.spark.rdd.Timer.time(Timer.scala:48)
  677. at org.bdgenomics.adam.rdd.ADAMRDDFunctions.saveRddAsParquet(ADAMRDDFunctions.scala:149)
  678. at org.bdgenomics.adam.rdd.AvroGenomicRDD.saveAsParquet(GenomicRDD.scala:2481)
  679. at org.bdgenomics.adam.rdd.AvroGenomicRDD.saveAsParquet(GenomicRDD.scala:2456)
  680. at org.bdgenomics.adam.rdd.feature.FeatureRDD.save(FeatureRDD.scala:411)
  681. at org.bdgenomics.adam.rdd.feature.CoverageRDD.save(CoverageRDD.scala:209)
  682. at org.bdgenomics.adam.cli.Reads2Coverage.run(Reads2Coverage.scala:102)
  683. at org.bdgenomics.utils.cli.BDGSparkCommand$class.run(BDGCommand.scala:55)
  684. at org.bdgenomics.adam.cli.Reads2Coverage.run(Reads2Coverage.scala:65)
  685. at org.bdgenomics.adam.cli.ADAMMain.apply(ADAMMain.scala:126)
  686. at org.bdgenomics.adam.cli.ADAMMain$.main(ADAMMain.scala:65)
  687. at org.bdgenomics.adam.cli.ADAMMain.main(ADAMMain.scala)
  688. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  689. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  690. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  691. at java.lang.reflect.Method.invoke(Method.java:498)
  692. at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:755)
  693. at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
  694. at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
  695. at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
  696. at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  697. Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, mango-2-w-1.c.mango-bdgenomics.internal, executor 1): com.google.cloud.storage.StorageException: Error code 404 trying to get security access token from Compute Engine metadata for the default service account. This may be because the virtual machine instance does not have permission scopes specified.
  698. at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:189)
  699. at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:340)
  700. at com.google.cloud.storage.StorageImpl$5.call(StorageImpl.java:197)
  701. at com.google.cloud.storage.StorageImpl$5.call(StorageImpl.java:194)
  702. at shaded.cloud_nio.com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:89)
  703. at com.google.cloud.RetryHelper.run(RetryHelper.java:74)
  704. at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:51)
  705. at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:194)
  706. at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:208)
  707. at com.google.cloud.storage.contrib.nio.CloudStorageFileSystemProvider.readAttributes(CloudStorageFileSystemProvider.java:647)
  708. at java.nio.file.Files.readAttributes(Files.java:1737)
  709. at java.nio.file.Files.isRegularFile(Files.java:2229)
  710. at htsjdk.samtools.SamFiles.lookForIndex(SamFiles.java:72)
  711. at htsjdk.samtools.SamFiles.findIndex(SamFiles.java:39)
  712. at org.seqdoop.hadoop_bam.BAMRecordReader.initialize(BAMRecordReader.java:143)
  713. at org.seqdoop.hadoop_bam.BAMInputFormat.createRecordReader(BAMInputFormat.java:226)
  714. at org.seqdoop.hadoop_bam.AnySAMInputFormat.createRecordReader(AnySAMInputFormat.java:190)
  715. at org.apache.spark.rdd.NewHadoopRDD$$anon$1.liftedTree1$1(NewHadoopRDD.scala:180)
  716. at org.apache.spark.rdd.NewHadoopRDD$$anon$1.<init>(NewHadoopRDD.scala:179)
  717. at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:134)
  718. at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:69)
  719. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  720. at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  721. at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
  722. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  723. at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  724. at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
  725. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  726. at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  727. at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
  728. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  729. at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  730. at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
  731. at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
  732. at org.apache.spark.scheduler.Task.run(Task.scala:108)
  733. at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
  734. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  735. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  736. at java.lang.Thread.run(Thread.java:748)
  737. Caused by: java.io.IOException: Error code 404 trying to get security access token from Compute Engine metadata for the default service account. This may be because the virtual machine instance does not have permission scopes specified.
  738. at shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials.refreshAccessToken(ComputeEngineCredentials.java:152)
  739. at shaded.cloud_nio.com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:175)
  740. at shaded.cloud_nio.com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:161)
  741. at shaded.cloud_nio.com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:96)
  742. at com.google.cloud.http.HttpTransportOptions$1.initialize(HttpTransportOptions.java:157)
  743. at shaded.cloud_nio.com.google.api.client.http.HttpRequestFactory.buildRequest(HttpRequestFactory.java:93)
  744. at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.buildHttpRequest(AbstractGoogleClientRequest.java:300)
  745. at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
  746. at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
  747. at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
  748. at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:338)
  749. ... 37 more
  750.  
  751. Driver stacktrace:
  752. at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1499)
  753. at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1487)
  754. at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1486)
  755. at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
  756. at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
  757. at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1486)
  758. at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:814)
  759. at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:814)
  760. at scala.Option.foreach(Option.scala:257)
  761. at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:814)
  762. at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1714)
  763. at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1669)
  764. at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1658)
  765. at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
  766. at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:630)
  767. at org.apache.spark.SparkContext.runJob(SparkContext.scala:2022)
  768. at org.apache.spark.SparkContext.runJob(SparkContext.scala:2043)
  769. at org.apache.spark.SparkContext.runJob(SparkContext.scala:2075)
  770. at org.apache.spark.internal.io.SparkHadoopMapReduceWriter$.write(SparkHadoopMapReduceWriter.scala:88)
  771. ... 40 more
  772. Caused by: com.google.cloud.storage.StorageException: Error code 404 trying to get security access token from Compute Engine metadata for the default service account. This may be because the virtual machine instance does not have permission scopes specified.
  773. at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:189)
  774. at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:340)
  775. at com.google.cloud.storage.StorageImpl$5.call(StorageImpl.java:197)
  776. at com.google.cloud.storage.StorageImpl$5.call(StorageImpl.java:194)
  777. at shaded.cloud_nio.com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:89)
  778. at com.google.cloud.RetryHelper.run(RetryHelper.java:74)
  779. at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:51)
  780. at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:194)
  781. at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:208)
  782. at com.google.cloud.storage.contrib.nio.CloudStorageFileSystemProvider.readAttributes(CloudStorageFileSystemProvider.java:647)
  783. at java.nio.file.Files.readAttributes(Files.java:1737)
  784. at java.nio.file.Files.isRegularFile(Files.java:2229)
  785. at htsjdk.samtools.SamFiles.lookForIndex(SamFiles.java:72)
  786. at htsjdk.samtools.SamFiles.findIndex(SamFiles.java:39)
  787. at org.seqdoop.hadoop_bam.BAMRecordReader.initialize(BAMRecordReader.java:143)
  788. at org.seqdoop.hadoop_bam.BAMInputFormat.createRecordReader(BAMInputFormat.java:226)
  789. at org.seqdoop.hadoop_bam.AnySAMInputFormat.createRecordReader(AnySAMInputFormat.java:190)
  790. at org.apache.spark.rdd.NewHadoopRDD$$anon$1.liftedTree1$1(NewHadoopRDD.scala:180)
  791. at org.apache.spark.rdd.NewHadoopRDD$$anon$1.<init>(NewHadoopRDD.scala:179)
  792. at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:134)
  793. at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:69)
  794. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  795. at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  796. at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
  797. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  798. at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  799. at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
  800. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  801. at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  802. at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
  803. at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  804. at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  805. at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
  806. at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
  807. at org.apache.spark.scheduler.Task.run(Task.scala:108)
  808. at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
  809. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  810. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  811. at java.lang.Thread.run(Thread.java:748)
  812. Caused by: java.io.IOException: Error code 404 trying to get security access token from Compute Engine metadata for the default service account. This may be because the virtual machine instance does not have permission scopes specified.
  813. at shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials.refreshAccessToken(ComputeEngineCredentials.java:152)
  814. at shaded.cloud_nio.com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:175)
  815. at shaded.cloud_nio.com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:161)
  816. at shaded.cloud_nio.com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:96)
  817. at com.google.cloud.http.HttpTransportOptions$1.initialize(HttpTransportOptions.java:157)
  818. at shaded.cloud_nio.com.google.api.client.http.HttpRequestFactory.buildRequest(HttpRequestFactory.java:93)
  819. at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.buildHttpRequest(AbstractGoogleClientRequest.java:300)
  820. at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
  821. at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
  822. at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
  823. at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:338)
  824. ... 37 more
  825. 17/12/05 07:55:27 INFO org.spark_project.jetty.server.AbstractConnector: Stopped Spark@2dc74087{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!