API tools faq
paste
Advertisement
Guest User

Untitled

a guest
Apr 21st, 2015
679
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 163.39 KB | None | 0 0
raw download clone embed print report
  1. Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
  2. 15/04/21 14:58:29 INFO CoarseGrainedExecutorBackend: Registered signal handlers for [TERM, HUP, INT]
  3. 15/04/21 14:58:29 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  4. 15/04/21 14:58:29 INFO SecurityManager: Changing view acls to: me
  5. 15/04/21 14:58:29 INFO SecurityManager: Changing modify acls to: me
  6. 15/04/21 14:58:29 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(me); users with modify permissions: Set(me)
  7. 15/04/21 14:58:30 INFO Slf4jLogger: Slf4jLogger started
  8. 15/04/21 14:58:30 INFO Remoting: Starting remoting
  9. 15/04/21 14:58:30 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://driverPropsFetcher@node7.someuniversity.edu:44699]
  10. 15/04/21 14:58:30 INFO Utils: Successfully started service 'driverPropsFetcher' on port 44699.
  11. 15/04/21 14:58:30 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
  12. 15/04/21 14:58:30 INFO SecurityManager: Changing view acls to: me
  13. 15/04/21 14:58:30 INFO SecurityManager: Changing modify acls to: me
  14. 15/04/21 14:58:30 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(me); users with modify permissions: Set(me)
  15. 15/04/21 14:58:30 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
  16. 15/04/21 14:58:30 INFO Slf4jLogger: Slf4jLogger started
  17. 15/04/21 14:58:30 INFO Remoting: Starting remoting
  18. 15/04/21 14:58:30 INFO RemoteActorRefProvider$RemotingTerminator: Remoting shut down.
  19. 15/04/21 14:58:30 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkExecutor@node7.someuniversity.edu:41485]
  20. 15/04/21 14:58:30 INFO Utils: Successfully started service 'sparkExecutor' on port 41485.
  21. 15/04/21 14:58:30 INFO AkkaUtils: Connecting to MapOutputTracker: akka.tcp://sparkDriver@node2.someuniversity.edu:59512/user/MapOutputTracker
  22. 15/04/21 14:58:30 INFO AkkaUtils: Connecting to BlockManagerMaster: akka.tcp://sparkDriver@node2.someuniversity.edu:59512/user/BlockManagerMaster
  23. 15/04/21 14:58:30 INFO DiskBlockManager: Created local directory at /tmp/spark-0f8d0598-b137-4d14-993a-568b2ab3709a/spark-12d5ff0a-2793-4b76-8a0b-d977a5924925/spark-7ad9382d-05cf-49d4-9a52-d42e6ca7117d/blockmgr-b72d4068-d065-47e6-8a10-867f723000db
  24. 15/04/21 14:58:30 INFO MemoryStore: MemoryStore started with capacity 14.5 GB
  25. 15/04/21 14:58:31 INFO AkkaUtils: Connecting to OutputCommitCoordinator: akka.tcp://sparkDriver@node2.someuniversity.edu:59512/user/OutputCommitCoordinator
  26. 15/04/21 14:58:31 INFO CoarseGrainedExecutorBackend: Connecting to driver: akka.tcp://sparkDriver@node2.someuniversity.edu:59512/user/CoarseGrainedScheduler
  27. 15/04/21 14:58:31 INFO WorkerWatcher: Connecting to worker akka.tcp://sparkWorker@node7.someuniversity.edu:48891/user/Worker
  28. 15/04/21 14:58:31 INFO WorkerWatcher: Successfully connected to akka.tcp://sparkWorker@node7.someuniversity.edu:48891/user/Worker
  29. 15/04/21 14:58:31 INFO CoarseGrainedExecutorBackend: Successfully registered with driver
  30. 15/04/21 14:58:31 INFO Executor: Starting executor ID 0 on host node7.someuniversity.edu
  31. 15/04/21 14:58:31 INFO PlatformDependent: Your platform does not provide complete low-level API for accessing direct buffers reliably. Unless explicitly requested, heap buffer will always be preferred to avoid potential system unstability.
  32. 15/04/21 14:58:31 INFO NettyBlockTransferService: Server created on 33110
  33. 15/04/21 14:58:31 INFO BlockManagerMaster: Trying to register BlockManager
  34. 15/04/21 14:58:31 INFO BlockManagerMaster: Registered BlockManager
  35. 15/04/21 14:58:31 INFO AkkaUtils: Connecting to HeartbeatReceiver: akka.tcp://sparkDriver@node2.someuniversity.edu:59512/user/HeartbeatReceiver
  36. 15/04/21 14:59:07 INFO BlockManager: Removing broadcast 1
  37. 15/04/21 14:59:22 INFO CoarseGrainedExecutorBackend: Got assigned task 1
  38. 15/04/21 14:59:22 INFO CoarseGrainedExecutorBackend: Got assigned task 5
  39. 15/04/21 14:59:22 INFO Executor: Running task 4.0 in stage 1.0 (TID 5)
  40. 15/04/21 14:59:22 INFO Executor: Running task 0.0 in stage 1.0 (TID 1)
  41. 15/04/21 14:59:22 INFO Executor: Fetching http://10.0.0.2:51740/jars/kv-placement-project_2.11-1.0.jar with timestamp 1429642703473
  42. 15/04/21 14:59:22 INFO Utils: Fetching http://10.0.0.2:51740/jars/kv-placement-project_2.11-1.0.jar to /tmp/spark-0f8d0598-b137-4d14-993a-568b2ab3709a/spark-12d5ff0a-2793-4b76-8a0b-d977a5924925/spark-fe3f6321-54dd-45c5-ad8f-203b694b2477/fetchFileTemp224004063349601061.tmp
  43. 15/04/21 14:59:22 INFO Utils: Copying /tmp/spark-0f8d0598-b137-4d14-993a-568b2ab3709a/spark-12d5ff0a-2793-4b76-8a0b-d977a5924925/spark-fe3f6321-54dd-45c5-ad8f-203b694b2477/-15067344311429642703473_cache to /home/me/spark-1.3.1/work/app-20150421145823-0000/0/./kv-placement-project_2.11-1.0.jar
  44. 15/04/21 14:59:22 INFO Executor: Adding file:/home/me/spark-1.3.1/work/app-20150421145823-0000/0/./kv-placement-project_2.11-1.0.jar to class loader
  45. 15/04/21 14:59:22 INFO TorrentBroadcast: Started reading broadcast variable 2
  46. 15/04/21 14:59:22 INFO MemoryStore: ensureFreeSpace(1919) called with curMem=0, maxMem=15558542622
  47. 15/04/21 14:59:22 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 1919.0 B, free 14.5 GB)
  48. 15/04/21 14:59:22 INFO BlockManagerMaster: Updated info of block broadcast_2_piece0
  49. 15/04/21 14:59:22 INFO TorrentBroadcast: Reading broadcast variable 2 took 174 ms
  50. 15/04/21 14:59:22 INFO MemoryStore: ensureFreeSpace(2560) called with curMem=1919, maxMem=15558542622
  51. 15/04/21 14:59:22 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 2.5 KB, free 14.5 GB)
  52. 15/04/21 14:59:22 INFO HadoopRDD: Input split: file:/home/me/kv-placement/input:134217728+33554432
  53. 15/04/21 14:59:22 INFO HadoopRDD: Input split: file:/home/me/kv-placement/input:0+33554432
  54. 15/04/21 14:59:22 INFO TorrentBroadcast: Started reading broadcast variable 0
  55. 15/04/21 14:59:22 INFO MemoryStore: ensureFreeSpace(22692) called with curMem=4479, maxMem=15558542622
  56. 15/04/21 14:59:22 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 22.2 KB, free 14.5 GB)
  57. 15/04/21 14:59:22 INFO BlockManagerMaster: Updated info of block broadcast_0_piece0
  58. 15/04/21 14:59:22 INFO TorrentBroadcast: Reading broadcast variable 0 took 86 ms
  59. 15/04/21 14:59:22 INFO MemoryStore: ensureFreeSpace(291893) called with curMem=27171, maxMem=15558542622
  60. 15/04/21 14:59:22 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 285.1 KB, free 14.5 GB)
  61. 15/04/21 14:59:23 INFO deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id
  62. 15/04/21 14:59:23 INFO deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
  63. 15/04/21 14:59:23 INFO deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
  64. 15/04/21 14:59:23 INFO deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
  65. 15/04/21 14:59:23 INFO deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
  66. 15/04/21 14:59:23 INFO Executor: Finished task 4.0 in stage 1.0 (TID 5). 1839 bytes result sent to driver
  67. 15/04/21 14:59:23 INFO Executor: Finished task 0.0 in stage 1.0 (TID 1). 1839 bytes result sent to driver
  68. 15/04/21 14:59:24 INFO CoarseGrainedExecutorBackend: Got assigned task 7
  69. 15/04/21 14:59:24 INFO Executor: Running task 1.0 in stage 2.0 (TID 7)
  70. 15/04/21 14:59:24 INFO CoarseGrainedExecutorBackend: Got assigned task 11
  71. 15/04/21 14:59:24 INFO Executor: Running task 5.0 in stage 2.0 (TID 11)
  72. 15/04/21 14:59:24 INFO TorrentBroadcast: Started reading broadcast variable 3
  73. 15/04/21 14:59:24 INFO MemoryStore: ensureFreeSpace(3785180) called with curMem=319064, maxMem=15558542622
  74. 15/04/21 14:59:24 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 3.6 MB, free 14.5 GB)
  75. 15/04/21 14:59:24 INFO BlockManagerMaster: Updated info of block broadcast_3_piece0
  76. 15/04/21 14:59:24 INFO TorrentBroadcast: Reading broadcast variable 3 took 139 ms
  77. 15/04/21 14:59:24 INFO MemoryStore: ensureFreeSpace(7603712) called with curMem=4104244, maxMem=15558542622
  78. 15/04/21 14:59:24 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 7.3 MB, free 14.5 GB)
  79. 15/04/21 14:59:25 INFO HadoopRDD: Input split: file:/home/me/kv-placement/input:33554432+33554432
  80. 15/04/21 14:59:25 INFO HadoopRDD: Input split: file:/home/me/kv-placement/input:167772160+6823682
  81. 15/04/21 14:59:25 INFO Executor: Finished task 5.0 in stage 2.0 (TID 11). 2017 bytes result sent to driver
  82. 15/04/21 14:59:26 INFO Executor: Finished task 1.0 in stage 2.0 (TID 7). 2017 bytes result sent to driver
  83. 15/04/21 14:59:27 INFO CoarseGrainedExecutorBackend: Got assigned task 13
  84. 15/04/21 14:59:27 INFO Executor: Running task 1.0 in stage 3.0 (TID 13)
  85. 15/04/21 14:59:27 INFO MapOutputTrackerWorker: Updating epoch to 1 and clearing cache
  86. 15/04/21 14:59:27 INFO TorrentBroadcast: Started reading broadcast variable 4
  87. 15/04/21 14:59:27 INFO MemoryStore: ensureFreeSpace(3783686) called with curMem=11707956, maxMem=15558542622
  88. 15/04/21 14:59:27 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 3.6 MB, free 14.5 GB)
  89. 15/04/21 14:59:27 INFO BlockManagerMaster: Updated info of block broadcast_4_piece0
  90. 15/04/21 14:59:27 INFO TorrentBroadcast: Reading broadcast variable 4 took 105 ms
  91. 15/04/21 14:59:27 INFO MemoryStore: ensureFreeSpace(7602648) called with curMem=15491642, maxMem=15558542622
  92. 15/04/21 14:59:27 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 7.3 MB, free 14.5 GB)
  93. 15/04/21 14:59:27 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 0, fetching them
  94. 15/04/21 14:59:27 INFO MapOutputTrackerWorker: Doing the fetch; tracker actor = Actor[akka.tcp://sparkDriver@node2.someuniversity.edu:59512/user/MapOutputTracker#5735832]
  95. 15/04/21 14:59:27 INFO MapOutputTrackerWorker: Got the output locations
  96. 15/04/21 14:59:27 INFO ShuffleBlockFetcherIterator: Getting 6 non-empty blocks out of 6 blocks
  97. 15/04/21 14:59:27 INFO ShuffleBlockFetcherIterator: Started 3 remote fetches in 43 ms
  98. 15/04/21 14:59:27 ERROR TransportResponseHandler: Still have 1 requests outstanding when connection from node6.someuniversity.edu/10.0.0.6:42828 is closed
  99. 15/04/21 14:59:27 INFO RetryingBlockFetcher: Retrying fetch (1/3) for 1 outstanding blocks after 5000 ms
  100. 15/04/21 14:59:28 INFO Executor: Finished task 1.0 in stage 3.0 (TID 13). 629 bytes result sent to driver
  101. 15/04/21 14:59:28 ERROR TransportRequestHandler: Error sending result ChunkFetchSuccess{streamChunkId=StreamChunkId{streamId=1601401593000, chunkIndex=0}, buffer=FileSegmentManagedBuffer{file=/tmp/spark-0f8d0598-b137-4d14-993a-568b2ab3709a/spark-12d5ff0a-2793-4b76-8a0b-d977a5924925/spark-7ad9382d-05cf-49d4-9a52-d42e6ca7117d/blockmgr-b72d4068-d065-47e6-8a10-867f723000db/15/shuffle_0_1_0.data, offset=26501223, length=6227612}} to /10.0.0.5:41160; closing connection
  102. java.io.IOException: Resource temporarily unavailable
  103. at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
  104. at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:415)
  105. at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:516)
  106. at org.apache.spark.network.buffer.LazyFileRegion.transferTo(LazyFileRegion.java:96)
  107. at org.apache.spark.network.protocol.MessageWithHeader.transferTo(MessageWithHeader.java:89)
  108. at io.netty.channel.socket.nio.NioSocketChannel.doWriteFileRegion(NioSocketChannel.java:237)
  109. at io.netty.channel.nio.AbstractNioByteChannel.doWrite(AbstractNioByteChannel.java:233)
  110. at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:264)
  111. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:707)
  112. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:315)
  113. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:676)
  114. at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1059)
  115. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  116. at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:669)
  117. at io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115)
  118. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  119. at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:718)
  120. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:706)
  121. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:741)
  122. at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:895)
  123. at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:240)
  124. at org.apache.spark.network.server.TransportRequestHandler.respond(TransportRequestHandler.java:147)
  125. at org.apache.spark.network.server.TransportRequestHandler.processFetchRequest(TransportRequestHandler.java:119)
  126. at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:95)
  127. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:91)
  128. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:44)
  129. at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
  130. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  131. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  132. at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
  133. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  134. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  135. at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
  136. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  137. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  138. at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
  139. at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
  140. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
  141. at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
  142. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
  143. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
  144. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  145. at java.lang.Thread.run(Thread.java:619)
  146. 15/04/21 14:59:28 ERROR TransportRequestHandler: Error sending result ChunkFetchSuccess{streamChunkId=StreamChunkId{streamId=1601401593000, chunkIndex=1}, buffer=FileSegmentManagedBuffer{file=/tmp/spark-0f8d0598-b137-4d14-993a-568b2ab3709a/spark-12d5ff0a-2793-4b76-8a0b-d977a5924925/spark-7ad9382d-05cf-49d4-9a52-d42e6ca7117d/blockmgr-b72d4068-d065-47e6-8a10-867f723000db/27/shuffle_0_5_0.data, offset=3792987, length=2862285}} to /10.0.0.5:41160; closing connection
  147. java.nio.channels.ClosedChannelException
  148. 15/04/21 14:59:28 ERROR TransportRequestHandler: Error sending result ChunkFetchSuccess{streamChunkId=StreamChunkId{streamId=1601401593002, chunkIndex=0}, buffer=FileSegmentManagedBuffer{file=/tmp/spark-0f8d0598-b137-4d14-993a-568b2ab3709a/spark-12d5ff0a-2793-4b76-8a0b-d977a5924925/spark-7ad9382d-05cf-49d4-9a52-d42e6ca7117d/blockmgr-b72d4068-d065-47e6-8a10-867f723000db/15/shuffle_0_1_0.data, offset=0, length=10993212}} to /10.0.0.6:42426; closing connection
  149. java.io.IOException: Resource temporarily unavailable
  150. at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
  151. at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:415)
  152. at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:516)
  153. at org.apache.spark.network.buffer.LazyFileRegion.transferTo(LazyFileRegion.java:96)
  154. at org.apache.spark.network.protocol.MessageWithHeader.transferTo(MessageWithHeader.java:89)
  155. at io.netty.channel.socket.nio.NioSocketChannel.doWriteFileRegion(NioSocketChannel.java:237)
  156. at io.netty.channel.nio.AbstractNioByteChannel.doWrite(AbstractNioByteChannel.java:233)
  157. at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:264)
  158. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:707)
  159. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:315)
  160. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:676)
  161. at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1059)
  162. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  163. at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:669)
  164. at io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115)
  165. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  166. at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:718)
  167. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:706)
  168. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:741)
  169. at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:895)
  170. at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:240)
  171. at org.apache.spark.network.server.TransportRequestHandler.respond(TransportRequestHandler.java:147)
  172. at org.apache.spark.network.server.TransportRequestHandler.processFetchRequest(TransportRequestHandler.java:119)
  173. at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:95)
  174. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:91)
  175. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:44)
  176. at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
  177. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  178. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  179. at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
  180. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  181. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  182. at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
  183. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  184. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  185. at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
  186. at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
  187. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
  188. at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
  189. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
  190. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
  191. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  192. at java.lang.Thread.run(Thread.java:619)
  193. 15/04/21 14:59:28 WARN TransportChannelHandler: Exception in connection from node5.someuniversity.edu/10.0.0.5:60089
  194. java.io.IOException: Connection reset by peer
  195. at sun.nio.ch.FileDispatcher.read0(Native Method)
  196. at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
  197. at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
  198. at sun.nio.ch.IOUtil.read(IOUtil.java:206)
  199. at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
  200. at io.netty.buffer.PooledHeapByteBuf.setBytes(PooledHeapByteBuf.java:234)
  201. at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:881)
  202. at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:225)
  203. at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)
  204. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
  205. at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
  206. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
  207. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
  208. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  209. at java.lang.Thread.run(Thread.java:619)
  210. 15/04/21 14:59:28 ERROR TransportResponseHandler: Still have 2 requests outstanding when connection from node5.someuniversity.edu/10.0.0.5:60089 is closed
  211. 15/04/21 14:59:28 INFO RetryingBlockFetcher: Retrying fetch (1/3) for 2 outstanding blocks after 5000 ms
  212. 15/04/21 14:59:28 ERROR TransportRequestHandler: Error sending result ChunkFetchSuccess{streamChunkId=StreamChunkId{streamId=1601401593003, chunkIndex=0}, buffer=FileSegmentManagedBuffer{file=/tmp/spark-0f8d0598-b137-4d14-993a-568b2ab3709a/spark-12d5ff0a-2793-4b76-8a0b-d977a5924925/spark-7ad9382d-05cf-49d4-9a52-d42e6ca7117d/blockmgr-b72d4068-d065-47e6-8a10-867f723000db/15/shuffle_0_1_0.data, offset=18488254, length=8012969}} to /10.0.0.3:41348; closing connection
  213. java.io.IOException: Resource temporarily unavailable
  214. at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
  215. at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:415)
  216. at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:516)
  217. at org.apache.spark.network.buffer.LazyFileRegion.transferTo(LazyFileRegion.java:96)
  218. at org.apache.spark.network.protocol.MessageWithHeader.transferTo(MessageWithHeader.java:89)
  219. at io.netty.channel.socket.nio.NioSocketChannel.doWriteFileRegion(NioSocketChannel.java:237)
  220. at io.netty.channel.nio.AbstractNioByteChannel.doWrite(AbstractNioByteChannel.java:233)
  221. at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:264)
  222. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:707)
  223. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:315)
  224. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:676)
  225. at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1059)
  226. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  227. at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:669)
  228. at io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115)
  229. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  230. at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:718)
  231. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:706)
  232. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:741)
  233. at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:895)
  234. at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:240)
  235. at org.apache.spark.network.server.TransportRequestHandler.respond(TransportRequestHandler.java:147)
  236. at org.apache.spark.network.server.TransportRequestHandler.processFetchRequest(TransportRequestHandler.java:119)
  237. at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:95)
  238. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:91)
  239. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:44)
  240. at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
  241. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  242. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  243. at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
  244. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  245. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  246. at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
  247. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  248. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  249. at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
  250. at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
  251. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
  252. at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
  253. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
  254. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
  255. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  256. at java.lang.Thread.run(Thread.java:619)
  257. 15/04/21 14:59:28 ERROR TransportResponseHandler: Still have 1 requests outstanding when connection from node3.someuniversity.edu/10.0.0.3:55531 is closed
  258. 15/04/21 14:59:28 INFO RetryingBlockFetcher: Retrying fetch (1/3) for 1 outstanding blocks after 5000 ms
  259. 15/04/21 14:59:29 INFO CoarseGrainedExecutorBackend: Got assigned task 18
  260. 15/04/21 14:59:29 INFO Executor: Running task 2.0 in stage 5.0 (TID 18)
  261. 15/04/21 14:59:29 INFO TorrentBroadcast: Started reading broadcast variable 5
  262. 15/04/21 14:59:29 INFO MemoryStore: ensureFreeSpace(3787132) called with curMem=23094290, maxMem=15558542622
  263. 15/04/21 14:59:29 INFO MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 3.6 MB, free 14.5 GB)
  264. 15/04/21 14:59:29 INFO BlockManagerMaster: Updated info of block broadcast_5_piece0
  265. 15/04/21 14:59:29 INFO TorrentBroadcast: Reading broadcast variable 5 took 161 ms
  266. 15/04/21 14:59:29 INFO MemoryStore: ensureFreeSpace(7602960) called with curMem=26881422, maxMem=15558542622
  267. 15/04/21 14:59:29 INFO MemoryStore: Block broadcast_5 stored as values in memory (estimated size 7.3 MB, free 14.5 GB)
  268. 15/04/21 14:59:29 INFO ShuffleBlockFetcherIterator: Getting 6 non-empty blocks out of 6 blocks
  269. 15/04/21 14:59:29 INFO TransportClientFactory: Found inactive connection to node3.someuniversity.edu/10.0.0.3:55531, creating a new one.
  270. 15/04/21 14:59:29 INFO TransportClientFactory: Found inactive connection to node5.someuniversity.edu/10.0.0.5:60089, creating a new one.
  271. 15/04/21 14:59:29 INFO TransportClientFactory: Found inactive connection to node6.someuniversity.edu/10.0.0.6:42828, creating a new one.
  272. 15/04/21 14:59:29 INFO ShuffleBlockFetcherIterator: Started 3 remote fetches in 46 ms
  273. 15/04/21 14:59:29 ERROR TransportRequestHandler: Error sending result ChunkFetchSuccess{streamChunkId=StreamChunkId{streamId=1601401593006, chunkIndex=0}, buffer=FileSegmentManagedBuffer{file=/tmp/spark-0f8d0598-b137-4d14-993a-568b2ab3709a/spark-12d5ff0a-2793-4b76-8a0b-d977a5924925/spark-7ad9382d-05cf-49d4-9a52-d42e6ca7117d/blockmgr-b72d4068-d065-47e6-8a10-867f723000db/15/shuffle_0_1_0.data, offset=0, length=10993212}} to /10.0.0.3:41351; closing connection
  274. java.io.IOException: Resource temporarily unavailable
  275. at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
  276. at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:415)
  277. at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:516)
  278. at org.apache.spark.network.buffer.LazyFileRegion.transferTo(LazyFileRegion.java:96)
  279. at org.apache.spark.network.protocol.MessageWithHeader.transferTo(MessageWithHeader.java:89)
  280. at io.netty.channel.socket.nio.NioSocketChannel.doWriteFileRegion(NioSocketChannel.java:237)
  281. at io.netty.channel.nio.AbstractNioByteChannel.doWrite(AbstractNioByteChannel.java:233)
  282. at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:264)
  283. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:707)
  284. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:315)
  285. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:676)
  286. at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1059)
  287. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  288. at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:669)
  289. at io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115)
  290. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  291. at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:718)
  292. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:706)
  293. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:741)
  294. at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:895)
  295. at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:240)
  296. at org.apache.spark.network.server.TransportRequestHandler.respond(TransportRequestHandler.java:147)
  297. at org.apache.spark.network.server.TransportRequestHandler.processFetchRequest(TransportRequestHandler.java:119)
  298. at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:95)
  299. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:91)
  300. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:44)
  301. at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
  302. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  303. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  304. at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
  305. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  306. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  307. at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
  308. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  309. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  310. at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
  311. at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
  312. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
  313. at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
  314. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
  315. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
  316. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  317. at java.lang.Thread.run(Thread.java:619)
  318. 15/04/21 14:59:29 ERROR TransportResponseHandler: Still have 1 requests outstanding when connection from node3.someuniversity.edu/10.0.0.3:55531 is closed
  319. 15/04/21 14:59:29 INFO RetryingBlockFetcher: Retrying fetch (1/3) for 1 outstanding blocks after 5000 ms
  320. 15/04/21 14:59:29 ERROR TransportResponseHandler: Still have 1 requests outstanding when connection from node6.someuniversity.edu/10.0.0.6:42828 is closed
  321. 15/04/21 14:59:29 INFO RetryingBlockFetcher: Retrying fetch (1/3) for 1 outstanding blocks after 5000 ms
  322. 15/04/21 14:59:29 ERROR TransportRequestHandler: Error sending result ChunkFetchSuccess{streamChunkId=StreamChunkId{streamId=1601401593004, chunkIndex=0}, buffer=FileSegmentManagedBuffer{file=/tmp/spark-0f8d0598-b137-4d14-993a-568b2ab3709a/spark-12d5ff0a-2793-4b76-8a0b-d977a5924925/spark-7ad9382d-05cf-49d4-9a52-d42e6ca7117d/blockmgr-b72d4068-d065-47e6-8a10-867f723000db/15/shuffle_0_1_0.data, offset=10993212, length=7495042}} to /10.0.0.5:41165; closing connection
  323. java.io.IOException: Resource temporarily unavailable
  324. at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
  325. at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:415)
  326. at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:516)
  327. at org.apache.spark.network.buffer.LazyFileRegion.transferTo(LazyFileRegion.java:96)
  328. at org.apache.spark.network.protocol.MessageWithHeader.transferTo(MessageWithHeader.java:89)
  329. at io.netty.channel.socket.nio.NioSocketChannel.doWriteFileRegion(NioSocketChannel.java:237)
  330. at io.netty.channel.nio.AbstractNioByteChannel.doWrite(AbstractNioByteChannel.java:233)
  331. at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:264)
  332. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:707)
  333. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:315)
  334. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:676)
  335. at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1059)
  336. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  337. at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:669)
  338. at io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115)
  339. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  340. at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:718)
  341. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:706)
  342. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:741)
  343. at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:895)
  344. at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:240)
  345. at org.apache.spark.network.server.TransportRequestHandler.respond(TransportRequestHandler.java:147)
  346. at org.apache.spark.network.server.TransportRequestHandler.processFetchRequest(TransportRequestHandler.java:119)
  347. at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:95)
  348. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:91)
  349. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:44)
  350. at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
  351. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  352. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  353. at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
  354. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  355. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  356. at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
  357. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  358. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  359. at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
  360. at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
  361. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
  362. at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
  363. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
  364. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
  365. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  366. at java.lang.Thread.run(Thread.java:619)
  367. 15/04/21 14:59:29 ERROR TransportRequestHandler: Error sending result ChunkFetchSuccess{streamChunkId=StreamChunkId{streamId=1601401593004, chunkIndex=1}, buffer=FileSegmentManagedBuffer{file=/tmp/spark-0f8d0598-b137-4d14-993a-568b2ab3709a/spark-12d5ff0a-2793-4b76-8a0b-d977a5924925/spark-7ad9382d-05cf-49d4-9a52-d42e6ca7117d/blockmgr-b72d4068-d065-47e6-8a10-867f723000db/27/shuffle_0_5_0.data, offset=26, length=2204361}} to /10.0.0.5:41165; closing connection
  368. java.nio.channels.ClosedChannelException
  369. 15/04/21 14:59:29 WARN TransportChannelHandler: Exception in connection from node5.someuniversity.edu/10.0.0.5:60089
  370. java.io.IOException: Connection reset by peer
  371. at sun.nio.ch.FileDispatcher.read0(Native Method)
  372. at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
  373. at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
  374. at sun.nio.ch.IOUtil.read(IOUtil.java:206)
  375. at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
  376. at io.netty.buffer.PooledHeapByteBuf.setBytes(PooledHeapByteBuf.java:234)
  377. at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:881)
  378. at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:225)
  379. at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)
  380. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
  381. at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
  382. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
  383. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
  384. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  385. at java.lang.Thread.run(Thread.java:619)
  386. 15/04/21 14:59:29 ERROR TransportResponseHandler: Still have 2 requests outstanding when connection from node5.someuniversity.edu/10.0.0.5:60089 is closed
  387. 15/04/21 14:59:29 INFO RetryingBlockFetcher: Retrying fetch (1/3) for 2 outstanding blocks after 5000 ms
  388. 15/04/21 14:59:29 ERROR TransportRequestHandler: Error sending result ChunkFetchSuccess{streamChunkId=StreamChunkId{streamId=1601401593007, chunkIndex=0}, buffer=FileSegmentManagedBuffer{file=/tmp/spark-0f8d0598-b137-4d14-993a-568b2ab3709a/spark-12d5ff0a-2793-4b76-8a0b-d977a5924925/spark-7ad9382d-05cf-49d4-9a52-d42e6ca7117d/blockmgr-b72d4068-d065-47e6-8a10-867f723000db/15/shuffle_0_1_0.data, offset=26501223, length=6227612}} to /10.0.0.6:42430; closing connection
  389. java.io.IOException: Resource temporarily unavailable
  390. at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
  391. at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:415)
  392. at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:516)
  393. at org.apache.spark.network.buffer.LazyFileRegion.transferTo(LazyFileRegion.java:96)
  394. at org.apache.spark.network.protocol.MessageWithHeader.transferTo(MessageWithHeader.java:89)
  395. at io.netty.channel.socket.nio.NioSocketChannel.doWriteFileRegion(NioSocketChannel.java:237)
  396. at io.netty.channel.nio.AbstractNioByteChannel.doWrite(AbstractNioByteChannel.java:233)
  397. at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:264)
  398. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:707)
  399. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:315)
  400. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:676)
  401. at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1059)
  402. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  403. at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:669)
  404. at io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115)
  405. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  406. at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:718)
  407. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:706)
  408. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:741)
  409. at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:895)
  410. at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:240)
  411. at org.apache.spark.network.server.TransportRequestHandler.respond(TransportRequestHandler.java:147)
  412. at org.apache.spark.network.server.TransportRequestHandler.processFetchRequest(TransportRequestHandler.java:119)
  413. at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:95)
  414. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:91)
  415. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:44)
  416. at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
  417. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  418. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  419. at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
  420. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  421. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  422. at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
  423. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  424. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  425. at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
  426. at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
  427. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
  428. at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
  429. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
  430. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
  431. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  432. at java.lang.Thread.run(Thread.java:619)
  433. 15/04/21 14:59:33 INFO TransportClientFactory: Found inactive connection to node6.someuniversity.edu/10.0.0.6:42828, creating a new one.
  434. 15/04/21 14:59:33 INFO TransportClientFactory: Found inactive connection to node5.someuniversity.edu/10.0.0.5:60089, creating a new one.
  435. 15/04/21 14:59:33 WARN TransportChannelHandler: Exception in connection from node5.someuniversity.edu/10.0.0.5:60089
  436. java.io.IOException: Connection reset by peer
  437. at sun.nio.ch.FileDispatcher.read0(Native Method)
  438. at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
  439. at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
  440. at sun.nio.ch.IOUtil.read(IOUtil.java:206)
  441. at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
  442. at io.netty.buffer.PooledHeapByteBuf.setBytes(PooledHeapByteBuf.java:234)
  443. at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:881)
  444. at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:225)
  445. at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)
  446. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
  447. at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
  448. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
  449. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
  450. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  451. at java.lang.Thread.run(Thread.java:619)
  452. 15/04/21 14:59:33 ERROR TransportResponseHandler: Still have 2 requests outstanding when connection from node5.someuniversity.edu/10.0.0.5:60089 is closed
  453. 15/04/21 14:59:33 INFO RetryingBlockFetcher: Retrying fetch (2/3) for 2 outstanding blocks after 5000 ms
  454. 15/04/21 14:59:33 ERROR TransportResponseHandler: Still have 1 requests outstanding when connection from node6.someuniversity.edu/10.0.0.6:42828 is closed
  455. 15/04/21 14:59:33 INFO TransportClientFactory: Found inactive connection to node3.someuniversity.edu/10.0.0.3:55531, creating a new one.
  456. 15/04/21 14:59:33 INFO RetryingBlockFetcher: Retrying fetch (2/3) for 1 outstanding blocks after 5000 ms
  457. 15/04/21 14:59:33 ERROR TransportRequestHandler: Error sending result ChunkFetchSuccess{streamChunkId=StreamChunkId{streamId=1601401593009, chunkIndex=0}, buffer=FileSegmentManagedBuffer{file=/tmp/spark-0f8d0598-b137-4d14-993a-568b2ab3709a/spark-12d5ff0a-2793-4b76-8a0b-d977a5924925/spark-7ad9382d-05cf-49d4-9a52-d42e6ca7117d/blockmgr-b72d4068-d065-47e6-8a10-867f723000db/15/shuffle_0_1_0.data, offset=26501223, length=6227612}} to /10.0.0.5:41168; closing connection
  458. java.io.IOException: Resource temporarily unavailable
  459. at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
  460. at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:415)
  461. at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:516)
  462. at org.apache.spark.network.buffer.LazyFileRegion.transferTo(LazyFileRegion.java:96)
  463. at org.apache.spark.network.protocol.MessageWithHeader.transferTo(MessageWithHeader.java:89)
  464. at io.netty.channel.socket.nio.NioSocketChannel.doWriteFileRegion(NioSocketChannel.java:237)
  465. at io.netty.channel.nio.AbstractNioByteChannel.doWrite(AbstractNioByteChannel.java:233)
  466. at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:264)
  467. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:707)
  468. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:315)
  469. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:676)
  470. at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1059)
  471. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  472. at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:669)
  473. at io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115)
  474. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  475. at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:718)
  476. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:706)
  477. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:741)
  478. at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:895)
  479. at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:240)
  480. at org.apache.spark.network.server.TransportRequestHandler.respond(TransportRequestHandler.java:147)
  481. at org.apache.spark.network.server.TransportRequestHandler.processFetchRequest(TransportRequestHandler.java:119)
  482. at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:95)
  483. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:91)
  484. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:44)
  485. at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
  486. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  487. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  488. at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
  489. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  490. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  491. at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
  492. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  493. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  494. at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
  495. at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
  496. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
  497. at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
  498. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
  499. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
  500. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  501. at java.lang.Thread.run(Thread.java:619)
  502. 15/04/21 14:59:33 ERROR TransportRequestHandler: Error sending result ChunkFetchSuccess{streamChunkId=StreamChunkId{streamId=1601401593010, chunkIndex=0}, buffer=FileSegmentManagedBuffer{file=/tmp/spark-0f8d0598-b137-4d14-993a-568b2ab3709a/spark-12d5ff0a-2793-4b76-8a0b-d977a5924925/spark-7ad9382d-05cf-49d4-9a52-d42e6ca7117d/blockmgr-b72d4068-d065-47e6-8a10-867f723000db/15/shuffle_0_1_0.data, offset=18488254, length=8012969}} to /10.0.0.3:41355; closing connection
  503. java.io.IOException: Resource temporarily unavailable
  504. at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
  505. at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:415)
  506. at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:516)
  507. at org.apache.spark.network.buffer.LazyFileRegion.transferTo(LazyFileRegion.java:96)
  508. at org.apache.spark.network.protocol.MessageWithHeader.transferTo(MessageWithHeader.java:89)
  509. at io.netty.channel.socket.nio.NioSocketChannel.doWriteFileRegion(NioSocketChannel.java:237)
  510. at io.netty.channel.nio.AbstractNioByteChannel.doWrite(AbstractNioByteChannel.java:233)
  511. at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:264)
  512. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:707)
  513. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:315)
  514. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:676)
  515. at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1059)
  516. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  517. at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:669)
  518. at io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115)
  519. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  520. at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:718)
  521. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:706)
  522. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:741)
  523. at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:895)
  524. at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:240)
  525. at org.apache.spark.network.server.TransportRequestHandler.respond(TransportRequestHandler.java:147)
  526. at org.apache.spark.network.server.TransportRequestHandler.processFetchRequest(TransportRequestHandler.java:119)
  527. at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:95)
  528. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:91)
  529. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:44)
  530. at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
  531. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  532. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  533. at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
  534. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  535. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  536. at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
  537. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  538. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  539. at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
  540. at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
  541. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
  542. at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
  543. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
  544. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
  545. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  546. at java.lang.Thread.run(Thread.java:619)
  547. 15/04/21 14:59:33 ERROR TransportRequestHandler: Error sending result ChunkFetchSuccess{streamChunkId=StreamChunkId{streamId=1601401593008, chunkIndex=0}, buffer=FileSegmentManagedBuffer{file=/tmp/spark-0f8d0598-b137-4d14-993a-568b2ab3709a/spark-12d5ff0a-2793-4b76-8a0b-d977a5924925/spark-7ad9382d-05cf-49d4-9a52-d42e6ca7117d/blockmgr-b72d4068-d065-47e6-8a10-867f723000db/15/shuffle_0_1_0.data, offset=0, length=10993212}} to /10.0.0.6:42433; closing connection
  548. java.io.IOException: Resource temporarily unavailable
  549. at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
  550. at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:415)
  551. at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:516)
  552. at org.apache.spark.network.buffer.LazyFileRegion.transferTo(LazyFileRegion.java:96)
  553. at org.apache.spark.network.protocol.MessageWithHeader.transferTo(MessageWithHeader.java:89)
  554. at io.netty.channel.socket.nio.NioSocketChannel.doWriteFileRegion(NioSocketChannel.java:237)
  555. at io.netty.channel.nio.AbstractNioByteChannel.doWrite(AbstractNioByteChannel.java:233)
  556. at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:264)
  557. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:707)
  558. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:315)
  559. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:676)
  560. at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1059)
  561. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  562. at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:669)
  563. at io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115)
  564. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  565. at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:718)
  566. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:706)
  567. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:741)
  568. at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:895)
  569. at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:240)
  570. at org.apache.spark.network.server.TransportRequestHandler.respond(TransportRequestHandler.java:147)
  571. at org.apache.spark.network.server.TransportRequestHandler.processFetchRequest(TransportRequestHandler.java:119)
  572. at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:95)
  573. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:91)
  574. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:44)
  575. at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
  576. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  577. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  578. at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
  579. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  580. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  581. at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
  582. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  583. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  584. at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
  585. at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
  586. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
  587. at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
  588. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
  589. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
  590. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  591. at java.lang.Thread.run(Thread.java:619)
  592. 15/04/21 14:59:33 ERROR TransportRequestHandler: Error sending result ChunkFetchSuccess{streamChunkId=StreamChunkId{streamId=1601401593010, chunkIndex=1}, buffer=FileSegmentManagedBuffer{file=/tmp/spark-0f8d0598-b137-4d14-993a-568b2ab3709a/spark-12d5ff0a-2793-4b76-8a0b-d977a5924925/spark-7ad9382d-05cf-49d4-9a52-d42e6ca7117d/blockmgr-b72d4068-d065-47e6-8a10-867f723000db/27/shuffle_0_5_0.data, offset=2204387, length=1588600}} to /10.0.0.3:41355; closing connection
  593. java.nio.channels.ClosedChannelException
  594. 15/04/21 14:59:33 ERROR TransportResponseHandler: Still have 1 requests outstanding when connection from node3.someuniversity.edu/10.0.0.3:55531 is closed
  595. 15/04/21 14:59:33 INFO RetryingBlockFetcher: Retrying fetch (2/3) for 1 outstanding blocks after 5000 ms
  596. 15/04/21 14:59:34 ERROR TransportRequestHandler: Error sending result ChunkFetchSuccess{streamChunkId=StreamChunkId{streamId=1601401593011, chunkIndex=0}, buffer=FileSegmentManagedBuffer{file=/tmp/spark-0f8d0598-b137-4d14-993a-568b2ab3709a/spark-12d5ff0a-2793-4b76-8a0b-d977a5924925/spark-7ad9382d-05cf-49d4-9a52-d42e6ca7117d/blockmgr-b72d4068-d065-47e6-8a10-867f723000db/15/shuffle_0_1_0.data, offset=0, length=10993212}} to /10.0.0.3:41357; closing connection
  597. java.io.IOException: Resource temporarily unavailable
  598. at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
  599. at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:415)
  600. at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:516)
  601. at org.apache.spark.network.buffer.LazyFileRegion.transferTo(LazyFileRegion.java:96)
  602. at org.apache.spark.network.protocol.MessageWithHeader.transferTo(MessageWithHeader.java:89)
  603. at io.netty.channel.socket.nio.NioSocketChannel.doWriteFileRegion(NioSocketChannel.java:237)
  604. at io.netty.channel.nio.AbstractNioByteChannel.doWrite(AbstractNioByteChannel.java:233)
  605. at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:264)
  606. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:707)
  607. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:315)
  608. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:676)
  609. at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1059)
  610. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  611. at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:669)
  612. at io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115)
  613. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  614. at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:718)
  615. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:706)
  616. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:741)
  617. at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:895)
  618. at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:240)
  619. at org.apache.spark.network.server.TransportRequestHandler.respond(TransportRequestHandler.java:147)
  620. at org.apache.spark.network.server.TransportRequestHandler.processFetchRequest(TransportRequestHandler.java:119)
  621. at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:95)
  622. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:91)
  623. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:44)
  624. at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
  625. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  626. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  627. at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
  628. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  629. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  630. at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
  631. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  632. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  633. at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
  634. at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
  635. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
  636. at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
  637. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
  638. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
  639. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  640. at java.lang.Thread.run(Thread.java:619)
  641. 15/04/21 14:59:34 INFO TransportClientFactory: Found inactive connection to node3.someuniversity.edu/10.0.0.3:55531, creating a new one.
  642. 15/04/21 14:59:34 ERROR TransportResponseHandler: Still have 1 requests outstanding when connection from node3.someuniversity.edu/10.0.0.3:55531 is closed
  643. 15/04/21 14:59:34 INFO RetryingBlockFetcher: Retrying fetch (2/3) for 1 outstanding blocks after 5000 ms
  644. 15/04/21 14:59:34 INFO TransportClientFactory: Found inactive connection to node6.someuniversity.edu/10.0.0.6:42828, creating a new one.
  645. 15/04/21 14:59:34 ERROR TransportResponseHandler: Still have 1 requests outstanding when connection from node6.someuniversity.edu/10.0.0.6:42828 is closed
  646. 15/04/21 14:59:34 INFO RetryingBlockFetcher: Retrying fetch (2/3) for 1 outstanding blocks after 5000 ms
  647. 15/04/21 14:59:34 ERROR TransportRequestHandler: Error sending result ChunkFetchSuccess{streamChunkId=StreamChunkId{streamId=1601401593012, chunkIndex=0}, buffer=FileSegmentManagedBuffer{file=/tmp/spark-0f8d0598-b137-4d14-993a-568b2ab3709a/spark-12d5ff0a-2793-4b76-8a0b-d977a5924925/spark-7ad9382d-05cf-49d4-9a52-d42e6ca7117d/blockmgr-b72d4068-d065-47e6-8a10-867f723000db/15/shuffle_0_1_0.data, offset=10993212, length=7495042}} to /10.0.0.5:41171; closing connection
  648. java.io.IOException: Resource temporarily unavailable
  649. at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
  650. at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:415)
  651. at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:516)
  652. at org.apache.spark.network.buffer.LazyFileRegion.transferTo(LazyFileRegion.java:96)
  653. at org.apache.spark.network.protocol.MessageWithHeader.transferTo(MessageWithHeader.java:89)
  654. at io.netty.channel.socket.nio.NioSocketChannel.doWriteFileRegion(NioSocketChannel.java:237)
  655. at io.netty.channel.nio.AbstractNioByteChannel.doWrite(AbstractNioByteChannel.java:233)
  656. at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:264)
  657. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:707)
  658. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:315)
  659. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:676)
  660. at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1059)
  661. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  662. at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:669)
  663. at io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115)
  664. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  665. at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:718)
  666. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:706)
  667. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:741)
  668. at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:895)
  669. at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:240)
  670. at org.apache.spark.network.server.TransportRequestHandler.respond(TransportRequestHandler.java:147)
  671. at org.apache.spark.network.server.TransportRequestHandler.processFetchRequest(TransportRequestHandler.java:119)
  672. at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:95)
  673. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:91)
  674. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:44)
  675. at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
  676. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  677. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  678. at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
  679. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  680. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  681. at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
  682. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  683. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  684. at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
  685. at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
  686. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
  687. at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
  688. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
  689. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
  690. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  691. at java.lang.Thread.run(Thread.java:619)
  692. 15/04/21 14:59:34 INFO TransportClientFactory: Found inactive connection to node5.someuniversity.edu/10.0.0.5:60089, creating a new one.
  693. 15/04/21 14:59:34 ERROR TransportResponseHandler: Still have 2 requests outstanding when connection from node5.someuniversity.edu/10.0.0.5:60089 is closed
  694. 15/04/21 14:59:34 INFO RetryingBlockFetcher: Retrying fetch (2/3) for 2 outstanding blocks after 5000 ms
  695. 15/04/21 14:59:34 ERROR TransportRequestHandler: Error sending result ChunkFetchSuccess{streamChunkId=StreamChunkId{streamId=1601401593013, chunkIndex=0}, buffer=FileSegmentManagedBuffer{file=/tmp/spark-0f8d0598-b137-4d14-993a-568b2ab3709a/spark-12d5ff0a-2793-4b76-8a0b-d977a5924925/spark-7ad9382d-05cf-49d4-9a52-d42e6ca7117d/blockmgr-b72d4068-d065-47e6-8a10-867f723000db/15/shuffle_0_1_0.data, offset=26501223, length=6227612}} to /10.0.0.6:42435; closing connection
  696. java.io.IOException: Resource temporarily unavailable
  697. at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
  698. at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:415)
  699. at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:516)
  700. at org.apache.spark.network.buffer.LazyFileRegion.transferTo(LazyFileRegion.java:96)
  701. at org.apache.spark.network.protocol.MessageWithHeader.transferTo(MessageWithHeader.java:89)
  702. at io.netty.channel.socket.nio.NioSocketChannel.doWriteFileRegion(NioSocketChannel.java:237)
  703. at io.netty.channel.nio.AbstractNioByteChannel.doWrite(AbstractNioByteChannel.java:233)
  704. at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:264)
  705. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:707)
  706. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:315)
  707. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:676)
  708. at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1059)
  709. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  710. at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:669)
  711. at io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115)
  712. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  713. at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:718)
  714. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:706)
  715. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:741)
  716. at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:895)
  717. at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:240)
  718. at org.apache.spark.network.server.TransportRequestHandler.respond(TransportRequestHandler.java:147)
  719. at org.apache.spark.network.server.TransportRequestHandler.processFetchRequest(TransportRequestHandler.java:119)
  720. at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:95)
  721. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:91)
  722. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:44)
  723. at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
  724. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  725. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  726. at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
  727. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  728. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  729. at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
  730. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  731. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  732. at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
  733. at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
  734. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
  735. at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
  736. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
  737. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
  738. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  739. at java.lang.Thread.run(Thread.java:619)
  740. 15/04/21 14:59:38 INFO TransportClientFactory: Found inactive connection to node5.someuniversity.edu/10.0.0.5:60089, creating a new one.
  741. 15/04/21 14:59:38 WARN TransportChannelHandler: Exception in connection from node5.someuniversity.edu/10.0.0.5:60089
  742. java.io.IOException: Connection reset by peer
  743. at sun.nio.ch.FileDispatcher.read0(Native Method)
  744. at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
  745. at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
  746. at sun.nio.ch.IOUtil.read(IOUtil.java:206)
  747. at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
  748. at io.netty.buffer.PooledHeapByteBuf.setBytes(PooledHeapByteBuf.java:234)
  749. at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:881)
  750. at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:225)
  751. at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)
  752. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
  753. at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
  754. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
  755. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
  756. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  757. at java.lang.Thread.run(Thread.java:619)
  758. 15/04/21 14:59:38 ERROR TransportResponseHandler: Still have 2 requests outstanding when connection from node5.someuniversity.edu/10.0.0.5:60089 is closed
  759. 15/04/21 14:59:38 INFO RetryingBlockFetcher: Retrying fetch (3/3) for 2 outstanding blocks after 5000 ms
  760. 15/04/21 14:59:38 INFO TransportClientFactory: Found inactive connection to node6.someuniversity.edu/10.0.0.6:42828, creating a new one.
  761. 15/04/21 14:59:38 ERROR TransportResponseHandler: Still have 1 requests outstanding when connection from node6.someuniversity.edu/10.0.0.6:42828 is closed
  762. 15/04/21 14:59:38 INFO RetryingBlockFetcher: Retrying fetch (3/3) for 1 outstanding blocks after 5000 ms
  763. 15/04/21 14:59:38 INFO TransportClientFactory: Found inactive connection to node3.someuniversity.edu/10.0.0.3:55531, creating a new one.
  764. 15/04/21 14:59:38 ERROR TransportRequestHandler: Error sending result ChunkFetchSuccess{streamChunkId=StreamChunkId{streamId=1601401593015, chunkIndex=0}, buffer=FileSegmentManagedBuffer{file=/tmp/spark-0f8d0598-b137-4d14-993a-568b2ab3709a/spark-12d5ff0a-2793-4b76-8a0b-d977a5924925/spark-7ad9382d-05cf-49d4-9a52-d42e6ca7117d/blockmgr-b72d4068-d065-47e6-8a10-867f723000db/15/shuffle_0_1_0.data, offset=0, length=10993212}} to /10.0.0.6:42439; closing connection
  765. java.io.IOException: Resource temporarily unavailable
  766. at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
  767. at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:415)
  768. at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:516)
  769. at org.apache.spark.network.buffer.LazyFileRegion.transferTo(LazyFileRegion.java:96)
  770. at org.apache.spark.network.protocol.MessageWithHeader.transferTo(MessageWithHeader.java:89)
  771. at io.netty.channel.socket.nio.NioSocketChannel.doWriteFileRegion(NioSocketChannel.java:237)
  772. at io.netty.channel.nio.AbstractNioByteChannel.doWrite(AbstractNioByteChannel.java:233)
  773. at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:264)
  774. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:707)
  775. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:315)
  776. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:676)
  777. at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1059)
  778. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  779. at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:669)
  780. at io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115)
  781. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  782. at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:718)
  783. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:706)
  784. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:741)
  785. at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:895)
  786. at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:240)
  787. at org.apache.spark.network.server.TransportRequestHandler.respond(TransportRequestHandler.java:147)
  788. at org.apache.spark.network.server.TransportRequestHandler.processFetchRequest(TransportRequestHandler.java:119)
  789. at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:95)
  790. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:91)
  791. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:44)
  792. at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
  793. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  794. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  795. at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
  796. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  797. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  798. at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
  799. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  800. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  801. at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
  802. at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
  803. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
  804. at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
  805. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
  806. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
  807. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  808. at java.lang.Thread.run(Thread.java:619)
  809. 15/04/21 14:59:38 ERROR TransportRequestHandler: Error sending result ChunkFetchSuccess{streamChunkId=StreamChunkId{streamId=1601401593016, chunkIndex=0}, buffer=FileSegmentManagedBuffer{file=/tmp/spark-0f8d0598-b137-4d14-993a-568b2ab3709a/spark-12d5ff0a-2793-4b76-8a0b-d977a5924925/spark-7ad9382d-05cf-49d4-9a52-d42e6ca7117d/blockmgr-b72d4068-d065-47e6-8a10-867f723000db/15/shuffle_0_1_0.data, offset=26501223, length=6227612}} to /10.0.0.5:41174; closing connection
  810. java.io.IOException: Resource temporarily unavailable
  811. at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
  812. at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:415)
  813. at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:516)
  814. at org.apache.spark.network.buffer.LazyFileRegion.transferTo(LazyFileRegion.java:96)
  815. at org.apache.spark.network.protocol.MessageWithHeader.transferTo(MessageWithHeader.java:89)
  816. at io.netty.channel.socket.nio.NioSocketChannel.doWriteFileRegion(NioSocketChannel.java:237)
  817. at io.netty.channel.nio.AbstractNioByteChannel.doWrite(AbstractNioByteChannel.java:233)
  818. at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:264)
  819. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:707)
  820. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:315)
  821. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:676)
  822. at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1059)
  823. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  824. at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:669)
  825. at io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115)
  826. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  827. at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:718)
  828. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:706)
  829. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:741)
  830. at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:895)
  831. at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:240)
  832. at org.apache.spark.network.server.TransportRequestHandler.respond(TransportRequestHandler.java:147)
  833. at org.apache.spark.network.server.TransportRequestHandler.processFetchRequest(TransportRequestHandler.java:119)
  834. at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:95)
  835. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:91)
  836. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:44)
  837. at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
  838. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  839. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  840. at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
  841. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  842. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  843. at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
  844. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  845. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  846. at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
  847. at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
  848. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
  849. at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
  850. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
  851. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
  852. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  853. at java.lang.Thread.run(Thread.java:619)
  854. 15/04/21 14:59:38 ERROR TransportRequestHandler: Error sending result ChunkFetchSuccess{streamChunkId=StreamChunkId{streamId=1601401593014, chunkIndex=0}, buffer=FileSegmentManagedBuffer{file=/tmp/spark-0f8d0598-b137-4d14-993a-568b2ab3709a/spark-12d5ff0a-2793-4b76-8a0b-d977a5924925/spark-7ad9382d-05cf-49d4-9a52-d42e6ca7117d/blockmgr-b72d4068-d065-47e6-8a10-867f723000db/15/shuffle_0_1_0.data, offset=18488254, length=8012969}} to /10.0.0.3:41361; closing connection
  855. java.io.IOException: Resource temporarily unavailable
  856. at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
  857. at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:415)
  858. at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:516)
  859. at org.apache.spark.network.buffer.LazyFileRegion.transferTo(LazyFileRegion.java:96)
  860. at org.apache.spark.network.protocol.MessageWithHeader.transferTo(MessageWithHeader.java:89)
  861. at io.netty.channel.socket.nio.NioSocketChannel.doWriteFileRegion(NioSocketChannel.java:237)
  862. at io.netty.channel.nio.AbstractNioByteChannel.doWrite(AbstractNioByteChannel.java:233)
  863. at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:264)
  864. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:707)
  865. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:315)
  866. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:676)
  867. at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1059)
  868. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  869. at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:669)
  870. at io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115)
  871. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  872. at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:718)
  873. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:706)
  874. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:741)
  875. at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:895)
  876. at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:240)
  877. at org.apache.spark.network.server.TransportRequestHandler.respond(TransportRequestHandler.java:147)
  878. at org.apache.spark.network.server.TransportRequestHandler.processFetchRequest(TransportRequestHandler.java:119)
  879. at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:95)
  880. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:91)
  881. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:44)
  882. at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
  883. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  884. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  885. at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
  886. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  887. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  888. at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
  889. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  890. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  891. at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
  892. at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
  893. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
  894. at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
  895. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
  896. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
  897. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  898. at java.lang.Thread.run(Thread.java:619)
  899. 15/04/21 14:59:38 ERROR TransportRequestHandler: Error sending result ChunkFetchSuccess{streamChunkId=StreamChunkId{streamId=1601401593014, chunkIndex=1}, buffer=FileSegmentManagedBuffer{file=/tmp/spark-0f8d0598-b137-4d14-993a-568b2ab3709a/spark-12d5ff0a-2793-4b76-8a0b-d977a5924925/spark-7ad9382d-05cf-49d4-9a52-d42e6ca7117d/blockmgr-b72d4068-d065-47e6-8a10-867f723000db/27/shuffle_0_5_0.data, offset=2204387, length=1588600}} to /10.0.0.3:41361; closing connection
  900. java.nio.channels.ClosedChannelException
  901. 15/04/21 14:59:38 ERROR TransportResponseHandler: Still have 1 requests outstanding when connection from node3.someuniversity.edu/10.0.0.3:55531 is closed
  902. 15/04/21 14:59:38 INFO RetryingBlockFetcher: Retrying fetch (3/3) for 1 outstanding blocks after 5000 ms
  903. 15/04/21 14:59:39 ERROR TransportRequestHandler: Error sending result ChunkFetchSuccess{streamChunkId=StreamChunkId{streamId=1601401593017, chunkIndex=0}, buffer=FileSegmentManagedBuffer{file=/tmp/spark-0f8d0598-b137-4d14-993a-568b2ab3709a/spark-12d5ff0a-2793-4b76-8a0b-d977a5924925/spark-7ad9382d-05cf-49d4-9a52-d42e6ca7117d/blockmgr-b72d4068-d065-47e6-8a10-867f723000db/15/shuffle_0_1_0.data, offset=0, length=10993212}} to /10.0.0.3:41363; closing connection
  904. java.io.IOException: Resource temporarily unavailable
  905. at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
  906. at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:415)
  907. at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:516)
  908. at org.apache.spark.network.buffer.LazyFileRegion.transferTo(LazyFileRegion.java:96)
  909. at org.apache.spark.network.protocol.MessageWithHeader.transferTo(MessageWithHeader.java:89)
  910. at io.netty.channel.socket.nio.NioSocketChannel.doWriteFileRegion(NioSocketChannel.java:237)
  911. at io.netty.channel.nio.AbstractNioByteChannel.doWrite(AbstractNioByteChannel.java:233)
  912. at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:264)
  913. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:707)
  914. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:315)
  915. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:676)
  916. at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1059)
  917. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  918. at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:669)
  919. at io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115)
  920. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  921. at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:718)
  922. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:706)
  923. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:741)
  924. at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:895)
  925. at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:240)
  926. at org.apache.spark.network.server.TransportRequestHandler.respond(TransportRequestHandler.java:147)
  927. at org.apache.spark.network.server.TransportRequestHandler.processFetchRequest(TransportRequestHandler.java:119)
  928. at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:95)
  929. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:91)
  930. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:44)
  931. at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
  932. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  933. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  934. at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
  935. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  936. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  937. at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
  938. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  939. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  940. at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
  941. at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
  942. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
  943. at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
  944. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
  945. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
  946. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  947. at java.lang.Thread.run(Thread.java:619)
  948. 15/04/21 14:59:39 INFO TransportClientFactory: Found inactive connection to node3.someuniversity.edu/10.0.0.3:55531, creating a new one.
  949. 15/04/21 14:59:39 ERROR TransportResponseHandler: Still have 1 requests outstanding when connection from node3.someuniversity.edu/10.0.0.3:55531 is closed
  950. 15/04/21 14:59:39 INFO RetryingBlockFetcher: Retrying fetch (3/3) for 1 outstanding blocks after 5000 ms
  951. 15/04/21 14:59:39 INFO TransportClientFactory: Found inactive connection to node6.someuniversity.edu/10.0.0.6:42828, creating a new one.
  952. 15/04/21 14:59:39 ERROR TransportResponseHandler: Still have 1 requests outstanding when connection from node6.someuniversity.edu/10.0.0.6:42828 is closed
  953. 15/04/21 14:59:39 INFO RetryingBlockFetcher: Retrying fetch (3/3) for 1 outstanding blocks after 5000 ms
  954. 15/04/21 14:59:39 ERROR TransportRequestHandler: Error sending result ChunkFetchSuccess{streamChunkId=StreamChunkId{streamId=1601401593018, chunkIndex=0}, buffer=FileSegmentManagedBuffer{file=/tmp/spark-0f8d0598-b137-4d14-993a-568b2ab3709a/spark-12d5ff0a-2793-4b76-8a0b-d977a5924925/spark-7ad9382d-05cf-49d4-9a52-d42e6ca7117d/blockmgr-b72d4068-d065-47e6-8a10-867f723000db/15/shuffle_0_1_0.data, offset=10993212, length=7495042}} to /10.0.0.5:41177; closing connection
  955. java.io.IOException: Resource temporarily unavailable
  956. at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
  957. at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:415)
  958. at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:516)
  959. at org.apache.spark.network.buffer.LazyFileRegion.transferTo(LazyFileRegion.java:96)
  960. at org.apache.spark.network.protocol.MessageWithHeader.transferTo(MessageWithHeader.java:89)
  961. at io.netty.channel.socket.nio.NioSocketChannel.doWriteFileRegion(NioSocketChannel.java:237)
  962. at io.netty.channel.nio.AbstractNioByteChannel.doWrite(AbstractNioByteChannel.java:233)
  963. at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:264)
  964. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:707)
  965. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:315)
  966. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:676)
  967. at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1059)
  968. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  969. at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:669)
  970. at io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115)
  971. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  972. at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:718)
  973. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:706)
  974. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:741)
  975. at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:895)
  976. at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:240)
  977. at org.apache.spark.network.server.TransportRequestHandler.respond(TransportRequestHandler.java:147)
  978. at org.apache.spark.network.server.TransportRequestHandler.processFetchRequest(TransportRequestHandler.java:119)
  979. at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:95)
  980. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:91)
  981. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:44)
  982. at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
  983. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  984. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  985. at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
  986. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  987. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  988. at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
  989. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  990. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  991. at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
  992. at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
  993. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
  994. at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
  995. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
  996. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
  997. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  998. at java.lang.Thread.run(Thread.java:619)
  999. 15/04/21 14:59:39 INFO TransportClientFactory: Found inactive connection to node5.someuniversity.edu/10.0.0.5:60089, creating a new one.
  1000. 15/04/21 14:59:39 WARN TransportChannelHandler: Exception in connection from node5.someuniversity.edu/10.0.0.5:60089
  1001. java.io.IOException: Connection reset by peer
  1002. at sun.nio.ch.FileDispatcher.read0(Native Method)
  1003. at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
  1004. at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
  1005. at sun.nio.ch.IOUtil.read(IOUtil.java:206)
  1006. at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
  1007. at io.netty.buffer.PooledHeapByteBuf.setBytes(PooledHeapByteBuf.java:234)
  1008. at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:881)
  1009. at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:225)
  1010. at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)
  1011. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
  1012. at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
  1013. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
  1014. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
  1015. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  1016. at java.lang.Thread.run(Thread.java:619)
  1017. 15/04/21 14:59:39 ERROR TransportResponseHandler: Still have 2 requests outstanding when connection from node5.someuniversity.edu/10.0.0.5:60089 is closed
  1018. 15/04/21 14:59:39 INFO RetryingBlockFetcher: Retrying fetch (3/3) for 2 outstanding blocks after 5000 ms
  1019. 15/04/21 14:59:39 ERROR TransportRequestHandler: Error sending result ChunkFetchSuccess{streamChunkId=StreamChunkId{streamId=1601401593019, chunkIndex=0}, buffer=FileSegmentManagedBuffer{file=/tmp/spark-0f8d0598-b137-4d14-993a-568b2ab3709a/spark-12d5ff0a-2793-4b76-8a0b-d977a5924925/spark-7ad9382d-05cf-49d4-9a52-d42e6ca7117d/blockmgr-b72d4068-d065-47e6-8a10-867f723000db/15/shuffle_0_1_0.data, offset=26501223, length=6227612}} to /10.0.0.6:42442; closing connection
  1020. java.io.IOException: Resource temporarily unavailable
  1021. at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
  1022. at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:415)
  1023. at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:516)
  1024. at org.apache.spark.network.buffer.LazyFileRegion.transferTo(LazyFileRegion.java:96)
  1025. at org.apache.spark.network.protocol.MessageWithHeader.transferTo(MessageWithHeader.java:89)
  1026. at io.netty.channel.socket.nio.NioSocketChannel.doWriteFileRegion(NioSocketChannel.java:237)
  1027. at io.netty.channel.nio.AbstractNioByteChannel.doWrite(AbstractNioByteChannel.java:233)
  1028. at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:264)
  1029. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:707)
  1030. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:315)
  1031. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:676)
  1032. at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1059)
  1033. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  1034. at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:669)
  1035. at io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115)
  1036. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  1037. at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:718)
  1038. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:706)
  1039. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:741)
  1040. at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:895)
  1041. at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:240)
  1042. at org.apache.spark.network.server.TransportRequestHandler.respond(TransportRequestHandler.java:147)
  1043. at org.apache.spark.network.server.TransportRequestHandler.processFetchRequest(TransportRequestHandler.java:119)
  1044. at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:95)
  1045. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:91)
  1046. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:44)
  1047. at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
  1048. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  1049. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  1050. at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
  1051. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  1052. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  1053. at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
  1054. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  1055. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  1056. at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
  1057. at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
  1058. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
  1059. at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
  1060. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
  1061. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
  1062. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  1063. at java.lang.Thread.run(Thread.java:619)
  1064. 15/04/21 14:59:43 INFO TransportClientFactory: Found inactive connection to node5.someuniversity.edu/10.0.0.5:60089, creating a new one.
  1065. 15/04/21 14:59:43 WARN TransportChannelHandler: Exception in connection from node5.someuniversity.edu/10.0.0.5:60089
  1066. java.io.IOException: Connection reset by peer
  1067. at sun.nio.ch.FileDispatcher.read0(Native Method)
  1068. at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
  1069. at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
  1070. at sun.nio.ch.IOUtil.read(IOUtil.java:206)
  1071. at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
  1072. at io.netty.buffer.PooledHeapByteBuf.setBytes(PooledHeapByteBuf.java:234)
  1073. at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:881)
  1074. at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:225)
  1075. at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)
  1076. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
  1077. at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
  1078. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
  1079. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
  1080. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  1081. at java.lang.Thread.run(Thread.java:619)
  1082. 15/04/21 14:59:43 ERROR TransportResponseHandler: Still have 2 requests outstanding when connection from node5.someuniversity.edu/10.0.0.5:60089 is closed
  1083. 15/04/21 14:59:43 ERROR RetryingBlockFetcher: Failed to fetch block shuffle_0_4_1, and will not retry (3 retries)
  1084. java.io.IOException: Connection reset by peer
  1085. at sun.nio.ch.FileDispatcher.read0(Native Method)
  1086. at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
  1087. at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
  1088. at sun.nio.ch.IOUtil.read(IOUtil.java:206)
  1089. at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
  1090. at io.netty.buffer.PooledHeapByteBuf.setBytes(PooledHeapByteBuf.java:234)
  1091. at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:881)
  1092. at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:225)
  1093. at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)
  1094. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
  1095. at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
  1096. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
  1097. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
  1098. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  1099. at java.lang.Thread.run(Thread.java:619)
  1100. 15/04/21 14:59:43 ERROR ShuffleBlockFetcherIterator: Failed to get block(s) from node5.someuniversity.edu:60089
  1101. java.io.IOException: Connection reset by peer
  1102. at sun.nio.ch.FileDispatcher.read0(Native Method)
  1103. at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
  1104. at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
  1105. at sun.nio.ch.IOUtil.read(IOUtil.java:206)
  1106. at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
  1107. at io.netty.buffer.PooledHeapByteBuf.setBytes(PooledHeapByteBuf.java:234)
  1108. at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:881)
  1109. at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:225)
  1110. at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)
  1111. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
  1112. at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
  1113. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
  1114. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
  1115. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  1116. at java.lang.Thread.run(Thread.java:619)
  1117. 15/04/21 14:59:43 ERROR RetryingBlockFetcher: Failed to fetch block shuffle_0_0_1, and will not retry (3 retries)
  1118. java.io.IOException: Connection reset by peer
  1119. at sun.nio.ch.FileDispatcher.read0(Native Method)
  1120. at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
  1121. at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
  1122. at sun.nio.ch.IOUtil.read(IOUtil.java:206)
  1123. at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
  1124. at io.netty.buffer.PooledHeapByteBuf.setBytes(PooledHeapByteBuf.java:234)
  1125. at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:881)
  1126. at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:225)
  1127. at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)
  1128. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
  1129. at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
  1130. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
  1131. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
  1132. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  1133. at java.lang.Thread.run(Thread.java:619)
  1134. 15/04/21 14:59:43 ERROR ShuffleBlockFetcherIterator: Failed to get block(s) from node5.someuniversity.edu:60089
  1135. java.io.IOException: Connection reset by peer
  1136. at sun.nio.ch.FileDispatcher.read0(Native Method)
  1137. at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
  1138. at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
  1139. at sun.nio.ch.IOUtil.read(IOUtil.java:206)
  1140. at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
  1141. at io.netty.buffer.PooledHeapByteBuf.setBytes(PooledHeapByteBuf.java:234)
  1142. at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:881)
  1143. at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:225)
  1144. at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)
  1145. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
  1146. at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
  1147. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
  1148. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
  1149. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  1150. at java.lang.Thread.run(Thread.java:619)
  1151. 15/04/21 14:59:43 INFO TransportClientFactory: Found inactive connection to node6.someuniversity.edu/10.0.0.6:42828, creating a new one.
  1152. 15/04/21 14:59:43 ERROR TransportResponseHandler: Still have 1 requests outstanding when connection from node6.someuniversity.edu/10.0.0.6:42828 is closed
  1153. 15/04/21 14:59:43 ERROR RetryingBlockFetcher: Failed to fetch block shuffle_0_3_1, and will not retry (3 retries)
  1154. java.io.IOException: Connection from node6.someuniversity.edu/10.0.0.6:42828 closed
  1155. at org.apache.spark.network.client.TransportResponseHandler.channelUnregistered(TransportResponseHandler.java:98)
  1156. at org.apache.spark.network.server.TransportChannelHandler.channelUnregistered(TransportChannelHandler.java:81)
  1157. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:183)
  1158. at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:169)
  1159. at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53)
  1160. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:183)
  1161. at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:169)
  1162. at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53)
  1163. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:183)
  1164. at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:169)
  1165. at io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:738)
  1166. at io.netty.channel.AbstractChannel$AbstractUnsafe$6.run(AbstractChannel.java:606)
  1167. at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380)
  1168. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
  1169. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  1170. at java.lang.Thread.run(Thread.java:619)
  1171. 15/04/21 14:59:43 ERROR ShuffleBlockFetcherIterator: Failed to get block(s) from node6.someuniversity.edu:42828
  1172. java.io.IOException: Connection from node6.someuniversity.edu/10.0.0.6:42828 closed
  1173. at org.apache.spark.network.client.TransportResponseHandler.channelUnregistered(TransportResponseHandler.java:98)
  1174. at org.apache.spark.network.server.TransportChannelHandler.channelUnregistered(TransportChannelHandler.java:81)
  1175. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:183)
  1176. at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:169)
  1177. at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53)
  1178. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:183)
  1179. at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:169)
  1180. at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53)
  1181. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:183)
  1182. at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:169)
  1183. at io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:738)
  1184. at io.netty.channel.AbstractChannel$AbstractUnsafe$6.run(AbstractChannel.java:606)
  1185. at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380)
  1186. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
  1187. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  1188. at java.lang.Thread.run(Thread.java:619)
  1189. 15/04/21 14:59:43 INFO TransportClientFactory: Found inactive connection to node3.someuniversity.edu/10.0.0.3:55531, creating a new one.
  1190. 15/04/21 14:59:43 ERROR TransportRequestHandler: Error sending result ChunkFetchSuccess{streamChunkId=StreamChunkId{streamId=1601401593020, chunkIndex=0}, buffer=FileSegmentManagedBuffer{file=/tmp/spark-0f8d0598-b137-4d14-993a-568b2ab3709a/spark-12d5ff0a-2793-4b76-8a0b-d977a5924925/spark-7ad9382d-05cf-49d4-9a52-d42e6ca7117d/blockmgr-b72d4068-d065-47e6-8a10-867f723000db/15/shuffle_0_1_0.data, offset=26501223, length=6227612}} to /10.0.0.5:41180; closing connection
  1191. java.io.IOException: Resource temporarily unavailable
  1192. at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
  1193. at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:415)
  1194. at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:516)
  1195. at org.apache.spark.network.buffer.LazyFileRegion.transferTo(LazyFileRegion.java:96)
  1196. at org.apache.spark.network.protocol.MessageWithHeader.transferTo(MessageWithHeader.java:89)
  1197. at io.netty.channel.socket.nio.NioSocketChannel.doWriteFileRegion(NioSocketChannel.java:237)
  1198. at io.netty.channel.nio.AbstractNioByteChannel.doWrite(AbstractNioByteChannel.java:233)
  1199. at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:264)
  1200. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:707)
  1201. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:315)
  1202. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:676)
  1203. at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1059)
  1204. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  1205. at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:669)
  1206. at io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115)
  1207. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  1208. at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:718)
  1209. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:706)
  1210. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:741)
  1211. at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:895)
  1212. at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:240)
  1213. at org.apache.spark.network.server.TransportRequestHandler.respond(TransportRequestHandler.java:147)
  1214. at org.apache.spark.network.server.TransportRequestHandler.processFetchRequest(TransportRequestHandler.java:119)
  1215. at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:95)
  1216. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:91)
  1217. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:44)
  1218. at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
  1219. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  1220. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  1221. at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
  1222. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  1223. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  1224. at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
  1225. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  1226. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  1227. at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
  1228. at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
  1229. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
  1230. at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
  1231. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
  1232. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
  1233. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  1234. at java.lang.Thread.run(Thread.java:619)
  1235. 15/04/21 14:59:43 ERROR TransportRequestHandler: Error sending result ChunkFetchSuccess{streamChunkId=StreamChunkId{streamId=1601401593021, chunkIndex=0}, buffer=FileSegmentManagedBuffer{file=/tmp/spark-0f8d0598-b137-4d14-993a-568b2ab3709a/spark-12d5ff0a-2793-4b76-8a0b-d977a5924925/spark-7ad9382d-05cf-49d4-9a52-d42e6ca7117d/blockmgr-b72d4068-d065-47e6-8a10-867f723000db/15/shuffle_0_1_0.data, offset=0, length=10993212}} to /10.0.0.6:42445; closing connection
  1236. java.io.IOException: Resource temporarily unavailable
  1237. at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
  1238. at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:415)
  1239. at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:516)
  1240. at org.apache.spark.network.buffer.LazyFileRegion.transferTo(LazyFileRegion.java:96)
  1241. at org.apache.spark.network.protocol.MessageWithHeader.transferTo(MessageWithHeader.java:89)
  1242. at io.netty.channel.socket.nio.NioSocketChannel.doWriteFileRegion(NioSocketChannel.java:237)
  1243. at io.netty.channel.nio.AbstractNioByteChannel.doWrite(AbstractNioByteChannel.java:233)
  1244. at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:264)
  1245. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:707)
  1246. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:315)
  1247. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:676)
  1248. at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1059)
  1249. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  1250. at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:669)
  1251. at io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115)
  1252. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  1253. at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:718)
  1254. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:706)
  1255. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:741)
  1256. at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:895)
  1257. at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:240)
  1258. at org.apache.spark.network.server.TransportRequestHandler.respond(TransportRequestHandler.java:147)
  1259. at org.apache.spark.network.server.TransportRequestHandler.processFetchRequest(TransportRequestHandler.java:119)
  1260. at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:95)
  1261. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:91)
  1262. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:44)
  1263. at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
  1264. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  1265. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  1266. at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
  1267. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  1268. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  1269. at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
  1270. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  1271. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  1272. at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
  1273. at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
  1274. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
  1275. at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
  1276. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
  1277. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
  1278. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  1279. at java.lang.Thread.run(Thread.java:619)
  1280. 15/04/21 14:59:43 ERROR TransportRequestHandler: Error sending result ChunkFetchSuccess{streamChunkId=StreamChunkId{streamId=1601401593020, chunkIndex=1}, buffer=FileSegmentManagedBuffer{file=/tmp/spark-0f8d0598-b137-4d14-993a-568b2ab3709a/spark-12d5ff0a-2793-4b76-8a0b-d977a5924925/spark-7ad9382d-05cf-49d4-9a52-d42e6ca7117d/blockmgr-b72d4068-d065-47e6-8a10-867f723000db/27/shuffle_0_5_0.data, offset=3792987, length=2862285}} to /10.0.0.5:41180; closing connection
  1281. java.nio.channels.ClosedChannelException
  1282. 15/04/21 14:59:43 ERROR TransportRequestHandler: Error sending result ChunkFetchSuccess{streamChunkId=StreamChunkId{streamId=1601401593022, chunkIndex=0}, buffer=FileSegmentManagedBuffer{file=/tmp/spark-0f8d0598-b137-4d14-993a-568b2ab3709a/spark-12d5ff0a-2793-4b76-8a0b-d977a5924925/spark-7ad9382d-05cf-49d4-9a52-d42e6ca7117d/blockmgr-b72d4068-d065-47e6-8a10-867f723000db/15/shuffle_0_1_0.data, offset=18488254, length=8012969}} to /10.0.0.3:41367; closing connection
  1283. java.io.IOException: Resource temporarily unavailable
  1284. at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
  1285. at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:415)
  1286. at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:516)
  1287. at org.apache.spark.network.buffer.LazyFileRegion.transferTo(LazyFileRegion.java:96)
  1288. at org.apache.spark.network.protocol.MessageWithHeader.transferTo(MessageWithHeader.java:89)
  1289. at io.netty.channel.socket.nio.NioSocketChannel.doWriteFileRegion(NioSocketChannel.java:237)
  1290. at io.netty.channel.nio.AbstractNioByteChannel.doWrite(AbstractNioByteChannel.java:233)
  1291. at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:264)
  1292. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:707)
  1293. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:315)
  1294. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:676)
  1295. at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1059)
  1296. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  1297. at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:669)
  1298. at io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115)
  1299. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  1300. at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:718)
  1301. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:706)
  1302. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:741)
  1303. at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:895)
  1304. at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:240)
  1305. at org.apache.spark.network.server.TransportRequestHandler.respond(TransportRequestHandler.java:147)
  1306. at org.apache.spark.network.server.TransportRequestHandler.processFetchRequest(TransportRequestHandler.java:119)
  1307. at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:95)
  1308. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:91)
  1309. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:44)
  1310. at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
  1311. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  1312. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  1313. at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
  1314. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  1315. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  1316. at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
  1317. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  1318. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  1319. at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
  1320. at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
  1321. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
  1322. at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
  1323. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
  1324. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
  1325. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  1326. at java.lang.Thread.run(Thread.java:619)
  1327. 15/04/21 14:59:43 ERROR TransportResponseHandler: Still have 1 requests outstanding when connection from node3.someuniversity.edu/10.0.0.3:55531 is closed
  1328. 15/04/21 14:59:43 ERROR RetryingBlockFetcher: Failed to fetch block shuffle_0_2_1, and will not retry (3 retries)
  1329. java.io.IOException: Connection from node3.someuniversity.edu/10.0.0.3:55531 closed
  1330. at org.apache.spark.network.client.TransportResponseHandler.channelUnregistered(TransportResponseHandler.java:98)
  1331. at org.apache.spark.network.server.TransportChannelHandler.channelUnregistered(TransportChannelHandler.java:81)
  1332. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:183)
  1333. at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:169)
  1334. at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53)
  1335. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:183)
  1336. at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:169)
  1337. at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53)
  1338. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:183)
  1339. at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:169)
  1340. at io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:738)
  1341. at io.netty.channel.AbstractChannel$AbstractUnsafe$6.run(AbstractChannel.java:606)
  1342. at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380)
  1343. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
  1344. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  1345. at java.lang.Thread.run(Thread.java:619)
  1346. 15/04/21 14:59:43 ERROR ShuffleBlockFetcherIterator: Failed to get block(s) from node3.someuniversity.edu:55531
  1347. java.io.IOException: Connection from node3.someuniversity.edu/10.0.0.3:55531 closed
  1348. at org.apache.spark.network.client.TransportResponseHandler.channelUnregistered(TransportResponseHandler.java:98)
  1349. at org.apache.spark.network.server.TransportChannelHandler.channelUnregistered(TransportChannelHandler.java:81)
  1350. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:183)
  1351. at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:169)
  1352. at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53)
  1353. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:183)
  1354. at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:169)
  1355. at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53)
  1356. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:183)
  1357. at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:169)
  1358. at io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:738)
  1359. at io.netty.channel.AbstractChannel$AbstractUnsafe$6.run(AbstractChannel.java:606)
  1360. at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380)
  1361. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
  1362. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  1363. at java.lang.Thread.run(Thread.java:619)
  1364. 15/04/21 14:59:44 ERROR TransportRequestHandler: Error sending result ChunkFetchSuccess{streamChunkId=StreamChunkId{streamId=1601401593023, chunkIndex=0}, buffer=FileSegmentManagedBuffer{file=/tmp/spark-0f8d0598-b137-4d14-993a-568b2ab3709a/spark-12d5ff0a-2793-4b76-8a0b-d977a5924925/spark-7ad9382d-05cf-49d4-9a52-d42e6ca7117d/blockmgr-b72d4068-d065-47e6-8a10-867f723000db/15/shuffle_0_1_0.data, offset=0, length=10993212}} to /10.0.0.3:41369; closing connection
  1365. java.io.IOException: Resource temporarily unavailable
  1366. at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
  1367. at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:415)
  1368. at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:516)
  1369. at org.apache.spark.network.buffer.LazyFileRegion.transferTo(LazyFileRegion.java:96)
  1370. at org.apache.spark.network.protocol.MessageWithHeader.transferTo(MessageWithHeader.java:89)
  1371. at io.netty.channel.socket.nio.NioSocketChannel.doWriteFileRegion(NioSocketChannel.java:237)
  1372. at io.netty.channel.nio.AbstractNioByteChannel.doWrite(AbstractNioByteChannel.java:233)
  1373. at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:264)
  1374. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:707)
  1375. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:315)
  1376. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:676)
  1377. at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1059)
  1378. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  1379. at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:669)
  1380. at io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115)
  1381. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  1382. at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:718)
  1383. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:706)
  1384. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:741)
  1385. at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:895)
  1386. at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:240)
  1387. at org.apache.spark.network.server.TransportRequestHandler.respond(TransportRequestHandler.java:147)
  1388. at org.apache.spark.network.server.TransportRequestHandler.processFetchRequest(TransportRequestHandler.java:119)
  1389. at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:95)
  1390. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:91)
  1391. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:44)
  1392. at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
  1393. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  1394. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  1395. at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
  1396. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  1397. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  1398. at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
  1399. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  1400. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  1401. at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
  1402. at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
  1403. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
  1404. at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
  1405. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
  1406. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
  1407. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  1408. at java.lang.Thread.run(Thread.java:619)
  1409. 15/04/21 14:59:44 INFO TransportClientFactory: Found inactive connection to node3.someuniversity.edu/10.0.0.3:55531, creating a new one.
  1410. 15/04/21 14:59:44 ERROR TransportResponseHandler: Still have 1 requests outstanding when connection from node3.someuniversity.edu/10.0.0.3:55531 is closed
  1411. 15/04/21 14:59:44 ERROR RetryingBlockFetcher: Failed to fetch block shuffle_0_2_2, and will not retry (3 retries)
  1412. java.io.IOException: Connection from node3.someuniversity.edu/10.0.0.3:55531 closed
  1413. at org.apache.spark.network.client.TransportResponseHandler.channelUnregistered(TransportResponseHandler.java:98)
  1414. at org.apache.spark.network.server.TransportChannelHandler.channelUnregistered(TransportChannelHandler.java:81)
  1415. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:183)
  1416. at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:169)
  1417. at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53)
  1418. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:183)
  1419. at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:169)
  1420. at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53)
  1421. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:183)
  1422. at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:169)
  1423. at io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:738)
  1424. at io.netty.channel.AbstractChannel$AbstractUnsafe$6.run(AbstractChannel.java:606)
  1425. at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380)
  1426. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
  1427. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  1428. at java.lang.Thread.run(Thread.java:619)
  1429. 15/04/21 14:59:44 ERROR ShuffleBlockFetcherIterator: Failed to get block(s) from node3.someuniversity.edu:55531
  1430. java.io.IOException: Connection from node3.someuniversity.edu/10.0.0.3:55531 closed
  1431. at org.apache.spark.network.client.TransportResponseHandler.channelUnregistered(TransportResponseHandler.java:98)
  1432. at org.apache.spark.network.server.TransportChannelHandler.channelUnregistered(TransportChannelHandler.java:81)
  1433. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:183)
  1434. at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:169)
  1435. at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53)
  1436. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:183)
  1437. at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:169)
  1438. at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53)
  1439. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:183)
  1440. at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:169)
  1441. at io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:738)
  1442. at io.netty.channel.AbstractChannel$AbstractUnsafe$6.run(AbstractChannel.java:606)
  1443. at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380)
  1444. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
  1445. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  1446. at java.lang.Thread.run(Thread.java:619)
  1447. 15/04/21 14:59:44 INFO TransportClientFactory: Found inactive connection to node6.someuniversity.edu/10.0.0.6:42828, creating a new one.
  1448. 15/04/21 14:59:44 ERROR TransportResponseHandler: Still have 1 requests outstanding when connection from node6.someuniversity.edu/10.0.0.6:42828 is closed
  1449. 15/04/21 14:59:44 ERROR RetryingBlockFetcher: Failed to fetch block shuffle_0_3_2, and will not retry (3 retries)
  1450. java.io.IOException: Connection from node6.someuniversity.edu/10.0.0.6:42828 closed
  1451. at org.apache.spark.network.client.TransportResponseHandler.channelUnregistered(TransportResponseHandler.java:98)
  1452. at org.apache.spark.network.server.TransportChannelHandler.channelUnregistered(TransportChannelHandler.java:81)
  1453. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:183)
  1454. at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:169)
  1455. at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53)
  1456. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:183)
  1457. at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:169)
  1458. at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53)
  1459. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:183)
  1460. at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:169)
  1461. at io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:738)
  1462. at io.netty.channel.AbstractChannel$AbstractUnsafe$6.run(AbstractChannel.java:606)
  1463. at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380)
  1464. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
  1465. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  1466. at java.lang.Thread.run(Thread.java:619)
  1467. 15/04/21 14:59:44 ERROR ShuffleBlockFetcherIterator: Failed to get block(s) from node6.someuniversity.edu:42828
  1468. java.io.IOException: Connection from node6.someuniversity.edu/10.0.0.6:42828 closed
  1469. at org.apache.spark.network.client.TransportResponseHandler.channelUnregistered(TransportResponseHandler.java:98)
  1470. at org.apache.spark.network.server.TransportChannelHandler.channelUnregistered(TransportChannelHandler.java:81)
  1471. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:183)
  1472. at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:169)
  1473. at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53)
  1474. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:183)
  1475. at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:169)
  1476. at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53)
  1477. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:183)
  1478. at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:169)
  1479. at io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:738)
  1480. at io.netty.channel.AbstractChannel$AbstractUnsafe$6.run(AbstractChannel.java:606)
  1481. at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380)
  1482. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
  1483. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  1484. at java.lang.Thread.run(Thread.java:619)
  1485. 15/04/21 14:59:44 ERROR TransportRequestHandler: Error sending result ChunkFetchSuccess{streamChunkId=StreamChunkId{streamId=1601401593024, chunkIndex=0}, buffer=FileSegmentManagedBuffer{file=/tmp/spark-0f8d0598-b137-4d14-993a-568b2ab3709a/spark-12d5ff0a-2793-4b76-8a0b-d977a5924925/spark-7ad9382d-05cf-49d4-9a52-d42e6ca7117d/blockmgr-b72d4068-d065-47e6-8a10-867f723000db/15/shuffle_0_1_0.data, offset=10993212, length=7495042}} to /10.0.0.5:41183; closing connection
  1486. java.io.IOException: Resource temporarily unavailable
  1487. at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
  1488. at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:415)
  1489. at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:516)
  1490. at org.apache.spark.network.buffer.LazyFileRegion.transferTo(LazyFileRegion.java:96)
  1491. at org.apache.spark.network.protocol.MessageWithHeader.transferTo(MessageWithHeader.java:89)
  1492. at io.netty.channel.socket.nio.NioSocketChannel.doWriteFileRegion(NioSocketChannel.java:237)
  1493. at io.netty.channel.nio.AbstractNioByteChannel.doWrite(AbstractNioByteChannel.java:233)
  1494. at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:264)
  1495. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:707)
  1496. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:315)
  1497. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:676)
  1498. at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1059)
  1499. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  1500. at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:669)
  1501. at io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115)
  1502. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  1503. at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:718)
  1504. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:706)
  1505. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:741)
  1506. at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:895)
  1507. at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:240)
  1508. at org.apache.spark.network.server.TransportRequestHandler.respond(TransportRequestHandler.java:147)
  1509. at org.apache.spark.network.server.TransportRequestHandler.processFetchRequest(TransportRequestHandler.java:119)
  1510. at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:95)
  1511. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:91)
  1512. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:44)
  1513. at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
  1514. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  1515. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  1516. at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
  1517. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  1518. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  1519. at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
  1520. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  1521. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  1522. at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
  1523. at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
  1524. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
  1525. at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
  1526. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
  1527. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
  1528. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  1529. at java.lang.Thread.run(Thread.java:619)
  1530. 15/04/21 14:59:44 INFO TransportClientFactory: Found inactive connection to node5.someuniversity.edu/10.0.0.5:60089, creating a new one.
  1531. 15/04/21 14:59:44 ERROR TransportResponseHandler: Still have 2 requests outstanding when connection from node5.someuniversity.edu/10.0.0.5:60089 is closed
  1532. 15/04/21 14:59:44 ERROR RetryingBlockFetcher: Failed to fetch block shuffle_0_4_2, and will not retry (3 retries)
  1533. java.io.IOException: Connection from node5.someuniversity.edu/10.0.0.5:60089 closed
  1534. at org.apache.spark.network.client.TransportResponseHandler.channelUnregistered(TransportResponseHandler.java:98)
  1535. at org.apache.spark.network.server.TransportChannelHandler.channelUnregistered(TransportChannelHandler.java:81)
  1536. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:183)
  1537. at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:169)
  1538. at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53)
  1539. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:183)
  1540. at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:169)
  1541. at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53)
  1542. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:183)
  1543. at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:169)
  1544. at io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:738)
  1545. at io.netty.channel.AbstractChannel$AbstractUnsafe$6.run(AbstractChannel.java:606)
  1546. at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380)
  1547. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
  1548. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  1549. at java.lang.Thread.run(Thread.java:619)
  1550. 15/04/21 14:59:44 ERROR ShuffleBlockFetcherIterator: Failed to get block(s) from node5.someuniversity.edu:60089
  1551. java.io.IOException: Connection from node5.someuniversity.edu/10.0.0.5:60089 closed
  1552. at org.apache.spark.network.client.TransportResponseHandler.channelUnregistered(TransportResponseHandler.java:98)
  1553. at org.apache.spark.network.server.TransportChannelHandler.channelUnregistered(TransportChannelHandler.java:81)
  1554. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:183)
  1555. at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:169)
  1556. at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53)
  1557. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:183)
  1558. at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:169)
  1559. at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53)
  1560. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:183)
  1561. at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:169)
  1562. at io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:738)
  1563. at io.netty.channel.AbstractChannel$AbstractUnsafe$6.run(AbstractChannel.java:606)
  1564. at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380)
  1565. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
  1566. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  1567. at java.lang.Thread.run(Thread.java:619)
  1568. 15/04/21 14:59:44 ERROR RetryingBlockFetcher: Failed to fetch block shuffle_0_0_2, and will not retry (3 retries)
  1569. java.io.IOException: Connection from node5.someuniversity.edu/10.0.0.5:60089 closed
  1570. at org.apache.spark.network.client.TransportResponseHandler.channelUnregistered(TransportResponseHandler.java:98)
  1571. at org.apache.spark.network.server.TransportChannelHandler.channelUnregistered(TransportChannelHandler.java:81)
  1572. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:183)
  1573. at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:169)
  1574. at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53)
  1575. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:183)
  1576. at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:169)
  1577. at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53)
  1578. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:183)
  1579. at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:169)
  1580. at io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:738)
  1581. at io.netty.channel.AbstractChannel$AbstractUnsafe$6.run(AbstractChannel.java:606)
  1582. at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380)
  1583. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
  1584. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  1585. at java.lang.Thread.run(Thread.java:619)
  1586. 15/04/21 14:59:44 ERROR ShuffleBlockFetcherIterator: Failed to get block(s) from node5.someuniversity.edu:60089
  1587. java.io.IOException: Connection from node5.someuniversity.edu/10.0.0.5:60089 closed
  1588. at org.apache.spark.network.client.TransportResponseHandler.channelUnregistered(TransportResponseHandler.java:98)
  1589. at org.apache.spark.network.server.TransportChannelHandler.channelUnregistered(TransportChannelHandler.java:81)
  1590. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:183)
  1591. at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:169)
  1592. at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53)
  1593. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:183)
  1594. at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:169)
  1595. at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53)
  1596. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:183)
  1597. at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:169)
  1598. at io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:738)
  1599. at io.netty.channel.AbstractChannel$AbstractUnsafe$6.run(AbstractChannel.java:606)
  1600. at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380)
  1601. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
  1602. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  1603. at java.lang.Thread.run(Thread.java:619)
  1604. 15/04/21 14:59:44 ERROR TransportRequestHandler: Error sending result ChunkFetchSuccess{streamChunkId=StreamChunkId{streamId=1601401593025, chunkIndex=0}, buffer=FileSegmentManagedBuffer{file=/tmp/spark-0f8d0598-b137-4d14-993a-568b2ab3709a/spark-12d5ff0a-2793-4b76-8a0b-d977a5924925/spark-7ad9382d-05cf-49d4-9a52-d42e6ca7117d/blockmgr-b72d4068-d065-47e6-8a10-867f723000db/15/shuffle_0_1_0.data, offset=26501223, length=6227612}} to /10.0.0.6:42447; closing connection
  1605. java.io.IOException: Resource temporarily unavailable
  1606. at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
  1607. at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:415)
  1608. at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:516)
  1609. at org.apache.spark.network.buffer.LazyFileRegion.transferTo(LazyFileRegion.java:96)
  1610. at org.apache.spark.network.protocol.MessageWithHeader.transferTo(MessageWithHeader.java:89)
  1611. at io.netty.channel.socket.nio.NioSocketChannel.doWriteFileRegion(NioSocketChannel.java:237)
  1612. at io.netty.channel.nio.AbstractNioByteChannel.doWrite(AbstractNioByteChannel.java:233)
  1613. at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:264)
  1614. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:707)
  1615. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:315)
  1616. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:676)
  1617. at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1059)
  1618. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  1619. at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:669)
  1620. at io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115)
  1621. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:688)
  1622. at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:718)
  1623. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:706)
  1624. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:741)
  1625. at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:895)
  1626. at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:240)
  1627. at org.apache.spark.network.server.TransportRequestHandler.respond(TransportRequestHandler.java:147)
  1628. at org.apache.spark.network.server.TransportRequestHandler.processFetchRequest(TransportRequestHandler.java:119)
  1629. at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:95)
  1630. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:91)
  1631. at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:44)
  1632. at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
  1633. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  1634. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  1635. at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
  1636. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  1637. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  1638. at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
  1639. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
  1640. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
  1641. at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
  1642. at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
  1643. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
  1644. at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
  1645. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
  1646. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
  1647. at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
  1648. at java.lang.Thread.run(Thread.java:619)
  1649. 15/04/21 14:59:44 ERROR TransportRequestHandler: Error sending result ChunkFetchSuccess{streamChunkId=StreamChunkId{streamId=1601401593025, chunkIndex=1}, buffer=FileSegmentManagedBuffer{file=/tmp/spark-0f8d0598-b137-4d14-993a-568b2ab3709a/spark-12d5ff0a-2793-4b76-8a0b-d977a5924925/spark-7ad9382d-05cf-49d4-9a52-d42e6ca7117d/blockmgr-b72d4068-d065-47e6-8a10-867f723000db/27/shuffle_0_5_0.data, offset=3792987, length=2862285}} to /10.0.0.6:42447; closing connection
  1650. java.nio.channels.ClosedChannelException
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!