API tools faq
paste
Advertisement
Guest User

elasticsearch failure node

a guest
Dec 12th, 2017
364
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 35.38 KB | None | 0 0
raw download clone embed print report
  1. [2017-12-11T12:02:01,689][INFO ][o.e.i.s.GlobalCheckpointSyncAction] [elastic-data-4] [dp-hits-hr.jutarnji-20171211][5] global checkpoint sync failed
  2. org.elasticsearch.transport.RemoteTransportException: [elastic-data-4][10.0.2.10:9300][indices:admin/seq_no/global_checkpoint_sync[p]]
  3. Caused by: java.io.IOException: Input/output error
  4. at sun.nio.ch.FileDispatcherImpl.force0(Native Method) ~[?:?]
  5. at sun.nio.ch.FileDispatcherImpl.force(FileDispatcherImpl.java:72) ~[?:?]
  6. at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:395) ~[?:?]
  7. at org.elasticsearch.index.translog.TranslogWriter.syncUpTo(TranslogWriter.java:350) ~[elasticsearch-6.0.0.jar:6.0.0]
  8. at org.elasticsearch.index.translog.TranslogWriter.sync(TranslogWriter.java:238) ~[elasticsearch-6.0.0.jar:6.0.0]
  9. at org.elasticsearch.index.translog.Translog.sync(Translog.java:670) ~[elasticsearch-6.0.0.jar:6.0.0]
  10. at org.elasticsearch.index.seqno.GlobalCheckpointSyncAction.shardOperationOnPrimary(GlobalCheckpointSyncAction.java:119) ~[elasticsearch-6.0.0.jar:6.0.0]
  11. at org.elasticsearch.index.seqno.GlobalCheckpointSyncAction.shardOperationOnPrimary(GlobalCheckpointSyncAction.java:51) ~[elasticsearch-6.0.0.jar:6.0.0]
  12. at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1033) ~[elasticsearch-6.0.0.jar:6.0.0]
  13. at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1011) ~[elasticsearch-6.0.0.jar:6.0.0]
  14. at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:104) ~[elasticsearch-6.0.0.jar:6.0.0]
  15. at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:358) [elasticsearch-6.0.0.jar:6.0.0]
  16. at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:298) [elasticsearch-6.0.0.jar:6.0.0]
  17. at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:974) [elasticsearch-6.0.0.jar:6.0.0]
  18. at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:971) [elasticsearch-6.0.0.jar:6.0.0]
  19. at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:238) [elasticsearch-6.0.0.jar:6.0.0]
  20. at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2188) [elasticsearch-6.0.0.jar:6.0.0]
  21. at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:983) [elasticsearch-6.0.0.jar:6.0.0]
  22. at org.elasticsearch.action.support.replication.TransportReplicationAction.access$500(TransportReplicationAction.java:97) [elasticsearch-6.0.0.jar:6.0.0]
  23. at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:319) [elasticsearch-6.0.0.jar:6.0.0]
  24. at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.0.0.jar:6.0.0]
  25. at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:294) [elasticsearch-6.0.0.jar:6.0.0]
  26. at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:281) [elasticsearch-6.0.0.jar:6.0.0]
  27. at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) [elasticsearch-6.0.0.jar:6.0.0]
  28. at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:659) [elasticsearch-6.0.0.jar:6.0.0]
  29. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-6.0.0.jar:6.0.0]
  30. at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.0.0.jar:6.0.0]
  31. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167) [?:?]
  32. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641) [?:?]
  33. at java.lang.Thread.run(Thread.java:844) [?:?]
  34. [2017-12-11T12:03:52,203][WARN ][o.e.m.j.JvmGcMonitorService] [elastic-data-4] [gc][young][248804][482174] duration [14.1s], collections [1]/[19.1s], total [14.1s]/[2.8h], memory [3.9gb]->[3.7gb]/[7.9gb], all_pools {
  35. [young] [265mb]->[11.7mb]/[266.2mb]}{[survivor] [27.2mb]->[28.4mb]/[33.2mb]}{[old] [3.6gb]->[3.6gb]/[7.6gb]}
  36. [2017-12-11T12:03:52,522][WARN ][o.e.m.j.JvmGcMonitorService] [elastic-data-4] [gc][248804] overhead, spent [14.1s] collecting in the last [19.1s]
  37. [2017-12-11T12:05:59,699][WARN ][o.e.m.j.JvmGcMonitorService] [elastic-data-4] [gc][young][248871][482175] duration [10.4s], collections [1]/[21.4s], total [10.4s]/[2.8h], memory [3.9gb]->[3.7gb]/[7.9gb], all_pools {
  38. [young] [263.3mb]->[29.9mb]/[266.2mb]}{[survivor] [28.4mb]->[20.4mb]/[33.2mb]}{[old] [3.6gb]->[3.6gb]/[7.6gb]}
  39. [2017-12-11T12:05:59,699][INFO ][o.e.m.j.JvmGcMonitorService] [elastic-data-4] [gc][248871] overhead, spent [10.4s] collecting in the last [21.4s]
  40. [2017-12-11T12:08:18,799][WARN ][o.e.m.j.JvmGcMonitorService] [elastic-data-4] [gc][young][248924][482176] duration [10.6s], collections [1]/[27.8s], total [10.6s]/[2.8h], memory [3.9gb]->[3.7gb]/[7.9gb], all_pools {
  41. [young] [265.6mb]->[48.9mb]/[266.2mb]}{[survivor] [20.4mb]->[25.4mb]/[33.2mb]}{[old] [3.6gb]->[3.6gb]/[7.6gb]}
  42. [2017-12-11T12:08:18,799][INFO ][o.e.m.j.JvmGcMonitorService] [elastic-data-4] [gc][248924] overhead, spent [10.6s] collecting in the last [27.8s]
  43. [2017-12-11T12:10:23,523][WARN ][o.e.m.j.JvmGcMonitorService] [elastic-data-4] [gc][young][248985][482177] duration [13.4s], collections [1]/[20.2s], total [13.4s]/[2.8h], memory [3.9gb]->[3.7gb]/[7.9gb], all_pools {
  44. [young] [265.8mb]->[21.3mb]/[266.2mb]}{[survivor] [25.4mb]->[24.8mb]/[33.2mb]}{[old] [3.6gb]->[3.6gb]/[7.6gb]}
  45. ...
  46.  
  47. [2017-12-11T15:37:46,543][WARN ][o.e.t.n.Netty4Transport ] [elastic-data-4] write and flush on the network layer failed (channel: [id: 0x48823c94, L:/10.0.2.10:9300 ! R:/10.0.2.2:38682])
  48. java.io.IOException: Connection reset by peer
  49. at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[?:?]
  50. at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[?:?]
  51. at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[?:?]
  52. at sun.nio.ch.IOUtil.write(IOUtil.java:51) ~[?:?]
  53. at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:473) ~[?:?]
  54. at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:417) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  55. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:856) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  56. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:362) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  57. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:823) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  58. at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1296) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  59. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  60. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  61. at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  62. at io.netty.handler.logging.LoggingHandler.flush(LoggingHandler.java:265) ~[netty-handler-4.1.13.Final.jar:4.1.13.Final]
  63. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  64. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  65. at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  66. at io.netty.channel.ChannelDuplexHandler.flush(ChannelDuplexHandler.java:117) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  67. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  68. at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  69. at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  70. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  71. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  72. at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1032) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  73. at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:296) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  74. at org.elasticsearch.transport.netty4.Netty4Transport.sendMessage(Netty4Transport.java:318) [transport-netty4-6.0.0.jar:6.0.0]
  75. at org.elasticsearch.transport.netty4.Netty4Transport.sendMessage(Netty4Transport.java:88) [transport-netty4-6.0.0.jar:6.0.0]
  76. at org.elasticsearch.transport.TcpTransport.internalSendMessage(TcpTransport.java:1110) [elasticsearch-6.0.0.jar:6.0.0]
  77. at org.elasticsearch.transport.TcpTransport.sendErrorResponse(TcpTransport.java:1143) [elasticsearch-6.0.0.jar:6.0.0]
  78. at org.elasticsearch.transport.TcpTransportChannel.sendResponse(TcpTransportChannel.java:76) [elasticsearch-6.0.0.jar:6.0.0]
  79. at org.elasticsearch.transport.DelegatingTransportChannel.sendResponse(DelegatingTransportChannel.java:70) [elasticsearch-6.0.0.jar:6.0.0]
  80. at org.elasticsearch.transport.RequestHandlerRegistry$TransportChannelWrapper.sendResponse(RequestHandlerRegistry.java:120) [elasticsearch-6.0.0.jar:6.0.0]
  81. at org.elasticsearch.action.support.replication.TransportReplicationAction$OperationTransportHandler$1.onFailure(TransportReplicationAction.java:264) [elasticsearch-6.0.0.jar:6.0.0]
  82. at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishAsFailed(TransportReplicationAction.java:908) [elasticsearch-6.0.0.jar:6.0.0]
  83. at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.handleException(TransportReplicationAction.java:864) [elasticsearch-6.0.0.jar:6.0.0]
  84. at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1069) [elasticsearch-6.0.0.jar:6.0.0]
  85. at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1173) [elasticsearch-6.0.0.jar:6.0.0]
  86. at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1151) [elasticsearch-6.0.0.jar:6.0.0]
  87. at org.elasticsearch.transport.TransportService$7.onFailure(TransportService.java:670) [elasticsearch-6.0.0.jar:6.0.0]
  88. at org.elasticsearch.common.util.concurrent.AbstractRunnable.onRejection(AbstractRunnable.java:63) [elasticsearch-6.0.0.jar:6.0.0]
  89. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onRejection(ThreadContext.java:628) [elasticsearch-6.0.0.jar:6.0.0]
  90. at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.doExecute(EsThreadPoolExecutor.java:100) [elasticsearch-6.0.0.jar:6.0.0]
  91. at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.execute(EsThreadPoolExecutor.java:89) [elasticsearch-6.0.0.jar:6.0.0]
  92. at org.elasticsearch.transport.TransportService.sendLocalRequest(TransportService.java:655) [elasticsearch-6.0.0.jar:6.0.0]
  93. at org.elasticsearch.transport.TransportService.access$000(TransportService.java:74) [elasticsearch-6.0.0.jar:6.0.0]
  94. at org.elasticsearch.transport.TransportService$3.sendRequest(TransportService.java:137) [elasticsearch-6.0.0.jar:6.0.0]
  95. at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:601) [elasticsearch-6.0.0.jar:6.0.0]
  96. at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:521) [elasticsearch-6.0.0.jar:6.0.0]
  97. at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:509) [elasticsearch-6.0.0.jar:6.0.0]
  98. at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.performAction(TransportReplicationAction.java:832) [elasticsearch-6.0.0.jar:6.0.0]
  99. at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.performLocalAction(TransportReplicationAction.java:750) [elasticsearch-6.0.0.jar:6.0.0]
  100. at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:738) [elasticsearch-6.0.0.jar:6.0.0]
  101. at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.0.0.jar:6.0.0]
  102. at org.elasticsearch.action.support.replication.TransportReplicationAction.doExecute(TransportReplicationAction.java:169) [elasticsearch-6.0.0.jar:6.0.0]
  103. at org.elasticsearch.action.support.replication.TransportReplicationAction.doExecute(TransportReplicationAction.java:97) [elasticsearch-6.0.0.jar:6.0.0]
  104. at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167) [elasticsearch-6.0.0.jar:6.0.0]
  105. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139) [elasticsearch-6.0.0.jar:6.0.0]
  106. at org.elasticsearch.action.support.replication.TransportReplicationAction$OperationTransportHandler.messageReceived(TransportReplicationAction.java:251) [elasticsearch-6.0.0.jar:6.0.0]
  107. at org.elasticsearch.action.support.replication.TransportReplicationAction$OperationTransportHandler.messageReceived(TransportReplicationAction.java:243) [elasticsearch-6.0.0.jar:6.0.0]
  108. at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) [elasticsearch-6.0.0.jar:6.0.0]
  109. at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1540) [elasticsearch-6.0.0.jar:6.0.0]
  110. at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.0.0.jar:6.0.0]
  111. at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:139) [elasticsearch-6.0.0.jar:6.0.0]
  112. at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1497) [elasticsearch-6.0.0.jar:6.0.0]
  113. at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1367) [elasticsearch-6.0.0.jar:6.0.0]
  114. at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:60) [transport-netty4-6.0.0.jar:6.0.0]
  115. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  116. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  117. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  118. at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.13.Final.jar:4.1.13.Final]
  119. at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297) [netty-codec-4.1.13.Final.jar:4.1.13.Final]
  120. at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413) [netty-codec-4.1.13.Final.jar:4.1.13.Final]
  121. at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) [netty-codec-4.1.13.Final.jar:4.1.13.Final]
  122. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  123. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  124. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  125. at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  126. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  127. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  128. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  129. at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) [netty-handler-4.1.13.Final.jar:4.1.13.Final]
  130. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  131. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  132. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  133. at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  134. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  135. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  136. at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  137. at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  138. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  139. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  140. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  141. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  142. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.13.Final.jar:4.1.13.Final]
  143. at java.lang.Thread.run(Thread.java:844) [?:?]
  144. [2017-12-11T15:37:51,402][WARN ][o.e.t.n.Netty4Transport ] [elastic-data-4] write and flush on the network layer failed (channel: [id: 0x27e3d0ad, L:/10.0.2.10:9300 ! R:/10.0.2.6:43980])
  145. java.io.IOException: Connection reset by peer
  146. at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[?:?]
  147. at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[?:?]
  148. at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[?:?]
  149. at sun.nio.ch.IOUtil.write(IOUtil.java:51) ~[?:?]
  150. at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:473) ~[?:?]
  151. at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:417) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  152. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:856) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  153. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:362) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  154. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:823) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  155. at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1296) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  156. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  157. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  158. at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  159. at io.netty.handler.logging.LoggingHandler.flush(LoggingHandler.java:265) ~[netty-handler-4.1.13.Final.jar:4.1.13.Final]
  160. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  161. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  162. at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  163. at io.netty.channel.ChannelDuplexHandler.flush(ChannelDuplexHandler.java:117) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  164. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  165. at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  166. at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  167. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  168. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  169. at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1032) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  170. at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:296) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  171. at org.elasticsearch.transport.netty4.Netty4Transport.sendMessage(Netty4Transport.java:318) [transport-netty4-6.0.0.jar:6.0.0]
  172. at org.elasticsearch.transport.netty4.Netty4Transport.sendMessage(Netty4Transport.java:88) [transport-netty4-6.0.0.jar:6.0.0]
  173. at org.elasticsearch.transport.TcpTransport.internalSendMessage(TcpTransport.java:1110) [elasticsearch-6.0.0.jar:6.0.0]
  174. at org.elasticsearch.transport.TcpTransport.sendErrorResponse(TcpTransport.java:1143) [elasticsearch-6.0.0.jar:6.0.0]
  175. at org.elasticsearch.transport.TcpTransportChannel.sendResponse(TcpTransportChannel.java:76) [elasticsearch-6.0.0.jar:6.0.0]
  176. at org.elasticsearch.transport.DelegatingTransportChannel.sendResponse(DelegatingTransportChannel.java:70) [elasticsearch-6.0.0.jar:6.0.0]
  177. at org.elasticsearch.transport.RequestHandlerRegistry$TransportChannelWrapper.sendResponse(RequestHandlerRegistry.java:120) [elasticsearch-6.0.0.jar:6.0.0]
  178. at org.elasticsearch.action.support.replication.TransportReplicationAction$OperationTransportHandler$1.onFailure(TransportReplicationAction.java:264) [elasticsearch-6.0.0.jar:6.0.0]
  179. at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishAsFailed(TransportReplicationAction.java:908) [elasticsearch-6.0.0.jar:6.0.0]
  180. at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.handleException(TransportReplicationAction.java:864) [elasticsearch-6.0.0.jar:6.0.0]
  181. at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1069) [elasticsearch-6.0.0.jar:6.0.0]
  182. at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1173) [elasticsearch-6.0.0.jar:6.0.0]
  183. at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1151) [elasticsearch-6.0.0.jar:6.0.0]
  184. at org.elasticsearch.transport.TransportService$7.onFailure(TransportService.java:670) [elasticsearch-6.0.0.jar:6.0.0]
  185. at org.elasticsearch.common.util.concurrent.AbstractRunnable.onRejection(AbstractRunnable.java:63) [elasticsearch-6.0.0.jar:6.0.0]
  186. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onRejection(ThreadContext.java:628) [elasticsearch-6.0.0.jar:6.0.0]
  187. at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.doExecute(EsThreadPoolExecutor.java:100) [elasticsearch-6.0.0.jar:6.0.0]
  188. at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.execute(EsThreadPoolExecutor.java:89) [elasticsearch-6.0.0.jar:6.0.0]
  189. at org.elasticsearch.transport.TransportService.sendLocalRequest(TransportService.java:655) [elasticsearch-6.0.0.jar:6.0.0]
  190. at org.elasticsearch.transport.TransportService.access$000(TransportService.java:74) [elasticsearch-6.0.0.jar:6.0.0]
  191. at org.elasticsearch.transport.TransportService$3.sendRequest(TransportService.java:137) [elasticsearch-6.0.0.jar:6.0.0]
  192. at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:601) [elasticsearch-6.0.0.jar:6.0.0]
  193. at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:521) [elasticsearch-6.0.0.jar:6.0.0]
  194. at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:509) [elasticsearch-6.0.0.jar:6.0.0]
  195. at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.performAction(TransportReplicationAction.java:832) [elasticsearch-6.0.0.jar:6.0.0]
  196. at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.performLocalAction(TransportReplicationAction.java:750) [elasticsearch-6.0.0.jar:6.0.0]
  197. at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:738) [elasticsearch-6.0.0.jar:6.0.0]
  198. at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.0.0.jar:6.0.0]
  199. at org.elasticsearch.action.support.replication.TransportReplicationAction.doExecute(TransportReplicationAction.java:169) [elasticsearch-6.0.0.jar:6.0.0]
  200. at org.elasticsearch.action.support.replication.TransportReplicationAction.doExecute(TransportReplicationAction.java:97) [elasticsearch-6.0.0.jar:6.0.0]
  201. at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167) [elasticsearch-6.0.0.jar:6.0.0]
  202. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139) [elasticsearch-6.0.0.jar:6.0.0]
  203. at org.elasticsearch.action.support.replication.TransportReplicationAction$OperationTransportHandler.messageReceived(TransportReplicationAction.java:251) [elasticsearch-6.0.0.jar:6.0.0]
  204. at org.elasticsearch.action.support.replication.TransportReplicationAction$OperationTransportHandler.messageReceived(TransportReplicationAction.java:243) [elasticsearch-6.0.0.jar:6.0.0]
  205. at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) [elasticsearch-6.0.0.jar:6.0.0]
  206. at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1540) [elasticsearch-6.0.0.jar:6.0.0]
  207. at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.0.0.jar:6.0.0]
  208. at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:139) [elasticsearch-6.0.0.jar:6.0.0]
  209. at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1497) [elasticsearch-6.0.0.jar:6.0.0]
  210. at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1367) [elasticsearch-6.0.0.jar:6.0.0]
  211. at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:60) [transport-netty4-6.0.0.jar:6.0.0]
  212. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  213. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  214. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  215. at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.13.Final.jar:4.1.13.Final]
  216. at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297) [netty-codec-4.1.13.Final.jar:4.1.13.Final]
  217. at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413) [netty-codec-4.1.13.Final.jar:4.1.13.Final]
  218. at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) [netty-codec-4.1.13.Final.jar:4.1.13.Final]
  219. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  220. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  221. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  222. at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  223. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  224. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  225. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  226. at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) [netty-handler-4.1.13.Final.jar:4.1.13.Final]
  227. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  228. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  229. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  230. at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  231. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  232. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  233. at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  234. at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  235. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  236. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  237. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  238. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  239. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.13.Final.jar:4.1.13.Final]
  240. at java.lang.Thread.run(Thread.java:844) [?:?]
  241. [2017-12-11T15:37:55,047][WARN ][o.e.t.n.Netty4Transport ] [elastic-data-4] write and flush on the network layer failed (channel: [id: 0x27e3d0ad, L:/10.0.2.10:9300 ! R:/10.0.2.6:43980])
  242. java.nio.channels.ClosedChannelException: null
  243. at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
  244. [2017-12-11T15:37:55,048][WARN ][o.e.t.n.Netty4Transport ] [elastic-data-4] write and flush on the network layer failed (channel: [id: 0x27e3d0ad, L:/10.0.2.10:9300 ! R:/10.0.2.6:43980])
  245. java.nio.channels.ClosedChannelException: null
  246. at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
  247. [2017-12-11T15:37:55,049][WARN ][o.e.t.n.Netty4Transport ] [elastic-data-4] write and flush on the network layer failed (channel: [id: 0x27e3d0ad, L:/10.0.2.10:9300 ! R:/10.0.2.6:43980])
  248. java.nio.channels.ClosedChannelException: null
  249. at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
  250. [2017-12-11T15:37:55,049][WARN ][o.e.t.n.Netty4Transport ] [elastic-data-4] write and flush on the network layer failed (channel: [id: 0x27e3d0ad, L:/10.0.2.10:9300 ! R:/10.0.2.6:43980])
  251. java.nio.channels.ClosedChannelException: null
  252. at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
  253. [2017-12-11T15:37:48,447][WARN ][o.e.t.n.Netty4Transport ] [elastic-data-4] write and flush on the network layer failed (channel: [id: 0x83eb7f7c, L:/10.0.2.10:9300 ! R:/10.0.2.6:43984])
  254. java.io.IOException: Connection reset by peer
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!