API tools faq
paste
Advertisement
Guest User

Untitled

a guest
Jan 25th, 2019
169
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 114.99 KB | None | 0 0
raw download clone embed print report
  1. [2019-01-25T10:42:42,212][INFO ][o.e.n.Node ] [] initializing ...
  2. [2019-01-25T10:42:42,318][INFO ][o.e.e.NodeEnvironment ] [yRj9xlW] using [1] data paths, mounts [[/home (/dev/mapper/centos_vanng--t570-home)]], net usable_space [145.6gb], net total_space [179.8gb], spins? [no], types [xfs]
  3. [2019-01-25T10:42:42,319][INFO ][o.e.e.NodeEnvironment ] [yRj9xlW] heap size [1.9gb], compressed ordinary object pointers [true]
  4. [2019-01-25T10:42:42,335][INFO ][o.e.n.Node ] node name [yRj9xlW] derived from node ID [yRj9xlW8Q-WgQ0j6H4DbOQ]; set [node.name] to override
  5. [2019-01-25T10:42:42,335][INFO ][o.e.n.Node ] version[5.6.9], pid[23706], build[877a590/2018-04-12T16:25:14.838Z], OS[Linux/3.10.0-862.3.3.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_181/25.181-b13]
  6. [2019-01-25T10:42:42,336][INFO ][o.e.n.Node ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/home/vanng/elasticsearch-5.6.9-1]
  7. [2019-01-25T10:42:43,018][INFO ][o.e.p.PluginsService ] [yRj9xlW] loaded module [aggs-matrix-stats]
  8. [2019-01-25T10:42:43,018][INFO ][o.e.p.PluginsService ] [yRj9xlW] loaded module [ingest-common]
  9. [2019-01-25T10:42:43,018][INFO ][o.e.p.PluginsService ] [yRj9xlW] loaded module [lang-expression]
  10. [2019-01-25T10:42:43,018][INFO ][o.e.p.PluginsService ] [yRj9xlW] loaded module [lang-groovy]
  11. [2019-01-25T10:42:43,018][INFO ][o.e.p.PluginsService ] [yRj9xlW] loaded module [lang-mustache]
  12. [2019-01-25T10:42:43,018][INFO ][o.e.p.PluginsService ] [yRj9xlW] loaded module [lang-painless]
  13. [2019-01-25T10:42:43,019][INFO ][o.e.p.PluginsService ] [yRj9xlW] loaded module [parent-join]
  14. [2019-01-25T10:42:43,019][INFO ][o.e.p.PluginsService ] [yRj9xlW] loaded module [percolator]
  15. [2019-01-25T10:42:43,019][INFO ][o.e.p.PluginsService ] [yRj9xlW] loaded module [reindex]
  16. [2019-01-25T10:42:43,019][INFO ][o.e.p.PluginsService ] [yRj9xlW] loaded module [transport-netty3]
  17. [2019-01-25T10:42:43,019][INFO ][o.e.p.PluginsService ] [yRj9xlW] loaded module [transport-netty4]
  18. [2019-01-25T10:42:43,020][INFO ][o.e.p.PluginsService ] [yRj9xlW] no plugins loaded
  19. [2019-01-25T10:42:44,295][INFO ][o.e.d.DiscoveryModule ] [yRj9xlW] using discovery type [zen]
  20. [2019-01-25T10:42:44,787][INFO ][o.e.n.Node ] initialized
  21. [2019-01-25T10:42:44,788][INFO ][o.e.n.Node ] [yRj9xlW] starting ...
  22. [2019-01-25T10:43:02,176][INFO ][o.e.t.TransportService ] [yRj9xlW] publish_address {10.231.83.238:9300}, bound_addresses {10.231.83.238:9300}
  23. [2019-01-25T10:43:02,188][INFO ][o.e.b.BootstrapChecks ] [yRj9xlW] bound or publishing to a non-loopback address, enforcing bootstrap checks
  24. [2019-01-25T10:43:05,348][INFO ][o.e.c.s.ClusterService ] [yRj9xlW] detected_master {lJgJwYh}{lJgJwYhmQA-fucaOBHD5Vg}{ho76tglrRPKjA8ui_UVJSg}{10.231.83.146}{10.231.83.146:9301}, added {{wAVZzky}{wAVZzkyrSfyBmuGRw0Bltw}{r1L7UXpESgiM_Kxq_jFUjw}{10.231.83.146}{10.231.83.146:9302},{lJgJwYh}{lJgJwYhmQA-fucaOBHD5Vg}{ho76tglrRPKjA8ui_UVJSg}{10.231.83.146}{10.231.83.146:9301},}, reason: zen-disco-receive(from master [master {lJgJwYh}{lJgJwYhmQA-fucaOBHD5Vg}{ho76tglrRPKjA8ui_UVJSg}{10.231.83.146}{10.231.83.146:9301} committed version [173]])
  25. [2019-01-25T10:43:05,454][INFO ][o.e.h.n.Netty4HttpServerTransport] [yRj9xlW] publish_address {10.231.83.238:9200}, bound_addresses {10.231.83.238:9200}
  26. [2019-01-25T10:43:05,454][INFO ][o.e.n.Node ] [yRj9xlW] started
  27. [2019-01-25T10:43:12,258][INFO ][o.e.d.z.ZenDiscovery ] [yRj9xlW] master_left [{lJgJwYh}{lJgJwYhmQA-fucaOBHD5Vg}{ho76tglrRPKjA8ui_UVJSg}{10.231.83.146}{10.231.83.146:9301}], reason [shut_down]
  28. [2019-01-25T10:43:12,260][WARN ][o.e.d.z.ZenDiscovery ] [yRj9xlW] master left (reason = shut_down), current nodes: nodes:
  29. {wAVZzky}{wAVZzkyrSfyBmuGRw0Bltw}{r1L7UXpESgiM_Kxq_jFUjw}{10.231.83.146}{10.231.83.146:9302}
  30. {yRj9xlW}{yRj9xlW8Q-WgQ0j6H4DbOQ}{SAn1BH7ES42OhsAknxVe0Q}{10.231.83.238}{10.231.83.238:9300}, local
  31. {lJgJwYh}{lJgJwYhmQA-fucaOBHD5Vg}{ho76tglrRPKjA8ui_UVJSg}{10.231.83.146}{10.231.83.146:9301}, master
  32.  
  33. [2019-01-25T10:43:12,279][WARN ][o.e.t.n.Netty4Transport ] [yRj9xlW] write and flush on the network layer failed (channel: [id: 0x6a653f36, L:0.0.0.0/0.0.0.0:9300 ! R:/10.231.83.146:50906])
  34. java.nio.channels.ClosedChannelException: null
  35. at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
  36. [2019-01-25T10:43:14,855][WARN ][o.e.c.NodeConnectionsService] [yRj9xlW] failed to connect to node {lJgJwYh}{lJgJwYhmQA-fucaOBHD5Vg}{ho76tglrRPKjA8ui_UVJSg}{10.231.83.146}{10.231.83.146:9301} (tried [1] times)
  37. org.elasticsearch.transport.ConnectTransportException: [lJgJwYh][10.231.83.146:9301] connect_timeout[30s]
  38. at org.elasticsearch.transport.netty4.Netty4Transport.connectToChannels(Netty4Transport.java:363) ~[?:?]
  39. at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:570) ~[elasticsearch-5.6.9.jar:5.6.9]
  40. at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:473) ~[elasticsearch-5.6.9.jar:5.6.9]
  41. at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:342) ~[elasticsearch-5.6.9.jar:5.6.9]
  42. at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:329) ~[elasticsearch-5.6.9.jar:5.6.9]
  43. at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:154) [elasticsearch-5.6.9.jar:5.6.9]
  44. at org.elasticsearch.cluster.NodeConnectionsService$ConnectionChecker.doRun(NodeConnectionsService.java:183) [elasticsearch-5.6.9.jar:5.6.9]
  45. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:674) [elasticsearch-5.6.9.jar:5.6.9]
  46. at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.9.jar:5.6.9]
  47. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
  48. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
  49. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  50. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: 10.231.83.146/10.231.83.146:9301
  51. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  52. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
  53. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352) ~[?:?]
  54. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  55. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?]
  56. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?]
  57. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?]
  58. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?]
  59. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
  60. ... 1 more
  61. Caused by: java.net.ConnectException: Connection refused
  62. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  63. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
  64. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352) ~[?:?]
  65. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  66. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?]
  67. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?]
  68. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?]
  69. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?]
  70. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
  71. ... 1 more
  72. [2019-01-25T10:43:15,292][INFO ][o.e.c.s.ClusterService ] [yRj9xlW] detected_master {wAVZzky}{wAVZzkyrSfyBmuGRw0Bltw}{r1L7UXpESgiM_Kxq_jFUjw}{10.231.83.146}{10.231.83.146:9302}, reason: zen-disco-receive(from master [master {wAVZzky}{wAVZzkyrSfyBmuGRw0Bltw}{r1L7UXpESgiM_Kxq_jFUjw}{10.231.83.146}{10.231.83.146:9302} committed version [181]])
  73. [2019-01-25T10:43:15,367][INFO ][o.e.c.s.ClusterService ] [yRj9xlW] removed {{lJgJwYh}{lJgJwYhmQA-fucaOBHD5Vg}{ho76tglrRPKjA8ui_UVJSg}{10.231.83.146}{10.231.83.146:9301},}, reason: zen-disco-receive(from master [master {wAVZzky}{wAVZzkyrSfyBmuGRw0Bltw}{r1L7UXpESgiM_Kxq_jFUjw}{10.231.83.146}{10.231.83.146:9302} committed version [182]])
  74. [2019-01-25T10:43:17,532][INFO ][o.e.d.z.ZenDiscovery ] [yRj9xlW] master_left [{wAVZzky}{wAVZzkyrSfyBmuGRw0Bltw}{r1L7UXpESgiM_Kxq_jFUjw}{10.231.83.146}{10.231.83.146:9302}], reason [shut_down]
  75. [2019-01-25T10:43:17,535][WARN ][o.e.d.z.ZenDiscovery ] [yRj9xlW] master left (reason = shut_down), current nodes: nodes:
  76. {wAVZzky}{wAVZzkyrSfyBmuGRw0Bltw}{r1L7UXpESgiM_Kxq_jFUjw}{10.231.83.146}{10.231.83.146:9302}, master
  77. {yRj9xlW}{yRj9xlW8Q-WgQ0j6H4DbOQ}{SAn1BH7ES42OhsAknxVe0Q}{10.231.83.238}{10.231.83.238:9300}, local
  78.  
  79. [2019-01-25T10:43:17,536][WARN ][o.e.t.n.Netty4Transport ] [yRj9xlW] write and flush on the network layer failed (channel: [id: 0xbc6c3d15, L:0.0.0.0/0.0.0.0:9300 ! R:/10.231.83.146:50934])
  80. java.nio.channels.ClosedChannelException: null
  81. at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
  82. [2019-01-25T10:43:17,548][WARN ][o.e.c.NodeConnectionsService] [yRj9xlW] failed to connect to node {wAVZzky}{wAVZzkyrSfyBmuGRw0Bltw}{r1L7UXpESgiM_Kxq_jFUjw}{10.231.83.146}{10.231.83.146:9302} (tried [1] times)
  83. org.elasticsearch.transport.ConnectTransportException: [wAVZzky][10.231.83.146:9302] connect_timeout[30s]
  84. at org.elasticsearch.transport.netty4.Netty4Transport.connectToChannels(Netty4Transport.java:363) ~[?:?]
  85. at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:570) ~[elasticsearch-5.6.9.jar:5.6.9]
  86. at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:473) ~[elasticsearch-5.6.9.jar:5.6.9]
  87. at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:342) ~[elasticsearch-5.6.9.jar:5.6.9]
  88. at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:329) ~[elasticsearch-5.6.9.jar:5.6.9]
  89. at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:154) [elasticsearch-5.6.9.jar:5.6.9]
  90. at org.elasticsearch.cluster.NodeConnectionsService$1.doRun(NodeConnectionsService.java:107) [elasticsearch-5.6.9.jar:5.6.9]
  91. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:674) [elasticsearch-5.6.9.jar:5.6.9]
  92. at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.9.jar:5.6.9]
  93. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
  94. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
  95. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  96. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: 10.231.83.146/10.231.83.146:9302
  97. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  98. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
  99. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352) ~[?:?]
  100. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  101. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?]
  102. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?]
  103. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?]
  104. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?]
  105. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
  106. ... 1 more
  107. Caused by: java.net.ConnectException: Connection refused
  108. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  109. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
  110. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352) ~[?:?]
  111. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  112. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?]
  113. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?]
  114. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?]
  115. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?]
  116. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
  117. ... 1 more
  118. [2019-01-25T10:43:22,537][INFO ][o.e.c.s.ClusterService ] [yRj9xlW] new_master {yRj9xlW}{yRj9xlW8Q-WgQ0j6H4DbOQ}{SAn1BH7ES42OhsAknxVe0Q}{10.231.83.238}{10.231.83.238:9300}, added {{lJgJwYh}{lJgJwYhmQA-fucaOBHD5Vg}{lbN7QcyiSWCOBYNL4jNkdg}{10.231.83.146}{10.231.83.146:9301},}, reason: zen-disco-elected-as-master ([1] nodes joined)[, ]
  119. [2019-01-25T10:43:22,586][WARN ][o.e.g.GatewayAllocator$InternalReplicaShardAllocator] [yRj9xlW] [pdai][0]: failed to list shard for shard_store on node [wAVZzkyrSfyBmuGRw0Bltw]
  120. org.elasticsearch.action.FailedNodeException: Failed node [wAVZzkyrSfyBmuGRw0Bltw]
  121. at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:239) ~[elasticsearch-5.6.9.jar:5.6.9]
  122. at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$200(TransportNodesAction.java:153) ~[elasticsearch-5.6.9.jar:5.6.9]
  123. at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:211) ~[elasticsearch-5.6.9.jar:5.6.9]
  124. at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:520) ~[elasticsearch-5.6.9.jar:5.6.9]
  125. at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.start(TransportNodesAction.java:197) ~[elasticsearch-5.6.9.jar:5.6.9]
  126. at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:89) ~[elasticsearch-5.6.9.jar:5.6.9]
  127. at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:52) ~[elasticsearch-5.6.9.jar:5.6.9]
  128. at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) ~[elasticsearch-5.6.9.jar:5.6.9]
  129. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) ~[elasticsearch-5.6.9.jar:5.6.9]
  130. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84) ~[elasticsearch-5.6.9.jar:5.6.9]
  131. at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.list(TransportNodesListShardStoreMetaData.java:90) ~[elasticsearch-5.6.9.jar:5.6.9]
  132. at org.elasticsearch.gateway.AsyncShardFetch.asyncFetch(AsyncShardFetch.java:289) ~[elasticsearch-5.6.9.jar:5.6.9]
  133. at org.elasticsearch.gateway.AsyncShardFetch.fetchData(AsyncShardFetch.java:127) ~[elasticsearch-5.6.9.jar:5.6.9]
  134. at org.elasticsearch.gateway.GatewayAllocator$InternalReplicaShardAllocator.fetchData(GatewayAllocator.java:188) ~[elasticsearch-5.6.9.jar:5.6.9]
  135. at org.elasticsearch.gateway.ReplicaShardAllocator.makeAllocationDecision(ReplicaShardAllocator.java:172) ~[elasticsearch-5.6.9.jar:5.6.9]
  136. at org.elasticsearch.gateway.BaseGatewayShardAllocator.allocateUnassigned(BaseGatewayShardAllocator.java:62) ~[elasticsearch-5.6.9.jar:5.6.9]
  137. at org.elasticsearch.gateway.GatewayAllocator.innerAllocatedUnassigned(GatewayAllocator.java:123) ~[elasticsearch-5.6.9.jar:5.6.9]
  138. at org.elasticsearch.gateway.GatewayAllocator.allocateUnassigned(GatewayAllocator.java:111) ~[elasticsearch-5.6.9.jar:5.6.9]
  139. at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:326) ~[elasticsearch-5.6.9.jar:5.6.9]
  140. at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:296) ~[elasticsearch-5.6.9.jar:5.6.9]
  141. at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:281) ~[elasticsearch-5.6.9.jar:5.6.9]
  142. at org.elasticsearch.discovery.zen.NodeJoinController$JoinTaskExecutor.execute(NodeJoinController.java:461) ~[elasticsearch-5.6.9.jar:5.6.9]
  143. at org.elasticsearch.cluster.service.ClusterService.executeTasks(ClusterService.java:634) ~[elasticsearch-5.6.9.jar:5.6.9]
  144. at org.elasticsearch.cluster.service.ClusterService.calculateTaskOutputs(ClusterService.java:612) ~[elasticsearch-5.6.9.jar:5.6.9]
  145. at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:571) ~[elasticsearch-5.6.9.jar:5.6.9]
  146. at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) ~[elasticsearch-5.6.9.jar:5.6.9]
  147. at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) ~[elasticsearch-5.6.9.jar:5.6.9]
  148. at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) ~[elasticsearch-5.6.9.jar:5.6.9]
  149. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) ~[elasticsearch-5.6.9.jar:5.6.9]
  150. at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) ~[elasticsearch-5.6.9.jar:5.6.9]
  151. at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) ~[elasticsearch-5.6.9.jar:5.6.9]
  152. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_181]
  153. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_181]
  154. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  155. Caused by: org.elasticsearch.transport.NodeNotConnectedException: [wAVZzky][10.231.83.146:9302] Node not connected
  156. at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:640) ~[elasticsearch-5.6.9.jar:5.6.9]
  157. at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:117) ~[elasticsearch-5.6.9.jar:5.6.9]
  158. at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:540) ~[elasticsearch-5.6.9.jar:5.6.9]
  159. at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:516) ~[elasticsearch-5.6.9.jar:5.6.9]
  160. ... 30 more
  161. [2019-01-25T10:43:22,589][WARN ][o.e.g.GatewayAllocator$InternalReplicaShardAllocator] [yRj9xlW] [routing][0]: failed to list shard for shard_store on node [wAVZzkyrSfyBmuGRw0Bltw]
  162. org.elasticsearch.action.FailedNodeException: Failed node [wAVZzkyrSfyBmuGRw0Bltw]
  163. at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:239) ~[elasticsearch-5.6.9.jar:5.6.9]
  164. at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$200(TransportNodesAction.java:153) ~[elasticsearch-5.6.9.jar:5.6.9]
  165. at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:211) ~[elasticsearch-5.6.9.jar:5.6.9]
  166. at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:520) ~[elasticsearch-5.6.9.jar:5.6.9]
  167. at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.start(TransportNodesAction.java:197) ~[elasticsearch-5.6.9.jar:5.6.9]
  168. at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:89) ~[elasticsearch-5.6.9.jar:5.6.9]
  169. at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:52) ~[elasticsearch-5.6.9.jar:5.6.9]
  170. at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) ~[elasticsearch-5.6.9.jar:5.6.9]
  171. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) ~[elasticsearch-5.6.9.jar:5.6.9]
  172. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84) ~[elasticsearch-5.6.9.jar:5.6.9]
  173. at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.list(TransportNodesListShardStoreMetaData.java:90) ~[elasticsearch-5.6.9.jar:5.6.9]
  174. at org.elasticsearch.gateway.AsyncShardFetch.asyncFetch(AsyncShardFetch.java:289) ~[elasticsearch-5.6.9.jar:5.6.9]
  175. at org.elasticsearch.gateway.AsyncShardFetch.fetchData(AsyncShardFetch.java:127) ~[elasticsearch-5.6.9.jar:5.6.9]
  176. at org.elasticsearch.gateway.GatewayAllocator$InternalReplicaShardAllocator.fetchData(GatewayAllocator.java:188) ~[elasticsearch-5.6.9.jar:5.6.9]
  177. at org.elasticsearch.gateway.ReplicaShardAllocator.makeAllocationDecision(ReplicaShardAllocator.java:172) ~[elasticsearch-5.6.9.jar:5.6.9]
  178. at org.elasticsearch.gateway.BaseGatewayShardAllocator.allocateUnassigned(BaseGatewayShardAllocator.java:62) ~[elasticsearch-5.6.9.jar:5.6.9]
  179. at org.elasticsearch.gateway.GatewayAllocator.innerAllocatedUnassigned(GatewayAllocator.java:123) ~[elasticsearch-5.6.9.jar:5.6.9]
  180. at org.elasticsearch.gateway.GatewayAllocator.allocateUnassigned(GatewayAllocator.java:111) ~[elasticsearch-5.6.9.jar:5.6.9]
  181. at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:326) ~[elasticsearch-5.6.9.jar:5.6.9]
  182. at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:296) ~[elasticsearch-5.6.9.jar:5.6.9]
  183. at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:281) ~[elasticsearch-5.6.9.jar:5.6.9]
  184. at org.elasticsearch.discovery.zen.NodeJoinController$JoinTaskExecutor.execute(NodeJoinController.java:461) ~[elasticsearch-5.6.9.jar:5.6.9]
  185. at org.elasticsearch.cluster.service.ClusterService.executeTasks(ClusterService.java:634) ~[elasticsearch-5.6.9.jar:5.6.9]
  186. at org.elasticsearch.cluster.service.ClusterService.calculateTaskOutputs(ClusterService.java:612) ~[elasticsearch-5.6.9.jar:5.6.9]
  187. at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:571) ~[elasticsearch-5.6.9.jar:5.6.9]
  188. at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) ~[elasticsearch-5.6.9.jar:5.6.9]
  189. at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) ~[elasticsearch-5.6.9.jar:5.6.9]
  190. at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) ~[elasticsearch-5.6.9.jar:5.6.9]
  191. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) ~[elasticsearch-5.6.9.jar:5.6.9]
  192. at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) ~[elasticsearch-5.6.9.jar:5.6.9]
  193. at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) ~[elasticsearch-5.6.9.jar:5.6.9]
  194. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_181]
  195. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_181]
  196. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  197. Caused by: org.elasticsearch.transport.NodeNotConnectedException: [wAVZzky][10.231.83.146:9302] Node not connected
  198. at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:640) ~[elasticsearch-5.6.9.jar:5.6.9]
  199. at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:117) ~[elasticsearch-5.6.9.jar:5.6.9]
  200. at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:540) ~[elasticsearch-5.6.9.jar:5.6.9]
  201. at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:516) ~[elasticsearch-5.6.9.jar:5.6.9]
  202. ... 30 more
  203. [2019-01-25T10:43:22,592][WARN ][o.e.g.GatewayAllocator$InternalReplicaShardAllocator] [yRj9xlW] [lc][0]: failed to list shard for shard_store on node [wAVZzkyrSfyBmuGRw0Bltw]
  204. org.elasticsearch.action.FailedNodeException: Failed node [wAVZzkyrSfyBmuGRw0Bltw]
  205. at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:239) ~[elasticsearch-5.6.9.jar:5.6.9]
  206. at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$200(TransportNodesAction.java:153) ~[elasticsearch-5.6.9.jar:5.6.9]
  207. at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:211) ~[elasticsearch-5.6.9.jar:5.6.9]
  208. at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:520) ~[elasticsearch-5.6.9.jar:5.6.9]
  209. at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.start(TransportNodesAction.java:197) ~[elasticsearch-5.6.9.jar:5.6.9]
  210. at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:89) ~[elasticsearch-5.6.9.jar:5.6.9]
  211. at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:52) ~[elasticsearch-5.6.9.jar:5.6.9]
  212. at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) ~[elasticsearch-5.6.9.jar:5.6.9]
  213. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) ~[elasticsearch-5.6.9.jar:5.6.9]
  214. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84) ~[elasticsearch-5.6.9.jar:5.6.9]
  215. at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.list(TransportNodesListShardStoreMetaData.java:90) ~[elasticsearch-5.6.9.jar:5.6.9]
  216. at org.elasticsearch.gateway.AsyncShardFetch.asyncFetch(AsyncShardFetch.java:289) ~[elasticsearch-5.6.9.jar:5.6.9]
  217. at org.elasticsearch.gateway.AsyncShardFetch.fetchData(AsyncShardFetch.java:127) ~[elasticsearch-5.6.9.jar:5.6.9]
  218. at org.elasticsearch.gateway.GatewayAllocator$InternalReplicaShardAllocator.fetchData(GatewayAllocator.java:188) ~[elasticsearch-5.6.9.jar:5.6.9]
  219. at org.elasticsearch.gateway.ReplicaShardAllocator.makeAllocationDecision(ReplicaShardAllocator.java:172) ~[elasticsearch-5.6.9.jar:5.6.9]
  220. at org.elasticsearch.gateway.BaseGatewayShardAllocator.allocateUnassigned(BaseGatewayShardAllocator.java:62) ~[elasticsearch-5.6.9.jar:5.6.9]
  221. at org.elasticsearch.gateway.GatewayAllocator.innerAllocatedUnassigned(GatewayAllocator.java:123) ~[elasticsearch-5.6.9.jar:5.6.9]
  222. at org.elasticsearch.gateway.GatewayAllocator.allocateUnassigned(GatewayAllocator.java:111) ~[elasticsearch-5.6.9.jar:5.6.9]
  223. at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:326) ~[elasticsearch-5.6.9.jar:5.6.9]
  224. at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:296) ~[elasticsearch-5.6.9.jar:5.6.9]
  225. at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:281) ~[elasticsearch-5.6.9.jar:5.6.9]
  226. at org.elasticsearch.discovery.zen.NodeJoinController$JoinTaskExecutor.execute(NodeJoinController.java:461) ~[elasticsearch-5.6.9.jar:5.6.9]
  227. at org.elasticsearch.cluster.service.ClusterService.executeTasks(ClusterService.java:634) ~[elasticsearch-5.6.9.jar:5.6.9]
  228. at org.elasticsearch.cluster.service.ClusterService.calculateTaskOutputs(ClusterService.java:612) ~[elasticsearch-5.6.9.jar:5.6.9]
  229. at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:571) ~[elasticsearch-5.6.9.jar:5.6.9]
  230. at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) ~[elasticsearch-5.6.9.jar:5.6.9]
  231. at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) ~[elasticsearch-5.6.9.jar:5.6.9]
  232. at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) ~[elasticsearch-5.6.9.jar:5.6.9]
  233. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) ~[elasticsearch-5.6.9.jar:5.6.9]
  234. at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) ~[elasticsearch-5.6.9.jar:5.6.9]
  235. at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) ~[elasticsearch-5.6.9.jar:5.6.9]
  236. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_181]
  237. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_181]
  238. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  239. Caused by: org.elasticsearch.transport.NodeNotConnectedException: [wAVZzky][10.231.83.146:9302] Node not connected
  240. at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:640) ~[elasticsearch-5.6.9.jar:5.6.9]
  241. at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:117) ~[elasticsearch-5.6.9.jar:5.6.9]
  242. at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:540) ~[elasticsearch-5.6.9.jar:5.6.9]
  243. at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:516) ~[elasticsearch-5.6.9.jar:5.6.9]
  244. ... 30 more
  245. [2019-01-25T10:43:22,592][WARN ][o.e.g.GatewayAllocator$InternalReplicaShardAllocator] [yRj9xlW] [vanmessagetopic][1]: failed to list shard for shard_store on node [wAVZzkyrSfyBmuGRw0Bltw]
  246. org.elasticsearch.action.FailedNodeException: Failed node [wAVZzkyrSfyBmuGRw0Bltw]
  247. at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:239) ~[elasticsearch-5.6.9.jar:5.6.9]
  248. at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$200(TransportNodesAction.java:153) ~[elasticsearch-5.6.9.jar:5.6.9]
  249. at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:211) ~[elasticsearch-5.6.9.jar:5.6.9]
  250. at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:520) ~[elasticsearch-5.6.9.jar:5.6.9]
  251. at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.start(TransportNodesAction.java:197) ~[elasticsearch-5.6.9.jar:5.6.9]
  252. at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:89) ~[elasticsearch-5.6.9.jar:5.6.9]
  253. at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:52) ~[elasticsearch-5.6.9.jar:5.6.9]
  254. at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) ~[elasticsearch-5.6.9.jar:5.6.9]
  255. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) ~[elasticsearch-5.6.9.jar:5.6.9]
  256. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84) ~[elasticsearch-5.6.9.jar:5.6.9]
  257. at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.list(TransportNodesListShardStoreMetaData.java:90) ~[elasticsearch-5.6.9.jar:5.6.9]
  258. at org.elasticsearch.gateway.AsyncShardFetch.asyncFetch(AsyncShardFetch.java:289) ~[elasticsearch-5.6.9.jar:5.6.9]
  259. at org.elasticsearch.gateway.AsyncShardFetch.fetchData(AsyncShardFetch.java:127) ~[elasticsearch-5.6.9.jar:5.6.9]
  260. at org.elasticsearch.gateway.GatewayAllocator$InternalReplicaShardAllocator.fetchData(GatewayAllocator.java:188) ~[elasticsearch-5.6.9.jar:5.6.9]
  261. at org.elasticsearch.gateway.ReplicaShardAllocator.makeAllocationDecision(ReplicaShardAllocator.java:172) ~[elasticsearch-5.6.9.jar:5.6.9]
  262. at org.elasticsearch.gateway.BaseGatewayShardAllocator.allocateUnassigned(BaseGatewayShardAllocator.java:62) ~[elasticsearch-5.6.9.jar:5.6.9]
  263. at org.elasticsearch.gateway.GatewayAllocator.innerAllocatedUnassigned(GatewayAllocator.java:123) ~[elasticsearch-5.6.9.jar:5.6.9]
  264. at org.elasticsearch.gateway.GatewayAllocator.allocateUnassigned(GatewayAllocator.java:111) ~[elasticsearch-5.6.9.jar:5.6.9]
  265. at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:326) ~[elasticsearch-5.6.9.jar:5.6.9]
  266. at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:296) ~[elasticsearch-5.6.9.jar:5.6.9]
  267. at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:281) ~[elasticsearch-5.6.9.jar:5.6.9]
  268. at org.elasticsearch.discovery.zen.NodeJoinController$JoinTaskExecutor.execute(NodeJoinController.java:461) ~[elasticsearch-5.6.9.jar:5.6.9]
  269. at org.elasticsearch.cluster.service.ClusterService.executeTasks(ClusterService.java:634) ~[elasticsearch-5.6.9.jar:5.6.9]
  270. at org.elasticsearch.cluster.service.ClusterService.calculateTaskOutputs(ClusterService.java:612) ~[elasticsearch-5.6.9.jar:5.6.9]
  271. at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:571) ~[elasticsearch-5.6.9.jar:5.6.9]
  272. at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) ~[elasticsearch-5.6.9.jar:5.6.9]
  273. at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) ~[elasticsearch-5.6.9.jar:5.6.9]
  274. at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) ~[elasticsearch-5.6.9.jar:5.6.9]
  275. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) ~[elasticsearch-5.6.9.jar:5.6.9]
  276. at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) ~[elasticsearch-5.6.9.jar:5.6.9]
  277. at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) ~[elasticsearch-5.6.9.jar:5.6.9]
  278. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_181]
  279. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_181]
  280. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  281. Caused by: org.elasticsearch.transport.NodeNotConnectedException: [wAVZzky][10.231.83.146:9302] Node not connected
  282. at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:640) ~[elasticsearch-5.6.9.jar:5.6.9]
  283. at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:117) ~[elasticsearch-5.6.9.jar:5.6.9]
  284. at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:540) ~[elasticsearch-5.6.9.jar:5.6.9]
  285. at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:516) ~[elasticsearch-5.6.9.jar:5.6.9]
  286. ... 30 more
  287. [2019-01-25T10:43:22,593][WARN ][o.e.g.GatewayAllocator$InternalReplicaShardAllocator] [yRj9xlW] [device][0]: failed to list shard for shard_store on node [wAVZzkyrSfyBmuGRw0Bltw]
  288. org.elasticsearch.action.FailedNodeException: Failed node [wAVZzkyrSfyBmuGRw0Bltw]
  289. at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:239) ~[elasticsearch-5.6.9.jar:5.6.9]
  290. at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$200(TransportNodesAction.java:153) ~[elasticsearch-5.6.9.jar:5.6.9]
  291. at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:211) ~[elasticsearch-5.6.9.jar:5.6.9]
  292. at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:520) ~[elasticsearch-5.6.9.jar:5.6.9]
  293. at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.start(TransportNodesAction.java:197) ~[elasticsearch-5.6.9.jar:5.6.9]
  294. at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:89) ~[elasticsearch-5.6.9.jar:5.6.9]
  295. at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:52) ~[elasticsearch-5.6.9.jar:5.6.9]
  296. at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) ~[elasticsearch-5.6.9.jar:5.6.9]
  297. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) ~[elasticsearch-5.6.9.jar:5.6.9]
  298. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84) ~[elasticsearch-5.6.9.jar:5.6.9]
  299. at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.list(TransportNodesListShardStoreMetaData.java:90) ~[elasticsearch-5.6.9.jar:5.6.9]
  300. at org.elasticsearch.gateway.AsyncShardFetch.asyncFetch(AsyncShardFetch.java:289) ~[elasticsearch-5.6.9.jar:5.6.9]
  301. at org.elasticsearch.gateway.AsyncShardFetch.fetchData(AsyncShardFetch.java:127) ~[elasticsearch-5.6.9.jar:5.6.9]
  302. at org.elasticsearch.gateway.GatewayAllocator$InternalReplicaShardAllocator.fetchData(GatewayAllocator.java:188) ~[elasticsearch-5.6.9.jar:5.6.9]
  303. at org.elasticsearch.gateway.ReplicaShardAllocator.makeAllocationDecision(ReplicaShardAllocator.java:172) ~[elasticsearch-5.6.9.jar:5.6.9]
  304. at org.elasticsearch.gateway.BaseGatewayShardAllocator.allocateUnassigned(BaseGatewayShardAllocator.java:62) ~[elasticsearch-5.6.9.jar:5.6.9]
  305. at org.elasticsearch.gateway.GatewayAllocator.innerAllocatedUnassigned(GatewayAllocator.java:123) ~[elasticsearch-5.6.9.jar:5.6.9]
  306. at org.elasticsearch.gateway.GatewayAllocator.allocateUnassigned(GatewayAllocator.java:111) ~[elasticsearch-5.6.9.jar:5.6.9]
  307. at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:326) ~[elasticsearch-5.6.9.jar:5.6.9]
  308. at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:296) ~[elasticsearch-5.6.9.jar:5.6.9]
  309. at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:281) ~[elasticsearch-5.6.9.jar:5.6.9]
  310. at org.elasticsearch.discovery.zen.NodeJoinController$JoinTaskExecutor.execute(NodeJoinController.java:461) ~[elasticsearch-5.6.9.jar:5.6.9]
  311. at org.elasticsearch.cluster.service.ClusterService.executeTasks(ClusterService.java:634) ~[elasticsearch-5.6.9.jar:5.6.9]
  312. at org.elasticsearch.cluster.service.ClusterService.calculateTaskOutputs(ClusterService.java:612) ~[elasticsearch-5.6.9.jar:5.6.9]
  313. at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:571) ~[elasticsearch-5.6.9.jar:5.6.9]
  314. at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) ~[elasticsearch-5.6.9.jar:5.6.9]
  315. at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) ~[elasticsearch-5.6.9.jar:5.6.9]
  316. at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) ~[elasticsearch-5.6.9.jar:5.6.9]
  317. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) ~[elasticsearch-5.6.9.jar:5.6.9]
  318. at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) ~[elasticsearch-5.6.9.jar:5.6.9]
  319. at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) ~[elasticsearch-5.6.9.jar:5.6.9]
  320. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_181]
  321. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_181]
  322. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  323. Caused by: org.elasticsearch.transport.NodeNotConnectedException: [wAVZzky][10.231.83.146:9302] Node not connected
  324. at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:640) ~[elasticsearch-5.6.9.jar:5.6.9]
  325. at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:117) ~[elasticsearch-5.6.9.jar:5.6.9]
  326. at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:540) ~[elasticsearch-5.6.9.jar:5.6.9]
  327. at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:516) ~[elasticsearch-5.6.9.jar:5.6.9]
  328. ... 30 more
  329. [2019-01-25T10:43:22,596][WARN ][o.e.g.GatewayAllocator$InternalReplicaShardAllocator] [yRj9xlW] [vanmessagetopic][3]: failed to list shard for shard_store on node [wAVZzkyrSfyBmuGRw0Bltw]
  330. org.elasticsearch.action.FailedNodeException: Failed node [wAVZzkyrSfyBmuGRw0Bltw]
  331. at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:239) ~[elasticsearch-5.6.9.jar:5.6.9]
  332. at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$200(TransportNodesAction.java:153) ~[elasticsearch-5.6.9.jar:5.6.9]
  333. at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:211) ~[elasticsearch-5.6.9.jar:5.6.9]
  334. at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:520) ~[elasticsearch-5.6.9.jar:5.6.9]
  335. at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.start(TransportNodesAction.java:197) ~[elasticsearch-5.6.9.jar:5.6.9]
  336. at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:89) ~[elasticsearch-5.6.9.jar:5.6.9]
  337. at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:52) ~[elasticsearch-5.6.9.jar:5.6.9]
  338. at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) ~[elasticsearch-5.6.9.jar:5.6.9]
  339. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) ~[elasticsearch-5.6.9.jar:5.6.9]
  340. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84) ~[elasticsearch-5.6.9.jar:5.6.9]
  341. at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.list(TransportNodesListShardStoreMetaData.java:90) ~[elasticsearch-5.6.9.jar:5.6.9]
  342. at org.elasticsearch.gateway.AsyncShardFetch.asyncFetch(AsyncShardFetch.java:289) ~[elasticsearch-5.6.9.jar:5.6.9]
  343. at org.elasticsearch.gateway.AsyncShardFetch.fetchData(AsyncShardFetch.java:127) ~[elasticsearch-5.6.9.jar:5.6.9]
  344. at org.elasticsearch.gateway.GatewayAllocator$InternalReplicaShardAllocator.fetchData(GatewayAllocator.java:188) ~[elasticsearch-5.6.9.jar:5.6.9]
  345. at org.elasticsearch.gateway.ReplicaShardAllocator.makeAllocationDecision(ReplicaShardAllocator.java:172) ~[elasticsearch-5.6.9.jar:5.6.9]
  346. at org.elasticsearch.gateway.BaseGatewayShardAllocator.allocateUnassigned(BaseGatewayShardAllocator.java:62) ~[elasticsearch-5.6.9.jar:5.6.9]
  347. at org.elasticsearch.gateway.GatewayAllocator.innerAllocatedUnassigned(GatewayAllocator.java:123) ~[elasticsearch-5.6.9.jar:5.6.9]
  348. at org.elasticsearch.gateway.GatewayAllocator.allocateUnassigned(GatewayAllocator.java:111) ~[elasticsearch-5.6.9.jar:5.6.9]
  349. at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:326) ~[elasticsearch-5.6.9.jar:5.6.9]
  350. at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:296) ~[elasticsearch-5.6.9.jar:5.6.9]
  351. at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:281) ~[elasticsearch-5.6.9.jar:5.6.9]
  352. at org.elasticsearch.discovery.zen.NodeJoinController$JoinTaskExecutor.execute(NodeJoinController.java:461) ~[elasticsearch-5.6.9.jar:5.6.9]
  353. at org.elasticsearch.cluster.service.ClusterService.executeTasks(ClusterService.java:634) ~[elasticsearch-5.6.9.jar:5.6.9]
  354. at org.elasticsearch.cluster.service.ClusterService.calculateTaskOutputs(ClusterService.java:612) ~[elasticsearch-5.6.9.jar:5.6.9]
  355. at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:571) ~[elasticsearch-5.6.9.jar:5.6.9]
  356. at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) ~[elasticsearch-5.6.9.jar:5.6.9]
  357. at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) ~[elasticsearch-5.6.9.jar:5.6.9]
  358. at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) ~[elasticsearch-5.6.9.jar:5.6.9]
  359. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) ~[elasticsearch-5.6.9.jar:5.6.9]
  360. at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) ~[elasticsearch-5.6.9.jar:5.6.9]
  361. at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) ~[elasticsearch-5.6.9.jar:5.6.9]
  362. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_181]
  363. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_181]
  364. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  365. Caused by: org.elasticsearch.transport.NodeNotConnectedException: [wAVZzky][10.231.83.146:9302] Node not connected
  366. at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:640) ~[elasticsearch-5.6.9.jar:5.6.9]
  367. at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:117) ~[elasticsearch-5.6.9.jar:5.6.9]
  368. at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:540) ~[elasticsearch-5.6.9.jar:5.6.9]
  369. at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:516) ~[elasticsearch-5.6.9.jar:5.6.9]
  370. ... 30 more
  371. [2019-01-25T10:43:22,597][WARN ][o.e.g.GatewayAllocator$InternalReplicaShardAllocator] [yRj9xlW] [vanmessagetopic][4]: failed to list shard for shard_store on node [wAVZzkyrSfyBmuGRw0Bltw]
  372. org.elasticsearch.action.FailedNodeException: Failed node [wAVZzkyrSfyBmuGRw0Bltw]
  373. at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:239) ~[elasticsearch-5.6.9.jar:5.6.9]
  374. at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$200(TransportNodesAction.java:153) ~[elasticsearch-5.6.9.jar:5.6.9]
  375. at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:211) ~[elasticsearch-5.6.9.jar:5.6.9]
  376. at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:520) ~[elasticsearch-5.6.9.jar:5.6.9]
  377. at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.start(TransportNodesAction.java:197) ~[elasticsearch-5.6.9.jar:5.6.9]
  378. at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:89) ~[elasticsearch-5.6.9.jar:5.6.9]
  379. at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:52) ~[elasticsearch-5.6.9.jar:5.6.9]
  380. at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) ~[elasticsearch-5.6.9.jar:5.6.9]
  381. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) ~[elasticsearch-5.6.9.jar:5.6.9]
  382. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84) ~[elasticsearch-5.6.9.jar:5.6.9]
  383. at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.list(TransportNodesListShardStoreMetaData.java:90) ~[elasticsearch-5.6.9.jar:5.6.9]
  384. at org.elasticsearch.gateway.AsyncShardFetch.asyncFetch(AsyncShardFetch.java:289) ~[elasticsearch-5.6.9.jar:5.6.9]
  385. at org.elasticsearch.gateway.AsyncShardFetch.fetchData(AsyncShardFetch.java:127) ~[elasticsearch-5.6.9.jar:5.6.9]
  386. at org.elasticsearch.gateway.GatewayAllocator$InternalReplicaShardAllocator.fetchData(GatewayAllocator.java:188) ~[elasticsearch-5.6.9.jar:5.6.9]
  387. at org.elasticsearch.gateway.ReplicaShardAllocator.makeAllocationDecision(ReplicaShardAllocator.java:172) ~[elasticsearch-5.6.9.jar:5.6.9]
  388. at org.elasticsearch.gateway.BaseGatewayShardAllocator.allocateUnassigned(BaseGatewayShardAllocator.java:62) ~[elasticsearch-5.6.9.jar:5.6.9]
  389. at org.elasticsearch.gateway.GatewayAllocator.innerAllocatedUnassigned(GatewayAllocator.java:123) ~[elasticsearch-5.6.9.jar:5.6.9]
  390. at org.elasticsearch.gateway.GatewayAllocator.allocateUnassigned(GatewayAllocator.java:111) ~[elasticsearch-5.6.9.jar:5.6.9]
  391. at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:326) ~[elasticsearch-5.6.9.jar:5.6.9]
  392. at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:296) ~[elasticsearch-5.6.9.jar:5.6.9]
  393. at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:281) ~[elasticsearch-5.6.9.jar:5.6.9]
  394. at org.elasticsearch.discovery.zen.NodeJoinController$JoinTaskExecutor.execute(NodeJoinController.java:461) ~[elasticsearch-5.6.9.jar:5.6.9]
  395. at org.elasticsearch.cluster.service.ClusterService.executeTasks(ClusterService.java:634) ~[elasticsearch-5.6.9.jar:5.6.9]
  396. at org.elasticsearch.cluster.service.ClusterService.calculateTaskOutputs(ClusterService.java:612) ~[elasticsearch-5.6.9.jar:5.6.9]
  397. at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:571) ~[elasticsearch-5.6.9.jar:5.6.9]
  398. at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) ~[elasticsearch-5.6.9.jar:5.6.9]
  399. at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) ~[elasticsearch-5.6.9.jar:5.6.9]
  400. at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) ~[elasticsearch-5.6.9.jar:5.6.9]
  401. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) ~[elasticsearch-5.6.9.jar:5.6.9]
  402. at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) ~[elasticsearch-5.6.9.jar:5.6.9]
  403. at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) ~[elasticsearch-5.6.9.jar:5.6.9]
  404. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_181]
  405. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_181]
  406. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  407. Caused by: org.elasticsearch.transport.NodeNotConnectedException: [wAVZzky][10.231.83.146:9302] Node not connected
  408. at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:640) ~[elasticsearch-5.6.9.jar:5.6.9]
  409. at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:117) ~[elasticsearch-5.6.9.jar:5.6.9]
  410. at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:540) ~[elasticsearch-5.6.9.jar:5.6.9]
  411. at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:516) ~[elasticsearch-5.6.9.jar:5.6.9]
  412. ... 30 more
  413. [2019-01-25T10:43:22,989][INFO ][o.e.c.r.DelayedAllocationService] [yRj9xlW] scheduling reroute for delayed shards in [52.3s] (7 delayed shards)
  414. [2019-01-25T10:43:22,993][DEBUG][o.e.a.a.c.n.s.TransportNodesStatsAction] [yRj9xlW] failed to execute on node [wAVZzkyrSfyBmuGRw0Bltw]
  415. org.elasticsearch.transport.NodeNotConnectedException: [wAVZzky][10.231.83.146:9302] Node not connected
  416. at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:640) ~[elasticsearch-5.6.9.jar:5.6.9]
  417. at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:117) ~[elasticsearch-5.6.9.jar:5.6.9]
  418. at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:540) ~[elasticsearch-5.6.9.jar:5.6.9]
  419. at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:516) ~[elasticsearch-5.6.9.jar:5.6.9]
  420. at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.start(TransportNodesAction.java:197) ~[elasticsearch-5.6.9.jar:5.6.9]
  421. at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:89) ~[elasticsearch-5.6.9.jar:5.6.9]
  422. at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:52) ~[elasticsearch-5.6.9.jar:5.6.9]
  423. at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) ~[elasticsearch-5.6.9.jar:5.6.9]
  424. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) ~[elasticsearch-5.6.9.jar:5.6.9]
  425. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84) ~[elasticsearch-5.6.9.jar:5.6.9]
  426. at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) ~[elasticsearch-5.6.9.jar:5.6.9]
  427. at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) ~[elasticsearch-5.6.9.jar:5.6.9]
  428. at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408) ~[elasticsearch-5.6.9.jar:5.6.9]
  429. at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:730) ~[elasticsearch-5.6.9.jar:5.6.9]
  430. at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.nodesStats(AbstractClient.java:826) ~[elasticsearch-5.6.9.jar:5.6.9]
  431. at org.elasticsearch.cluster.InternalClusterInfoService.updateNodeStats(InternalClusterInfoService.java:256) ~[elasticsearch-5.6.9.jar:5.6.9]
  432. at org.elasticsearch.cluster.InternalClusterInfoService.refresh(InternalClusterInfoService.java:292) ~[elasticsearch-5.6.9.jar:5.6.9]
  433. at org.elasticsearch.cluster.InternalClusterInfoService.maybeRefresh(InternalClusterInfoService.java:277) ~[elasticsearch-5.6.9.jar:5.6.9]
  434. at org.elasticsearch.cluster.InternalClusterInfoService.lambda$onMaster$0(InternalClusterInfoService.java:137) ~[elasticsearch-5.6.9.jar:5.6.9]
  435. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.9.jar:5.6.9]
  436. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
  437. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
  438. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  439. [2019-01-25T10:43:23,020][DEBUG][o.e.a.a.i.s.TransportIndicesStatsAction] [yRj9xlW] failed to execute [indices:monitor/stats] on node [wAVZzkyrSfyBmuGRw0Bltw]
  440. org.elasticsearch.transport.NodeNotConnectedException: [wAVZzky][10.231.83.146:9302] Node not connected
  441. at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:640) ~[elasticsearch-5.6.9.jar:5.6.9]
  442. at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:117) ~[elasticsearch-5.6.9.jar:5.6.9]
  443. at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:540) ~[elasticsearch-5.6.9.jar:5.6.9]
  444. at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:503) ~[elasticsearch-5.6.9.jar:5.6.9]
  445. at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.sendNodeRequest(TransportBroadcastByNodeAction.java:322) ~[elasticsearch-5.6.9.jar:5.6.9]
  446. at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.start(TransportBroadcastByNodeAction.java:311) ~[elasticsearch-5.6.9.jar:5.6.9]
  447. at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:234) ~[elasticsearch-5.6.9.jar:5.6.9]
  448. at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:79) ~[elasticsearch-5.6.9.jar:5.6.9]
  449. at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) ~[elasticsearch-5.6.9.jar:5.6.9]
  450. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) ~[elasticsearch-5.6.9.jar:5.6.9]
  451. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84) ~[elasticsearch-5.6.9.jar:5.6.9]
  452. at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) ~[elasticsearch-5.6.9.jar:5.6.9]
  453. at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) ~[elasticsearch-5.6.9.jar:5.6.9]
  454. at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408) ~[elasticsearch-5.6.9.jar:5.6.9]
  455. at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1256) ~[elasticsearch-5.6.9.jar:5.6.9]
  456. at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.stats(AbstractClient.java:1577) ~[elasticsearch-5.6.9.jar:5.6.9]
  457. at org.elasticsearch.cluster.InternalClusterInfoService.updateIndicesStats(InternalClusterInfoService.java:270) ~[elasticsearch-5.6.9.jar:5.6.9]
  458. at org.elasticsearch.cluster.InternalClusterInfoService.refresh(InternalClusterInfoService.java:321) ~[elasticsearch-5.6.9.jar:5.6.9]
  459. at org.elasticsearch.cluster.InternalClusterInfoService.maybeRefresh(InternalClusterInfoService.java:277) ~[elasticsearch-5.6.9.jar:5.6.9]
  460. at org.elasticsearch.cluster.InternalClusterInfoService.lambda$onMaster$0(InternalClusterInfoService.java:137) ~[elasticsearch-5.6.9.jar:5.6.9]
  461. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.9.jar:5.6.9]
  462. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
  463. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
  464. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  465. [2019-01-25T10:43:23,034][INFO ][o.e.c.r.a.AllocationService] [yRj9xlW] Cluster health status changed from [YELLOW] to [RED] (reason: []).
  466. [2019-01-25T10:43:23,035][INFO ][o.e.c.s.ClusterService ] [yRj9xlW] removed {{wAVZzky}{wAVZzkyrSfyBmuGRw0Bltw}{r1L7UXpESgiM_Kxq_jFUjw}{10.231.83.146}{10.231.83.146:9302},}, reason: zen-disco-node-failed({wAVZzky}{wAVZzkyrSfyBmuGRw0Bltw}{r1L7UXpESgiM_Kxq_jFUjw}{10.231.83.146}{10.231.83.146:9302}), reason(transport disconnected)
  467. [2019-01-25T10:43:24,112][INFO ][o.e.c.r.a.AllocationService] [yRj9xlW] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[vanmessagetopic][4], [vanmessagetopic][3]] ...]).
  468. [2019-01-25T10:43:24,484][INFO ][o.e.c.s.ClusterService ] [yRj9xlW] added {{wAVZzky}{wAVZzkyrSfyBmuGRw0Bltw}{Qc8kODytQreZFj_24t5O7g}{10.231.83.146}{10.231.83.146:9302},}, reason: zen-disco-node-join
  469. [2019-01-25T10:43:29,088][INFO ][o.e.c.r.a.AllocationService] [yRj9xlW] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[device][0]] ...]).
  470. [2019-01-25T10:44:30,199][INFO ][o.e.c.r.a.AllocationService] [yRj9xlW] Cluster health status changed from [GREEN] to [YELLOW] (reason: []).
  471. [2019-01-25T10:44:30,200][INFO ][o.e.c.s.ClusterService ] [yRj9xlW] removed {{lJgJwYh}{lJgJwYhmQA-fucaOBHD5Vg}{lbN7QcyiSWCOBYNL4jNkdg}{10.231.83.146}{10.231.83.146:9301},}, reason: zen-disco-node-failed({lJgJwYh}{lJgJwYhmQA-fucaOBHD5Vg}{lbN7QcyiSWCOBYNL4jNkdg}{10.231.83.146}{10.231.83.146:9301}), reason(failed to ping, tried [3] times, each with maximum [4s] timeout)
  472. [2019-01-25T10:44:38,008][DEBUG][o.e.a.a.c.n.s.TransportNodesStatsAction] [yRj9xlW] failed to execute on node [lJgJwYhmQA-fucaOBHD5Vg]
  473. org.elasticsearch.transport.ReceiveTimeoutTransportException: [lJgJwYh][10.231.83.146:9301][cluster:monitor/nodes/stats[n]] request_id [328] timed out after [15000ms]
  474. at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:961) [elasticsearch-5.6.9.jar:5.6.9]
  475. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.9.jar:5.6.9]
  476. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
  477. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
  478. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  479. [2019-01-25T10:44:38,007][DEBUG][o.e.a.a.c.n.s.TransportNodesStatsAction] [yRj9xlW] failed to execute on node [wAVZzkyrSfyBmuGRw0Bltw]
  480. org.elasticsearch.transport.ReceiveTimeoutTransportException: [wAVZzky][10.231.83.146:9302][cluster:monitor/nodes/stats[n]] request_id [329] timed out after [15000ms]
  481. at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:961) [elasticsearch-5.6.9.jar:5.6.9]
  482. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.9.jar:5.6.9]
  483. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
  484. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
  485. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  486. [2019-01-25T10:45:00,202][WARN ][o.e.c.s.ClusterService ] [yRj9xlW] failing [zen-disco-node-failed({lJgJwYh}{lJgJwYhmQA-fucaOBHD5Vg}{lbN7QcyiSWCOBYNL4jNkdg}{10.231.83.146}{10.231.83.146:9301}), reason(failed to ping, tried [3] times, each with maximum [4s] timeout)]: failed to commit cluster state version [200]
  487. org.elasticsearch.discovery.Discovery$FailedToCommitClusterStateException: timed out while waiting for enough masters to ack sent cluster state. [1] left
  488. at org.elasticsearch.discovery.zen.PublishClusterStateAction$SendingController.waitForCommit(PublishClusterStateAction.java:574) ~[elasticsearch-5.6.9.jar:5.6.9]
  489. at org.elasticsearch.discovery.zen.PublishClusterStateAction.innerPublish(PublishClusterStateAction.java:202) ~[elasticsearch-5.6.9.jar:5.6.9]
  490. at org.elasticsearch.discovery.zen.PublishClusterStateAction.publish(PublishClusterStateAction.java:167) ~[elasticsearch-5.6.9.jar:5.6.9]
  491. at org.elasticsearch.discovery.zen.ZenDiscovery.publish(ZenDiscovery.java:310) ~[elasticsearch-5.6.9.jar:5.6.9]
  492. at org.elasticsearch.cluster.service.ClusterService.publishAndApplyChanges(ClusterService.java:741) [elasticsearch-5.6.9.jar:5.6.9]
  493. at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:587) [elasticsearch-5.6.9.jar:5.6.9]
  494. at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) [elasticsearch-5.6.9.jar:5.6.9]
  495. at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-5.6.9.jar:5.6.9]
  496. at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-5.6.9.jar:5.6.9]
  497. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.9.jar:5.6.9]
  498. at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) [elasticsearch-5.6.9.jar:5.6.9]
  499. at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) [elasticsearch-5.6.9.jar:5.6.9]
  500. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
  501. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
  502. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  503. [2019-01-25T10:45:00,204][ERROR][o.e.d.z.ZenDiscovery ] [yRj9xlW] unexpected failure during [zen-disco-node-failed({lJgJwYh}{lJgJwYhmQA-fucaOBHD5Vg}{lbN7QcyiSWCOBYNL4jNkdg}{10.231.83.146}{10.231.83.146:9301}), reason(failed to ping, tried [3] times, each with maximum [4s] timeout)]
  504. org.elasticsearch.discovery.Discovery$FailedToCommitClusterStateException: timed out while waiting for enough masters to ack sent cluster state. [1] left
  505. at org.elasticsearch.discovery.zen.PublishClusterStateAction$SendingController.waitForCommit(PublishClusterStateAction.java:574) ~[elasticsearch-5.6.9.jar:5.6.9]
  506. at org.elasticsearch.discovery.zen.PublishClusterStateAction.innerPublish(PublishClusterStateAction.java:202) ~[elasticsearch-5.6.9.jar:5.6.9]
  507. at org.elasticsearch.discovery.zen.PublishClusterStateAction.publish(PublishClusterStateAction.java:167) ~[elasticsearch-5.6.9.jar:5.6.9]
  508. at org.elasticsearch.discovery.zen.ZenDiscovery.publish(ZenDiscovery.java:310) ~[elasticsearch-5.6.9.jar:5.6.9]
  509. at org.elasticsearch.cluster.service.ClusterService.publishAndApplyChanges(ClusterService.java:741) [elasticsearch-5.6.9.jar:5.6.9]
  510. at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:587) [elasticsearch-5.6.9.jar:5.6.9]
  511. at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) [elasticsearch-5.6.9.jar:5.6.9]
  512. at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-5.6.9.jar:5.6.9]
  513. at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-5.6.9.jar:5.6.9]
  514. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.9.jar:5.6.9]
  515. at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) [elasticsearch-5.6.9.jar:5.6.9]
  516. at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) [elasticsearch-5.6.9.jar:5.6.9]
  517. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
  518. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
  519. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  520. [2019-01-25T10:45:00,204][WARN ][o.e.c.s.ClusterService ] [yRj9xlW] cluster state update task [zen-disco-node-failed({lJgJwYh}{lJgJwYhmQA-fucaOBHD5Vg}{lbN7QcyiSWCOBYNL4jNkdg}{10.231.83.146}{10.231.83.146:9301}), reason(failed to ping, tried [3] times, each with maximum [4s] timeout)] took [30s] above the warn threshold of 30s
  521. [2019-01-25T10:45:00,208][INFO ][o.e.c.r.a.AllocationService] [yRj9xlW] Cluster health status changed from [GREEN] to [YELLOW] (reason: []).
  522. [2019-01-25T10:45:00,209][INFO ][o.e.c.s.ClusterService ] [yRj9xlW] removed {{wAVZzky}{wAVZzkyrSfyBmuGRw0Bltw}{Qc8kODytQreZFj_24t5O7g}{10.231.83.146}{10.231.83.146:9302},}, reason: zen-disco-node-failed({wAVZzky}{wAVZzkyrSfyBmuGRw0Bltw}{Qc8kODytQreZFj_24t5O7g}{10.231.83.146}{10.231.83.146:9302}), reason(failed to ping, tried [3] times, each with maximum [4s] timeout)
  523. [2019-01-25T10:45:21,837][DEBUG][o.e.a.a.i.s.TransportIndicesStatsAction] [yRj9xlW] failed to execute [indices:monitor/stats] on node [wAVZzkyrSfyBmuGRw0Bltw]
  524. org.elasticsearch.transport.NodeDisconnectedException: [wAVZzky][10.231.83.146:9302][indices:monitor/stats[n]] disconnected
  525. [2019-01-25T10:45:21,840][WARN ][o.e.g.GatewayAllocator$InternalReplicaShardAllocator] [yRj9xlW] [vanmessagetopic][1]: failed to list shard for shard_store on node [wAVZzkyrSfyBmuGRw0Bltw]
  526. org.elasticsearch.action.FailedNodeException: Failed node [wAVZzkyrSfyBmuGRw0Bltw]
  527. at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:239) ~[elasticsearch-5.6.9.jar:5.6.9]
  528. at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$200(TransportNodesAction.java:153) ~[elasticsearch-5.6.9.jar:5.6.9]
  529. at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:211) ~[elasticsearch-5.6.9.jar:5.6.9]
  530. at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1077) ~[elasticsearch-5.6.9.jar:5.6.9]
  531. at org.elasticsearch.transport.TransportService$Adapter.lambda$onConnectionClosed$6(TransportService.java:903) ~[elasticsearch-5.6.9.jar:5.6.9]
  532. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.9.jar:5.6.9]
  533. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
  534. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
  535. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  536. Caused by: org.elasticsearch.transport.NodeDisconnectedException: [wAVZzky][10.231.83.146:9302][internal:cluster/nodes/indices/shard/store[n]] disconnected
  537. [2019-01-25T10:45:21,947][WARN ][o.e.t.n.Netty4Transport ] [yRj9xlW] write and flush on the network layer failed (channel: [id: 0x237db2db, L:/10.231.83.238:9300 ! R:/10.231.83.146:51160])
  538. java.io.IOException: Broken pipe
  539. at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[?:?]
  540. at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[?:?]
  541. at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[?:?]
  542. at sun.nio.ch.IOUtil.write(IOUtil.java:51) ~[?:?]
  543. at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) ~[?:?]
  544. at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:417) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  545. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:856) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  546. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:362) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  547. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:823) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  548. at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1296) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  549. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  550. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  551. at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  552. at io.netty.handler.logging.LoggingHandler.flush(LoggingHandler.java:265) ~[netty-handler-4.1.13.Final.jar:4.1.13.Final]
  553. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  554. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  555. at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  556. at io.netty.channel.ChannelDuplexHandler.flush(ChannelDuplexHandler.java:117) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  557. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  558. at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  559. at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  560. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  561. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  562. at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1032) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  563. at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:296) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  564. at org.elasticsearch.transport.netty4.Netty4Transport.sendMessage(Netty4Transport.java:400) [transport-netty4-5.6.9.jar:5.6.9]
  565. at org.elasticsearch.transport.netty4.Netty4Transport.sendMessage(Netty4Transport.java:96) [transport-netty4-5.6.9.jar:5.6.9]
  566. at org.elasticsearch.transport.TcpTransport.internalSendMessage(TcpTransport.java:1138) [elasticsearch-5.6.9.jar:5.6.9]
  567. at org.elasticsearch.transport.TcpTransport.sendResponse(TcpTransport.java:1205) [elasticsearch-5.6.9.jar:5.6.9]
  568. at org.elasticsearch.transport.TcpTransport.sendResponse(TcpTransport.java:1181) [elasticsearch-5.6.9.jar:5.6.9]
  569. at org.elasticsearch.transport.TcpTransportChannel.sendResponse(TcpTransportChannel.java:67) [elasticsearch-5.6.9.jar:5.6.9]
  570. at org.elasticsearch.transport.TcpTransportChannel.sendResponse(TcpTransportChannel.java:61) [elasticsearch-5.6.9.jar:5.6.9]
  571. at org.elasticsearch.transport.DelegatingTransportChannel.sendResponse(DelegatingTransportChannel.java:60) [elasticsearch-5.6.9.jar:5.6.9]
  572. at org.elasticsearch.transport.RequestHandlerRegistry$TransportChannelWrapper.sendResponse(RequestHandlerRegistry.java:111) [elasticsearch-5.6.9.jar:5.6.9]
  573. at org.elasticsearch.discovery.zen.UnicastZenPing$UnicastPingRequestHandler.messageReceived(UnicastZenPing.java:616) [elasticsearch-5.6.9.jar:5.6.9]
  574. at org.elasticsearch.discovery.zen.UnicastZenPing$UnicastPingRequestHandler.messageReceived(UnicastZenPing.java:608) [elasticsearch-5.6.9.jar:5.6.9]
  575. at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33) [elasticsearch-5.6.9.jar:5.6.9]
  576. at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) [elasticsearch-5.6.9.jar:5.6.9]
  577. at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1556) [elasticsearch-5.6.9.jar:5.6.9]
  578. at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.9.jar:5.6.9]
  579. at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:110) [elasticsearch-5.6.9.jar:5.6.9]
  580. at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1513) [elasticsearch-5.6.9.jar:5.6.9]
  581. at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1396) [elasticsearch-5.6.9.jar:5.6.9]
  582. at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:75) [transport-netty4-5.6.9.jar:5.6.9]
  583. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  584. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  585. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  586. at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.13.Final.jar:4.1.13.Final]
  587. at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297) [netty-codec-4.1.13.Final.jar:4.1.13.Final]
  588. at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413) [netty-codec-4.1.13.Final.jar:4.1.13.Final]
  589. at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) [netty-codec-4.1.13.Final.jar:4.1.13.Final]
  590. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  591. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  592. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  593. at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  594. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  595. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  596. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  597. at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) [netty-handler-4.1.13.Final.jar:4.1.13.Final]
  598. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  599. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  600. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  601. at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  602. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  603. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  604. at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  605. at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  606. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  607. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  608. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  609. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  610. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.13.Final.jar:4.1.13.Final]
  611. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  612. [2019-01-25T10:45:23,010][DEBUG][o.e.a.a.c.n.s.TransportNodesStatsAction] [yRj9xlW] failed to execute on node [wAVZzkyrSfyBmuGRw0Bltw]
  613. org.elasticsearch.transport.NodeNotConnectedException: [wAVZzky][10.231.83.146:9302] Node not connected
  614. at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:640) ~[elasticsearch-5.6.9.jar:5.6.9]
  615. at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:117) ~[elasticsearch-5.6.9.jar:5.6.9]
  616. at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:540) ~[elasticsearch-5.6.9.jar:5.6.9]
  617. at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:516) ~[elasticsearch-5.6.9.jar:5.6.9]
  618. at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.start(TransportNodesAction.java:197) ~[elasticsearch-5.6.9.jar:5.6.9]
  619. at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:89) ~[elasticsearch-5.6.9.jar:5.6.9]
  620. at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:52) ~[elasticsearch-5.6.9.jar:5.6.9]
  621. at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) ~[elasticsearch-5.6.9.jar:5.6.9]
  622. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) ~[elasticsearch-5.6.9.jar:5.6.9]
  623. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84) ~[elasticsearch-5.6.9.jar:5.6.9]
  624. at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) ~[elasticsearch-5.6.9.jar:5.6.9]
  625. at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) ~[elasticsearch-5.6.9.jar:5.6.9]
  626. at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408) ~[elasticsearch-5.6.9.jar:5.6.9]
  627. at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:730) ~[elasticsearch-5.6.9.jar:5.6.9]
  628. at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.nodesStats(AbstractClient.java:826) ~[elasticsearch-5.6.9.jar:5.6.9]
  629. at org.elasticsearch.cluster.InternalClusterInfoService.updateNodeStats(InternalClusterInfoService.java:256) ~[elasticsearch-5.6.9.jar:5.6.9]
  630. at org.elasticsearch.cluster.InternalClusterInfoService.refresh(InternalClusterInfoService.java:292) ~[elasticsearch-5.6.9.jar:5.6.9]
  631. at org.elasticsearch.cluster.InternalClusterInfoService.maybeRefresh(InternalClusterInfoService.java:277) ~[elasticsearch-5.6.9.jar:5.6.9]
  632. at org.elasticsearch.cluster.InternalClusterInfoService.access$500(InternalClusterInfoService.java:67) ~[elasticsearch-5.6.9.jar:5.6.9]
  633. at org.elasticsearch.cluster.InternalClusterInfoService$SubmitReschedulingClusterInfoUpdatedJob.lambda$run$0(InternalClusterInfoService.java:224) ~[elasticsearch-5.6.9.jar:5.6.9]
  634. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.9.jar:5.6.9]
  635. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
  636. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
  637. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  638. [2019-01-25T10:45:23,014][DEBUG][o.e.a.a.i.s.TransportIndicesStatsAction] [yRj9xlW] failed to execute [indices:monitor/stats] on node [wAVZzkyrSfyBmuGRw0Bltw]
  639. org.elasticsearch.transport.NodeNotConnectedException: [wAVZzky][10.231.83.146:9302] Node not connected
  640. at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:640) ~[elasticsearch-5.6.9.jar:5.6.9]
  641. at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:117) ~[elasticsearch-5.6.9.jar:5.6.9]
  642. at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:540) ~[elasticsearch-5.6.9.jar:5.6.9]
  643. at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:503) ~[elasticsearch-5.6.9.jar:5.6.9]
  644. at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.sendNodeRequest(TransportBroadcastByNodeAction.java:322) ~[elasticsearch-5.6.9.jar:5.6.9]
  645. at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.start(TransportBroadcastByNodeAction.java:311) ~[elasticsearch-5.6.9.jar:5.6.9]
  646. at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:234) ~[elasticsearch-5.6.9.jar:5.6.9]
  647. at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:79) ~[elasticsearch-5.6.9.jar:5.6.9]
  648. at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) ~[elasticsearch-5.6.9.jar:5.6.9]
  649. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) ~[elasticsearch-5.6.9.jar:5.6.9]
  650. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84) ~[elasticsearch-5.6.9.jar:5.6.9]
  651. at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) ~[elasticsearch-5.6.9.jar:5.6.9]
  652. at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) ~[elasticsearch-5.6.9.jar:5.6.9]
  653. at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408) ~[elasticsearch-5.6.9.jar:5.6.9]
  654. at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1256) ~[elasticsearch-5.6.9.jar:5.6.9]
  655. at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.stats(AbstractClient.java:1577) ~[elasticsearch-5.6.9.jar:5.6.9]
  656. at org.elasticsearch.cluster.InternalClusterInfoService.updateIndicesStats(InternalClusterInfoService.java:270) ~[elasticsearch-5.6.9.jar:5.6.9]
  657. at org.elasticsearch.cluster.InternalClusterInfoService.refresh(InternalClusterInfoService.java:321) ~[elasticsearch-5.6.9.jar:5.6.9]
  658. at org.elasticsearch.cluster.InternalClusterInfoService.maybeRefresh(InternalClusterInfoService.java:277) ~[elasticsearch-5.6.9.jar:5.6.9]
  659. at org.elasticsearch.cluster.InternalClusterInfoService.access$500(InternalClusterInfoService.java:67) ~[elasticsearch-5.6.9.jar:5.6.9]
  660. at org.elasticsearch.cluster.InternalClusterInfoService$SubmitReschedulingClusterInfoUpdatedJob.lambda$run$0(InternalClusterInfoService.java:224) ~[elasticsearch-5.6.9.jar:5.6.9]
  661. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.9.jar:5.6.9]
  662. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
  663. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
  664. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  665. [2019-01-25T10:45:23,026][WARN ][o.e.t.TransportService ] [yRj9xlW] Received response for a request that has timed out, sent [60019ms] ago, timed out [45019ms] ago, action [cluster:monitor/nodes/stats[n]], node [{lJgJwYh}{lJgJwYhmQA-fucaOBHD5Vg}{lbN7QcyiSWCOBYNL4jNkdg}{10.231.83.146}{10.231.83.146:9301}], id [328]
  666. ^C[2019-01-25T10:45:24,254][INFO ][o.e.n.Node ] [yRj9xlW] stopping ...
  667. [2019-01-25T10:45:24,515][DEBUG][o.e.a.a.i.s.TransportIndicesStatsAction] [yRj9xlW] failed to execute [indices:monitor/stats] on node [lJgJwYhmQA-fucaOBHD5Vg]
  668. org.elasticsearch.transport.NodeDisconnectedException: [lJgJwYh][10.231.83.146:9301][indices:monitor/stats[n]] disconnected
  669. [2019-01-25T10:45:24,518][WARN ][o.e.t.n.Netty4Transport ] [yRj9xlW] write and flush on the network layer failed (channel: [id: 0xdc753be3, L:/10.231.83.238:9300 ! R:/10.231.83.146:51098])
  670. java.nio.channels.ClosedChannelException: null
  671. at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
  672. [2019-01-25T10:45:24,515][WARN ][o.e.g.GatewayAllocator$InternalReplicaShardAllocator] [yRj9xlW] [vanmessagetopic][3]: failed to list shard for shard_store on node [lJgJwYhmQA-fucaOBHD5Vg]
  673. org.elasticsearch.action.FailedNodeException: Failed node [lJgJwYhmQA-fucaOBHD5Vg]
  674. at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:239) ~[elasticsearch-5.6.9.jar:5.6.9]
  675. at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$200(TransportNodesAction.java:153) ~[elasticsearch-5.6.9.jar:5.6.9]
  676. at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:211) ~[elasticsearch-5.6.9.jar:5.6.9]
  677. at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1077) ~[elasticsearch-5.6.9.jar:5.6.9]
  678. at org.elasticsearch.transport.TransportService$Adapter.lambda$onConnectionClosed$6(TransportService.java:903) ~[elasticsearch-5.6.9.jar:5.6.9]
  679. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.9.jar:5.6.9]
  680. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
  681. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
  682. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  683. Caused by: org.elasticsearch.transport.NodeDisconnectedException: [lJgJwYh][10.231.83.146:9301][internal:cluster/nodes/indices/shard/store[n]] disconnected
  684. [2019-01-25T10:45:24,519][WARN ][o.e.t.n.Netty4Transport ] [yRj9xlW] write and flush on the network layer failed (channel: [id: 0x3fc1eeb3, L:/10.231.83.238:9300 ! R:/10.231.83.146:51100])
  685. java.nio.channels.ClosedChannelException: null
  686. at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
  687. [2019-01-25T10:45:24,522][WARN ][o.e.g.GatewayAllocator$InternalReplicaShardAllocator] [yRj9xlW] [vanmessagetopic][2]: failed to list shard for shard_store on node [lJgJwYhmQA-fucaOBHD5Vg]
  688. org.elasticsearch.action.FailedNodeException: Failed node [lJgJwYhmQA-fucaOBHD5Vg]
  689. at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:239) ~[elasticsearch-5.6.9.jar:5.6.9]
  690. at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$200(TransportNodesAction.java:153) ~[elasticsearch-5.6.9.jar:5.6.9]
  691. at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:211) ~[elasticsearch-5.6.9.jar:5.6.9]
  692. at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1077) ~[elasticsearch-5.6.9.jar:5.6.9]
  693. at org.elasticsearch.transport.TransportService$Adapter.lambda$onConnectionClosed$6(TransportService.java:903) ~[elasticsearch-5.6.9.jar:5.6.9]
  694. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.9.jar:5.6.9]
  695. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
  696. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
  697. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  698. Caused by: org.elasticsearch.transport.NodeDisconnectedException: [lJgJwYh][10.231.83.146:9301][internal:cluster/nodes/indices/shard/store[n]] disconnected
  699. [2019-01-25T10:45:24,525][WARN ][o.e.c.s.ClusterService ] [yRj9xlW] failing [zen-disco-node-failed({wAVZzky}{wAVZzkyrSfyBmuGRw0Bltw}{Qc8kODytQreZFj_24t5O7g}{10.231.83.146}{10.231.83.146:9302}), reason(failed to ping, tried [3] times, each with maximum [4s] timeout)]: failed to commit cluster state version [200]
  700. org.elasticsearch.discovery.Discovery$FailedToCommitClusterStateException: failed to get enough masters to ack sent cluster state. [1] left
  701. at org.elasticsearch.discovery.zen.PublishClusterStateAction$SendingController.waitForCommit(PublishClusterStateAction.java:574) ~[elasticsearch-5.6.9.jar:5.6.9]
  702. at org.elasticsearch.discovery.zen.PublishClusterStateAction.innerPublish(PublishClusterStateAction.java:202) ~[elasticsearch-5.6.9.jar:5.6.9]
  703. at org.elasticsearch.discovery.zen.PublishClusterStateAction.publish(PublishClusterStateAction.java:167) ~[elasticsearch-5.6.9.jar:5.6.9]
  704. at org.elasticsearch.discovery.zen.ZenDiscovery.publish(ZenDiscovery.java:310) ~[elasticsearch-5.6.9.jar:5.6.9]
  705. at org.elasticsearch.cluster.service.ClusterService.publishAndApplyChanges(ClusterService.java:741) [elasticsearch-5.6.9.jar:5.6.9]
  706. at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:587) [elasticsearch-5.6.9.jar:5.6.9]
  707. at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) [elasticsearch-5.6.9.jar:5.6.9]
  708. at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-5.6.9.jar:5.6.9]
  709. at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-5.6.9.jar:5.6.9]
  710. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.9.jar:5.6.9]
  711. at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) [elasticsearch-5.6.9.jar:5.6.9]
  712. at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) [elasticsearch-5.6.9.jar:5.6.9]
  713. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
  714. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
  715. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  716. [2019-01-25T10:45:24,526][WARN ][o.e.g.GatewayAllocator$InternalReplicaShardAllocator] [yRj9xlW] [vanmessagetopic][0]: failed to list shard for shard_store on node [lJgJwYhmQA-fucaOBHD5Vg]
  717. org.elasticsearch.action.FailedNodeException: Failed node [lJgJwYhmQA-fucaOBHD5Vg]
  718. at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:239) ~[elasticsearch-5.6.9.jar:5.6.9]
  719. at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$200(TransportNodesAction.java:153) ~[elasticsearch-5.6.9.jar:5.6.9]
  720. at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:211) ~[elasticsearch-5.6.9.jar:5.6.9]
  721. at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1077) ~[elasticsearch-5.6.9.jar:5.6.9]
  722. at org.elasticsearch.transport.TransportService$Adapter.lambda$onConnectionClosed$6(TransportService.java:903) ~[elasticsearch-5.6.9.jar:5.6.9]
  723. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.9.jar:5.6.9]
  724. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
  725. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
  726. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  727. Caused by: org.elasticsearch.transport.NodeDisconnectedException: [lJgJwYh][10.231.83.146:9301][internal:cluster/nodes/indices/shard/store[n]] disconnected
  728. [2019-01-25T10:45:24,521][WARN ][o.e.t.n.Netty4Transport ] [yRj9xlW] write and flush on the network layer failed (channel: [id: 0x80fc56a7, L:/10.231.83.238:9300 ! R:/10.231.83.146:51090])
  729. java.io.IOException: Broken pipe
  730. at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[?:?]
  731. at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[?:?]
  732. at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[?:?]
  733. at sun.nio.ch.IOUtil.write(IOUtil.java:51) ~[?:?]
  734. at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) ~[?:?]
  735. at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:417) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  736. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:856) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  737. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:362) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  738. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:823) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  739. at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1296) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  740. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  741. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  742. at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  743. at io.netty.handler.logging.LoggingHandler.flush(LoggingHandler.java:265) ~[netty-handler-4.1.13.Final.jar:4.1.13.Final]
  744. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  745. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  746. at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  747. at io.netty.channel.ChannelDuplexHandler.flush(ChannelDuplexHandler.java:117) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  748. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  749. at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  750. at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  751. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  752. at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  753. at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1032) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  754. at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:296) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  755. at org.elasticsearch.transport.netty4.Netty4Transport.sendMessage(Netty4Transport.java:400) [transport-netty4-5.6.9.jar:5.6.9]
  756. at org.elasticsearch.transport.netty4.Netty4Transport.sendMessage(Netty4Transport.java:96) [transport-netty4-5.6.9.jar:5.6.9]
  757. at org.elasticsearch.transport.TcpTransport.internalSendMessage(TcpTransport.java:1138) [elasticsearch-5.6.9.jar:5.6.9]
  758. at org.elasticsearch.transport.TcpTransport.sendErrorResponse(TcpTransport.java:1170) [elasticsearch-5.6.9.jar:5.6.9]
  759. at org.elasticsearch.transport.TcpTransportChannel.sendResponse(TcpTransportChannel.java:76) [elasticsearch-5.6.9.jar:5.6.9]
  760. at org.elasticsearch.transport.TcpTransport$RequestHandler.onFailure(TcpTransport.java:1569) [elasticsearch-5.6.9.jar:5.6.9]
  761. at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39) [elasticsearch-5.6.9.jar:5.6.9]
  762. at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:110) [elasticsearch-5.6.9.jar:5.6.9]
  763. at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1513) [elasticsearch-5.6.9.jar:5.6.9]
  764. at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1396) [elasticsearch-5.6.9.jar:5.6.9]
  765. at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:75) [transport-netty4-5.6.9.jar:5.6.9]
  766. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  767. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  768. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  769. at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.13.Final.jar:4.1.13.Final]
  770. at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297) [netty-codec-4.1.13.Final.jar:4.1.13.Final]
  771. at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413) [netty-codec-4.1.13.Final.jar:4.1.13.Final]
  772. at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) [netty-codec-4.1.13.Final.jar:4.1.13.Final]
  773. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  774. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  775. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  776. at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  777. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  778. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  779. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  780. at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) [netty-handler-4.1.13.Final.jar:4.1.13.Final]
  781. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  782. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  783. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  784. at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  785. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  786. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  787. at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  788. at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  789. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  790. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  791. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  792. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  793. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.13.Final.jar:4.1.13.Final]
  794. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  795. [2019-01-25T10:45:24,536][WARN ][o.e.c.NodeConnectionsService] [yRj9xlW] failed to connect to node {wAVZzky}{wAVZzkyrSfyBmuGRw0Bltw}{Qc8kODytQreZFj_24t5O7g}{10.231.83.146}{10.231.83.146:9302} (tried [1] times)
  796. org.elasticsearch.transport.ConnectTransportException: [wAVZzky][10.231.83.146:9302] connect_timeout[30s]
  797. at org.elasticsearch.transport.netty4.Netty4Transport.connectToChannels(Netty4Transport.java:363) ~[?:?]
  798. at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:570) ~[elasticsearch-5.6.9.jar:5.6.9]
  799. at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:473) ~[elasticsearch-5.6.9.jar:5.6.9]
  800. at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:342) ~[elasticsearch-5.6.9.jar:5.6.9]
  801. at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:329) ~[elasticsearch-5.6.9.jar:5.6.9]
  802. at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:154) [elasticsearch-5.6.9.jar:5.6.9]
  803. at org.elasticsearch.cluster.NodeConnectionsService$1.doRun(NodeConnectionsService.java:107) [elasticsearch-5.6.9.jar:5.6.9]
  804. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:674) [elasticsearch-5.6.9.jar:5.6.9]
  805. at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.9.jar:5.6.9]
  806. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
  807. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
  808. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  809. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: 10.231.83.146/10.231.83.146:9302
  810. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  811. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
  812. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352) ~[?:?]
  813. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  814. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?]
  815. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?]
  816. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?]
  817. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?]
  818. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
  819. ... 1 more
  820. Caused by: java.net.ConnectException: Connection refused
  821. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  822. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
  823. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352) ~[?:?]
  824. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  825. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?]
  826. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?]
  827. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?]
  828. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?]
  829. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
  830. ... 1 more
  831. [2019-01-25T10:45:24,557][WARN ][o.e.t.n.Netty4Transport ] [yRj9xlW] write and flush on the network layer failed (channel: [id: 0xe0ce28dc, L:/10.231.83.238:9300 ! R:/10.231.83.146:51106])
  832. java.io.IOException: Broken pipe
  833. at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[?:?]
  834. at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[?:?]
  835. at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[?:?]
  836. at sun.nio.ch.IOUtil.write(IOUtil.java:51) ~[?:?]
  837. at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) ~[?:?]
  838. at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:417) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  839. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:856) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  840. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  841. at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:823) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  842. at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1296) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  843. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  844. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  845. at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  846. at io.netty.handler.logging.LoggingHandler.flush(LoggingHandler.java:265) [netty-handler-4.1.13.Final.jar:4.1.13.Final]
  847. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  848. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  849. at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  850. at io.netty.channel.ChannelDuplexHandler.flush(ChannelDuplexHandler.java:117) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  851. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  852. at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  853. at io.netty.channel.AbstractChannelHandlerContext.access$1500(AbstractChannelHandlerContext.java:38) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  854. at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1137) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  855. at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:1078) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  856. at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [netty-common-4.1.13.Final.jar:4.1.13.Final]
  857. at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403) [netty-common-4.1.13.Final.jar:4.1.13.Final]
  858. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  859. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.13.Final.jar:4.1.13.Final]
  860. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  861. [2019-01-25T10:45:24,543][WARN ][o.e.c.NodeConnectionsService] [yRj9xlW] failed to connect to node {lJgJwYh}{lJgJwYhmQA-fucaOBHD5Vg}{lbN7QcyiSWCOBYNL4jNkdg}{10.231.83.146}{10.231.83.146:9301} (tried [1] times)
  862. org.elasticsearch.transport.ConnectTransportException: [lJgJwYh][10.231.83.146:9301] connect_timeout[30s]
  863. at org.elasticsearch.transport.netty4.Netty4Transport.connectToChannels(Netty4Transport.java:363) ~[?:?]
  864. at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:570) ~[elasticsearch-5.6.9.jar:5.6.9]
  865. at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:473) ~[elasticsearch-5.6.9.jar:5.6.9]
  866. at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:342) ~[elasticsearch-5.6.9.jar:5.6.9]
  867. at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:329) ~[elasticsearch-5.6.9.jar:5.6.9]
  868. at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:154) [elasticsearch-5.6.9.jar:5.6.9]
  869. at org.elasticsearch.cluster.NodeConnectionsService$1.doRun(NodeConnectionsService.java:107) [elasticsearch-5.6.9.jar:5.6.9]
  870. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:674) [elasticsearch-5.6.9.jar:5.6.9]
  871. at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.9.jar:5.6.9]
  872. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
  873. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
  874. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  875. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: 10.231.83.146/10.231.83.146:9301
  876. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  877. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
  878. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352) ~[?:?]
  879. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  880. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?]
  881. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?]
  882. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?]
  883. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?]
  884. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
  885. ... 1 more
  886. Caused by: java.net.ConnectException: Connection refused
  887. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  888. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
  889. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352) ~[?:?]
  890. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  891. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?]
  892. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?]
  893. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?]
  894. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?]
  895. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
  896. ... 1 more
  897. [2019-01-25T10:45:24,562][ERROR][o.e.d.z.ZenDiscovery ] [yRj9xlW] unexpected failure during [zen-disco-node-failed({wAVZzky}{wAVZzkyrSfyBmuGRw0Bltw}{Qc8kODytQreZFj_24t5O7g}{10.231.83.146}{10.231.83.146:9302}), reason(failed to ping, tried [3] times, each with maximum [4s] timeout)]
  898. org.elasticsearch.discovery.Discovery$FailedToCommitClusterStateException: failed to get enough masters to ack sent cluster state. [1] left
  899. at org.elasticsearch.discovery.zen.PublishClusterStateAction$SendingController.waitForCommit(PublishClusterStateAction.java:574) ~[elasticsearch-5.6.9.jar:5.6.9]
  900. at org.elasticsearch.discovery.zen.PublishClusterStateAction.innerPublish(PublishClusterStateAction.java:202) ~[elasticsearch-5.6.9.jar:5.6.9]
  901. at org.elasticsearch.discovery.zen.PublishClusterStateAction.publish(PublishClusterStateAction.java:167) ~[elasticsearch-5.6.9.jar:5.6.9]
  902. at org.elasticsearch.discovery.zen.ZenDiscovery.publish(ZenDiscovery.java:310) ~[elasticsearch-5.6.9.jar:5.6.9]
  903. at org.elasticsearch.cluster.service.ClusterService.publishAndApplyChanges(ClusterService.java:741) [elasticsearch-5.6.9.jar:5.6.9]
  904. at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:587) [elasticsearch-5.6.9.jar:5.6.9]
  905. at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) [elasticsearch-5.6.9.jar:5.6.9]
  906. at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-5.6.9.jar:5.6.9]
  907. at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-5.6.9.jar:5.6.9]
  908. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.9.jar:5.6.9]
  909. at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) [elasticsearch-5.6.9.jar:5.6.9]
  910. at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) [elasticsearch-5.6.9.jar:5.6.9]
  911. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
  912. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
  913. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  914. [2019-01-25T10:45:24,586][INFO ][o.e.n.Node ] [yRj9xlW] stopped
  915. [2019-01-25T10:45:24,586][INFO ][o.e.n.Node ] [yRj9xlW] closing ...
  916. [2019-01-25T10:45:24,592][INFO ][o.e.n.Node ] [yRj9xlW] closed
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!