SHARE
TWEET

Untitled

a guest Jan 25th, 2019 72 Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
  1. [2019-01-25T10:42:42,212][INFO ][o.e.n.Node               ] [] initializing ...
  2. [2019-01-25T10:42:42,318][INFO ][o.e.e.NodeEnvironment    ] [yRj9xlW] using [1] data paths, mounts [[/home (/dev/mapper/centos_vanng--t570-home)]], net usable_space [145.6gb], net total_space [179.8gb], spins? [no], types [xfs]
  3. [2019-01-25T10:42:42,319][INFO ][o.e.e.NodeEnvironment    ] [yRj9xlW] heap size [1.9gb], compressed ordinary object pointers [true]
  4. [2019-01-25T10:42:42,335][INFO ][o.e.n.Node               ] node name [yRj9xlW] derived from node ID [yRj9xlW8Q-WgQ0j6H4DbOQ]; set [node.name] to override
  5. [2019-01-25T10:42:42,335][INFO ][o.e.n.Node               ] version[5.6.9], pid[23706], build[877a590/2018-04-12T16:25:14.838Z], OS[Linux/3.10.0-862.3.3.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_181/25.181-b13]
  6. [2019-01-25T10:42:42,336][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/home/vanng/elasticsearch-5.6.9-1]
  7. [2019-01-25T10:42:43,018][INFO ][o.e.p.PluginsService     ] [yRj9xlW] loaded module [aggs-matrix-stats]
  8. [2019-01-25T10:42:43,018][INFO ][o.e.p.PluginsService     ] [yRj9xlW] loaded module [ingest-common]
  9. [2019-01-25T10:42:43,018][INFO ][o.e.p.PluginsService     ] [yRj9xlW] loaded module [lang-expression]
  10. [2019-01-25T10:42:43,018][INFO ][o.e.p.PluginsService     ] [yRj9xlW] loaded module [lang-groovy]
  11. [2019-01-25T10:42:43,018][INFO ][o.e.p.PluginsService     ] [yRj9xlW] loaded module [lang-mustache]
  12. [2019-01-25T10:42:43,018][INFO ][o.e.p.PluginsService     ] [yRj9xlW] loaded module [lang-painless]
  13. [2019-01-25T10:42:43,019][INFO ][o.e.p.PluginsService     ] [yRj9xlW] loaded module [parent-join]
  14. [2019-01-25T10:42:43,019][INFO ][o.e.p.PluginsService     ] [yRj9xlW] loaded module [percolator]
  15. [2019-01-25T10:42:43,019][INFO ][o.e.p.PluginsService     ] [yRj9xlW] loaded module [reindex]
  16. [2019-01-25T10:42:43,019][INFO ][o.e.p.PluginsService     ] [yRj9xlW] loaded module [transport-netty3]
  17. [2019-01-25T10:42:43,019][INFO ][o.e.p.PluginsService     ] [yRj9xlW] loaded module [transport-netty4]
  18. [2019-01-25T10:42:43,020][INFO ][o.e.p.PluginsService     ] [yRj9xlW] no plugins loaded
  19. [2019-01-25T10:42:44,295][INFO ][o.e.d.DiscoveryModule    ] [yRj9xlW] using discovery type [zen]
  20. [2019-01-25T10:42:44,787][INFO ][o.e.n.Node               ] initialized
  21. [2019-01-25T10:42:44,788][INFO ][o.e.n.Node               ] [yRj9xlW] starting ...
  22. [2019-01-25T10:43:02,176][INFO ][o.e.t.TransportService   ] [yRj9xlW] publish_address {10.231.83.238:9300}, bound_addresses {10.231.83.238:9300}
  23. [2019-01-25T10:43:02,188][INFO ][o.e.b.BootstrapChecks    ] [yRj9xlW] bound or publishing to a non-loopback address, enforcing bootstrap checks
  24. [2019-01-25T10:43:05,348][INFO ][o.e.c.s.ClusterService   ] [yRj9xlW] detected_master {lJgJwYh}{lJgJwYhmQA-fucaOBHD5Vg}{ho76tglrRPKjA8ui_UVJSg}{10.231.83.146}{10.231.83.146:9301}, added {{wAVZzky}{wAVZzkyrSfyBmuGRw0Bltw}{r1L7UXpESgiM_Kxq_jFUjw}{10.231.83.146}{10.231.83.146:9302},{lJgJwYh}{lJgJwYhmQA-fucaOBHD5Vg}{ho76tglrRPKjA8ui_UVJSg}{10.231.83.146}{10.231.83.146:9301},}, reason: zen-disco-receive(from master [master {lJgJwYh}{lJgJwYhmQA-fucaOBHD5Vg}{ho76tglrRPKjA8ui_UVJSg}{10.231.83.146}{10.231.83.146:9301} committed version [173]])
  25. [2019-01-25T10:43:05,454][INFO ][o.e.h.n.Netty4HttpServerTransport] [yRj9xlW] publish_address {10.231.83.238:9200}, bound_addresses {10.231.83.238:9200}
  26. [2019-01-25T10:43:05,454][INFO ][o.e.n.Node               ] [yRj9xlW] started
  27. [2019-01-25T10:43:12,258][INFO ][o.e.d.z.ZenDiscovery     ] [yRj9xlW] master_left [{lJgJwYh}{lJgJwYhmQA-fucaOBHD5Vg}{ho76tglrRPKjA8ui_UVJSg}{10.231.83.146}{10.231.83.146:9301}], reason [shut_down]
  28. [2019-01-25T10:43:12,260][WARN ][o.e.d.z.ZenDiscovery     ] [yRj9xlW] master left (reason = shut_down), current nodes: nodes:
  29.    {wAVZzky}{wAVZzkyrSfyBmuGRw0Bltw}{r1L7UXpESgiM_Kxq_jFUjw}{10.231.83.146}{10.231.83.146:9302}
  30.    {yRj9xlW}{yRj9xlW8Q-WgQ0j6H4DbOQ}{SAn1BH7ES42OhsAknxVe0Q}{10.231.83.238}{10.231.83.238:9300}, local
  31.    {lJgJwYh}{lJgJwYhmQA-fucaOBHD5Vg}{ho76tglrRPKjA8ui_UVJSg}{10.231.83.146}{10.231.83.146:9301}, master
  32.  
  33. [2019-01-25T10:43:12,279][WARN ][o.e.t.n.Netty4Transport  ] [yRj9xlW] write and flush on the network layer failed (channel: [id: 0x6a653f36, L:0.0.0.0/0.0.0.0:9300 ! R:/10.231.83.146:50906])
  34. java.nio.channels.ClosedChannelException: null
  35.     at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
  36. [2019-01-25T10:43:14,855][WARN ][o.e.c.NodeConnectionsService] [yRj9xlW] failed to connect to node {lJgJwYh}{lJgJwYhmQA-fucaOBHD5Vg}{ho76tglrRPKjA8ui_UVJSg}{10.231.83.146}{10.231.83.146:9301} (tried [1] times)
  37. org.elasticsearch.transport.ConnectTransportException: [lJgJwYh][10.231.83.146:9301] connect_timeout[30s]
  38.     at org.elasticsearch.transport.netty4.Netty4Transport.connectToChannels(Netty4Transport.java:363) ~[?:?]
  39.     at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:570) ~[elasticsearch-5.6.9.jar:5.6.9]
  40.     at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:473) ~[elasticsearch-5.6.9.jar:5.6.9]
  41.     at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:342) ~[elasticsearch-5.6.9.jar:5.6.9]
  42.     at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:329) ~[elasticsearch-5.6.9.jar:5.6.9]
  43.     at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:154) [elasticsearch-5.6.9.jar:5.6.9]
  44.     at org.elasticsearch.cluster.NodeConnectionsService$ConnectionChecker.doRun(NodeConnectionsService.java:183) [elasticsearch-5.6.9.jar:5.6.9]
  45.     at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:674) [elasticsearch-5.6.9.jar:5.6.9]
  46.     at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.9.jar:5.6.9]
  47.     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
  48.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
  49.     at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  50. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: 10.231.83.146/10.231.83.146:9301
  51.     at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  52.     at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
  53.     at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352) ~[?:?]
  54.     at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  55.     at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?]
  56.     at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?]
  57.     at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?]
  58.     at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?]
  59.     at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
  60.     ... 1 more
  61. Caused by: java.net.ConnectException: Connection refused
  62.     at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  63.     at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
  64.     at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352) ~[?:?]
  65.     at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  66.     at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?]
  67.     at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?]
  68.     at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?]
  69.     at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?]
  70.     at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
  71.     ... 1 more
  72. [2019-01-25T10:43:15,292][INFO ][o.e.c.s.ClusterService   ] [yRj9xlW] detected_master {wAVZzky}{wAVZzkyrSfyBmuGRw0Bltw}{r1L7UXpESgiM_Kxq_jFUjw}{10.231.83.146}{10.231.83.146:9302}, reason: zen-disco-receive(from master [master {wAVZzky}{wAVZzkyrSfyBmuGRw0Bltw}{r1L7UXpESgiM_Kxq_jFUjw}{10.231.83.146}{10.231.83.146:9302} committed version [181]])
  73. [2019-01-25T10:43:15,367][INFO ][o.e.c.s.ClusterService   ] [yRj9xlW] removed {{lJgJwYh}{lJgJwYhmQA-fucaOBHD5Vg}{ho76tglrRPKjA8ui_UVJSg}{10.231.83.146}{10.231.83.146:9301},}, reason: zen-disco-receive(from master [master {wAVZzky}{wAVZzkyrSfyBmuGRw0Bltw}{r1L7UXpESgiM_Kxq_jFUjw}{10.231.83.146}{10.231.83.146:9302} committed version [182]])
  74. [2019-01-25T10:43:17,532][INFO ][o.e.d.z.ZenDiscovery     ] [yRj9xlW] master_left [{wAVZzky}{wAVZzkyrSfyBmuGRw0Bltw}{r1L7UXpESgiM_Kxq_jFUjw}{10.231.83.146}{10.231.83.146:9302}], reason [shut_down]
  75. [2019-01-25T10:43:17,535][WARN ][o.e.d.z.ZenDiscovery     ] [yRj9xlW] master left (reason = shut_down), current nodes: nodes:
  76.    {wAVZzky}{wAVZzkyrSfyBmuGRw0Bltw}{r1L7UXpESgiM_Kxq_jFUjw}{10.231.83.146}{10.231.83.146:9302}, master
  77.    {yRj9xlW}{yRj9xlW8Q-WgQ0j6H4DbOQ}{SAn1BH7ES42OhsAknxVe0Q}{10.231.83.238}{10.231.83.238:9300}, local
  78.  
  79. [2019-01-25T10:43:17,536][WARN ][o.e.t.n.Netty4Transport  ] [yRj9xlW] write and flush on the network layer failed (channel: [id: 0xbc6c3d15, L:0.0.0.0/0.0.0.0:9300 ! R:/10.231.83.146:50934])
  80. java.nio.channels.ClosedChannelException: null
  81.     at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
  82. [2019-01-25T10:43:17,548][WARN ][o.e.c.NodeConnectionsService] [yRj9xlW] failed to connect to node {wAVZzky}{wAVZzkyrSfyBmuGRw0Bltw}{r1L7UXpESgiM_Kxq_jFUjw}{10.231.83.146}{10.231.83.146:9302} (tried [1] times)
  83. org.elasticsearch.transport.ConnectTransportException: [wAVZzky][10.231.83.146:9302] connect_timeout[30s]
  84.     at org.elasticsearch.transport.netty4.Netty4Transport.connectToChannels(Netty4Transport.java:363) ~[?:?]
  85.     at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:570) ~[elasticsearch-5.6.9.jar:5.6.9]
  86.     at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:473) ~[elasticsearch-5.6.9.jar:5.6.9]
  87.     at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:342) ~[elasticsearch-5.6.9.jar:5.6.9]
  88.     at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:329) ~[elasticsearch-5.6.9.jar:5.6.9]
  89.     at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:154) [elasticsearch-5.6.9.jar:5.6.9]
  90.     at org.elasticsearch.cluster.NodeConnectionsService$1.doRun(NodeConnectionsService.java:107) [elasticsearch-5.6.9.jar:5.6.9]
  91.     at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:674) [elasticsearch-5.6.9.jar:5.6.9]
  92.     at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.9.jar:5.6.9]
  93.     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
  94.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
  95.     at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  96. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: 10.231.83.146/10.231.83.146:9302
  97.     at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  98.     at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
  99.     at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352) ~[?:?]
  100.     at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  101.     at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?]
  102.     at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?]
  103.     at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?]
  104.     at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?]
  105.     at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
  106.     ... 1 more
  107. Caused by: java.net.ConnectException: Connection refused
  108.     at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  109.     at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
  110.     at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352) ~[?:?]
  111.     at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  112.     at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?]
  113.     at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?]
  114.     at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?]
  115.     at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?]
  116.     at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
  117.     ... 1 more
  118. [2019-01-25T10:43:22,537][INFO ][o.e.c.s.ClusterService   ] [yRj9xlW] new_master {yRj9xlW}{yRj9xlW8Q-WgQ0j6H4DbOQ}{SAn1BH7ES42OhsAknxVe0Q}{10.231.83.238}{10.231.83.238:9300}, added {{lJgJwYh}{lJgJwYhmQA-fucaOBHD5Vg}{lbN7QcyiSWCOBYNL4jNkdg}{10.231.83.146}{10.231.83.146:9301},}, reason: zen-disco-elected-as-master ([1] nodes joined)[, ]
  119. [2019-01-25T10:43:22,586][WARN ][o.e.g.GatewayAllocator$InternalReplicaShardAllocator] [yRj9xlW] [pdai][0]: failed to list shard for shard_store on node [wAVZzkyrSfyBmuGRw0Bltw]
  120. org.elasticsearch.action.FailedNodeException: Failed node [wAVZzkyrSfyBmuGRw0Bltw]
  121.     at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:239) ~[elasticsearch-5.6.9.jar:5.6.9]
  122.     at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$200(TransportNodesAction.java:153) ~[elasticsearch-5.6.9.jar:5.6.9]
  123.     at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:211) ~[elasticsearch-5.6.9.jar:5.6.9]
  124.     at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:520) ~[elasticsearch-5.6.9.jar:5.6.9]
  125.     at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.start(TransportNodesAction.java:197) ~[elasticsearch-5.6.9.jar:5.6.9]
  126.     at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:89) ~[elasticsearch-5.6.9.jar:5.6.9]
  127.     at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:52) ~[elasticsearch-5.6.9.jar:5.6.9]
  128.     at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) ~[elasticsearch-5.6.9.jar:5.6.9]
  129.     at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) ~[elasticsearch-5.6.9.jar:5.6.9]
  130.     at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84) ~[elasticsearch-5.6.9.jar:5.6.9]
  131.     at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.list(TransportNodesListShardStoreMetaData.java:90) ~[elasticsearch-5.6.9.jar:5.6.9]
  132.     at org.elasticsearch.gateway.AsyncShardFetch.asyncFetch(AsyncShardFetch.java:289) ~[elasticsearch-5.6.9.jar:5.6.9]
  133.     at org.elasticsearch.gateway.AsyncShardFetch.fetchData(AsyncShardFetch.java:127) ~[elasticsearch-5.6.9.jar:5.6.9]
  134.     at org.elasticsearch.gateway.GatewayAllocator$InternalReplicaShardAllocator.fetchData(GatewayAllocator.java:188) ~[elasticsearch-5.6.9.jar:5.6.9]
  135.     at org.elasticsearch.gateway.ReplicaShardAllocator.makeAllocationDecision(ReplicaShardAllocator.java:172) ~[elasticsearch-5.6.9.jar:5.6.9]
  136.     at org.elasticsearch.gateway.BaseGatewayShardAllocator.allocateUnassigned(BaseGatewayShardAllocator.java:62) ~[elasticsearch-5.6.9.jar:5.6.9]
  137.     at org.elasticsearch.gateway.GatewayAllocator.innerAllocatedUnassigned(GatewayAllocator.java:123) ~[elasticsearch-5.6.9.jar:5.6.9]
  138.     at org.elasticsearch.gateway.GatewayAllocator.allocateUnassigned(GatewayAllocator.java:111) ~[elasticsearch-5.6.9.jar:5.6.9]
  139.     at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:326) ~[elasticsearch-5.6.9.jar:5.6.9]
  140.     at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:296) ~[elasticsearch-5.6.9.jar:5.6.9]
  141.     at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:281) ~[elasticsearch-5.6.9.jar:5.6.9]
  142.     at org.elasticsearch.discovery.zen.NodeJoinController$JoinTaskExecutor.execute(NodeJoinController.java:461) ~[elasticsearch-5.6.9.jar:5.6.9]
  143.     at org.elasticsearch.cluster.service.ClusterService.executeTasks(ClusterService.java:634) ~[elasticsearch-5.6.9.jar:5.6.9]
  144.     at org.elasticsearch.cluster.service.ClusterService.calculateTaskOutputs(ClusterService.java:612) ~[elasticsearch-5.6.9.jar:5.6.9]
  145.     at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:571) ~[elasticsearch-5.6.9.jar:5.6.9]
  146.     at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) ~[elasticsearch-5.6.9.jar:5.6.9]
  147.     at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) ~[elasticsearch-5.6.9.jar:5.6.9]
  148.     at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) ~[elasticsearch-5.6.9.jar:5.6.9]
  149.     at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) ~[elasticsearch-5.6.9.jar:5.6.9]
  150.     at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) ~[elasticsearch-5.6.9.jar:5.6.9]
  151.     at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) ~[elasticsearch-5.6.9.jar:5.6.9]
  152.     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_181]
  153.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_181]
  154.     at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  155. Caused by: org.elasticsearch.transport.NodeNotConnectedException: [wAVZzky][10.231.83.146:9302] Node not connected
  156.     at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:640) ~[elasticsearch-5.6.9.jar:5.6.9]
  157.     at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:117) ~[elasticsearch-5.6.9.jar:5.6.9]
  158.     at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:540) ~[elasticsearch-5.6.9.jar:5.6.9]
  159.     at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:516) ~[elasticsearch-5.6.9.jar:5.6.9]
  160.     ... 30 more
  161. [2019-01-25T10:43:22,589][WARN ][o.e.g.GatewayAllocator$InternalReplicaShardAllocator] [yRj9xlW] [routing][0]: failed to list shard for shard_store on node [wAVZzkyrSfyBmuGRw0Bltw]
  162. org.elasticsearch.action.FailedNodeException: Failed node [wAVZzkyrSfyBmuGRw0Bltw]
  163.     at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:239) ~[elasticsearch-5.6.9.jar:5.6.9]
  164.     at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$200(TransportNodesAction.java:153) ~[elasticsearch-5.6.9.jar:5.6.9]
  165.     at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:211) ~[elasticsearch-5.6.9.jar:5.6.9]
  166.     at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:520) ~[elasticsearch-5.6.9.jar:5.6.9]
  167.     at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.start(TransportNodesAction.java:197) ~[elasticsearch-5.6.9.jar:5.6.9]
  168.     at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:89) ~[elasticsearch-5.6.9.jar:5.6.9]
  169.     at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:52) ~[elasticsearch-5.6.9.jar:5.6.9]
  170.     at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) ~[elasticsearch-5.6.9.jar:5.6.9]
  171.     at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) ~[elasticsearch-5.6.9.jar:5.6.9]
  172.     at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84) ~[elasticsearch-5.6.9.jar:5.6.9]
  173.     at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.list(TransportNodesListShardStoreMetaData.java:90) ~[elasticsearch-5.6.9.jar:5.6.9]
  174.     at org.elasticsearch.gateway.AsyncShardFetch.asyncFetch(AsyncShardFetch.java:289) ~[elasticsearch-5.6.9.jar:5.6.9]
  175.     at org.elasticsearch.gateway.AsyncShardFetch.fetchData(AsyncShardFetch.java:127) ~[elasticsearch-5.6.9.jar:5.6.9]
  176.     at org.elasticsearch.gateway.GatewayAllocator$InternalReplicaShardAllocator.fetchData(GatewayAllocator.java:188) ~[elasticsearch-5.6.9.jar:5.6.9]
  177.     at org.elasticsearch.gateway.ReplicaShardAllocator.makeAllocationDecision(ReplicaShardAllocator.java:172) ~[elasticsearch-5.6.9.jar:5.6.9]
  178.     at org.elasticsearch.gateway.BaseGatewayShardAllocator.allocateUnassigned(BaseGatewayShardAllocator.java:62) ~[elasticsearch-5.6.9.jar:5.6.9]
  179.     at org.elasticsearch.gateway.GatewayAllocator.innerAllocatedUnassigned(GatewayAllocator.java:123) ~[elasticsearch-5.6.9.jar:5.6.9]
  180.     at org.elasticsearch.gateway.GatewayAllocator.allocateUnassigned(GatewayAllocator.java:111) ~[elasticsearch-5.6.9.jar:5.6.9]
  181.     at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:326) ~[elasticsearch-5.6.9.jar:5.6.9]
  182.     at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:296) ~[elasticsearch-5.6.9.jar:5.6.9]
  183.     at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:281) ~[elasticsearch-5.6.9.jar:5.6.9]
  184.     at org.elasticsearch.discovery.zen.NodeJoinController$JoinTaskExecutor.execute(NodeJoinController.java:461) ~[elasticsearch-5.6.9.jar:5.6.9]
  185.     at org.elasticsearch.cluster.service.ClusterService.executeTasks(ClusterService.java:634) ~[elasticsearch-5.6.9.jar:5.6.9]
  186.     at org.elasticsearch.cluster.service.ClusterService.calculateTaskOutputs(ClusterService.java:612) ~[elasticsearch-5.6.9.jar:5.6.9]
  187.     at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:571) ~[elasticsearch-5.6.9.jar:5.6.9]
  188.     at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) ~[elasticsearch-5.6.9.jar:5.6.9]
  189.     at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) ~[elasticsearch-5.6.9.jar:5.6.9]
  190.     at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) ~[elasticsearch-5.6.9.jar:5.6.9]
  191.     at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) ~[elasticsearch-5.6.9.jar:5.6.9]
  192.     at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) ~[elasticsearch-5.6.9.jar:5.6.9]
  193.     at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) ~[elasticsearch-5.6.9.jar:5.6.9]
  194.     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_181]
  195.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_181]
  196.     at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  197. Caused by: org.elasticsearch.transport.NodeNotConnectedException: [wAVZzky][10.231.83.146:9302] Node not connected
  198.     at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:640) ~[elasticsearch-5.6.9.jar:5.6.9]
  199.     at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:117) ~[elasticsearch-5.6.9.jar:5.6.9]
  200.     at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:540) ~[elasticsearch-5.6.9.jar:5.6.9]
  201.     at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:516) ~[elasticsearch-5.6.9.jar:5.6.9]
  202.     ... 30 more
  203. [2019-01-25T10:43:22,592][WARN ][o.e.g.GatewayAllocator$InternalReplicaShardAllocator] [yRj9xlW] [lc][0]: failed to list shard for shard_store on node [wAVZzkyrSfyBmuGRw0Bltw]
  204. org.elasticsearch.action.FailedNodeException: Failed node [wAVZzkyrSfyBmuGRw0Bltw]
  205.     at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:239) ~[elasticsearch-5.6.9.jar:5.6.9]
  206.     at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$200(TransportNodesAction.java:153) ~[elasticsearch-5.6.9.jar:5.6.9]
  207.     at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:211) ~[elasticsearch-5.6.9.jar:5.6.9]
  208.     at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:520) ~[elasticsearch-5.6.9.jar:5.6.9]
  209.     at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.start(TransportNodesAction.java:197) ~[elasticsearch-5.6.9.jar:5.6.9]
  210.     at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:89) ~[elasticsearch-5.6.9.jar:5.6.9]
  211.     at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:52) ~[elasticsearch-5.6.9.jar:5.6.9]
  212.     at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) ~[elasticsearch-5.6.9.jar:5.6.9]
  213.     at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) ~[elasticsearch-5.6.9.jar:5.6.9]
  214.     at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84) ~[elasticsearch-5.6.9.jar:5.6.9]
  215.     at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.list(TransportNodesListShardStoreMetaData.java:90) ~[elasticsearch-5.6.9.jar:5.6.9]
  216.     at org.elasticsearch.gateway.AsyncShardFetch.asyncFetch(AsyncShardFetch.java:289) ~[elasticsearch-5.6.9.jar:5.6.9]
  217.     at org.elasticsearch.gateway.AsyncShardFetch.fetchData(AsyncShardFetch.java:127) ~[elasticsearch-5.6.9.jar:5.6.9]
  218.     at org.elasticsearch.gateway.GatewayAllocator$InternalReplicaShardAllocator.fetchData(GatewayAllocator.java:188) ~[elasticsearch-5.6.9.jar:5.6.9]
  219.     at org.elasticsearch.gateway.ReplicaShardAllocator.makeAllocationDecision(ReplicaShardAllocator.java:172) ~[elasticsearch-5.6.9.jar:5.6.9]
  220.     at org.elasticsearch.gateway.BaseGatewayShardAllocator.allocateUnassigned(BaseGatewayShardAllocator.java:62) ~[elasticsearch-5.6.9.jar:5.6.9]
  221.     at org.elasticsearch.gateway.GatewayAllocator.innerAllocatedUnassigned(GatewayAllocator.java:123) ~[elasticsearch-5.6.9.jar:5.6.9]
  222.     at org.elasticsearch.gateway.GatewayAllocator.allocateUnassigned(GatewayAllocator.java:111) ~[elasticsearch-5.6.9.jar:5.6.9]
  223.     at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:326) ~[elasticsearch-5.6.9.jar:5.6.9]
  224.     at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:296) ~[elasticsearch-5.6.9.jar:5.6.9]
  225.     at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:281) ~[elasticsearch-5.6.9.jar:5.6.9]
  226.     at org.elasticsearch.discovery.zen.NodeJoinController$JoinTaskExecutor.execute(NodeJoinController.java:461) ~[elasticsearch-5.6.9.jar:5.6.9]
  227.     at org.elasticsearch.cluster.service.ClusterService.executeTasks(ClusterService.java:634) ~[elasticsearch-5.6.9.jar:5.6.9]
  228.     at org.elasticsearch.cluster.service.ClusterService.calculateTaskOutputs(ClusterService.java:612) ~[elasticsearch-5.6.9.jar:5.6.9]
  229.     at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:571) ~[elasticsearch-5.6.9.jar:5.6.9]
  230.     at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) ~[elasticsearch-5.6.9.jar:5.6.9]
  231.     at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) ~[elasticsearch-5.6.9.jar:5.6.9]
  232.     at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) ~[elasticsearch-5.6.9.jar:5.6.9]
  233.     at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) ~[elasticsearch-5.6.9.jar:5.6.9]
  234.     at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) ~[elasticsearch-5.6.9.jar:5.6.9]
  235.     at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) ~[elasticsearch-5.6.9.jar:5.6.9]
  236.     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_181]
  237.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_181]
  238.     at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  239. Caused by: org.elasticsearch.transport.NodeNotConnectedException: [wAVZzky][10.231.83.146:9302] Node not connected
  240.     at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:640) ~[elasticsearch-5.6.9.jar:5.6.9]
  241.     at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:117) ~[elasticsearch-5.6.9.jar:5.6.9]
  242.     at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:540) ~[elasticsearch-5.6.9.jar:5.6.9]
  243.     at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:516) ~[elasticsearch-5.6.9.jar:5.6.9]
  244.     ... 30 more
  245. [2019-01-25T10:43:22,592][WARN ][o.e.g.GatewayAllocator$InternalReplicaShardAllocator] [yRj9xlW] [vanmessagetopic][1]: failed to list shard for shard_store on node [wAVZzkyrSfyBmuGRw0Bltw]
  246. org.elasticsearch.action.FailedNodeException: Failed node [wAVZzkyrSfyBmuGRw0Bltw]
  247.     at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:239) ~[elasticsearch-5.6.9.jar:5.6.9]
  248.     at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$200(TransportNodesAction.java:153) ~[elasticsearch-5.6.9.jar:5.6.9]
  249.     at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:211) ~[elasticsearch-5.6.9.jar:5.6.9]
  250.     at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:520) ~[elasticsearch-5.6.9.jar:5.6.9]
  251.     at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.start(TransportNodesAction.java:197) ~[elasticsearch-5.6.9.jar:5.6.9]
  252.     at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:89) ~[elasticsearch-5.6.9.jar:5.6.9]
  253.     at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:52) ~[elasticsearch-5.6.9.jar:5.6.9]
  254.     at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) ~[elasticsearch-5.6.9.jar:5.6.9]
  255.     at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) ~[elasticsearch-5.6.9.jar:5.6.9]
  256.     at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84) ~[elasticsearch-5.6.9.jar:5.6.9]
  257.     at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.list(TransportNodesListShardStoreMetaData.java:90) ~[elasticsearch-5.6.9.jar:5.6.9]
  258.     at org.elasticsearch.gateway.AsyncShardFetch.asyncFetch(AsyncShardFetch.java:289) ~[elasticsearch-5.6.9.jar:5.6.9]
  259.     at org.elasticsearch.gateway.AsyncShardFetch.fetchData(AsyncShardFetch.java:127) ~[elasticsearch-5.6.9.jar:5.6.9]
  260.     at org.elasticsearch.gateway.GatewayAllocator$InternalReplicaShardAllocator.fetchData(GatewayAllocator.java:188) ~[elasticsearch-5.6.9.jar:5.6.9]
  261.     at org.elasticsearch.gateway.ReplicaShardAllocator.makeAllocationDecision(ReplicaShardAllocator.java:172) ~[elasticsearch-5.6.9.jar:5.6.9]
  262.     at org.elasticsearch.gateway.BaseGatewayShardAllocator.allocateUnassigned(BaseGatewayShardAllocator.java:62) ~[elasticsearch-5.6.9.jar:5.6.9]
  263.     at org.elasticsearch.gateway.GatewayAllocator.innerAllocatedUnassigned(GatewayAllocator.java:123) ~[elasticsearch-5.6.9.jar:5.6.9]
  264.     at org.elasticsearch.gateway.GatewayAllocator.allocateUnassigned(GatewayAllocator.java:111) ~[elasticsearch-5.6.9.jar:5.6.9]
  265.     at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:326) ~[elasticsearch-5.6.9.jar:5.6.9]
  266.     at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:296) ~[elasticsearch-5.6.9.jar:5.6.9]
  267.     at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:281) ~[elasticsearch-5.6.9.jar:5.6.9]
  268.     at org.elasticsearch.discovery.zen.NodeJoinController$JoinTaskExecutor.execute(NodeJoinController.java:461) ~[elasticsearch-5.6.9.jar:5.6.9]
  269.     at org.elasticsearch.cluster.service.ClusterService.executeTasks(ClusterService.java:634) ~[elasticsearch-5.6.9.jar:5.6.9]
  270.     at org.elasticsearch.cluster.service.ClusterService.calculateTaskOutputs(ClusterService.java:612) ~[elasticsearch-5.6.9.jar:5.6.9]
  271.     at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:571) ~[elasticsearch-5.6.9.jar:5.6.9]
  272.     at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) ~[elasticsearch-5.6.9.jar:5.6.9]
  273.     at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) ~[elasticsearch-5.6.9.jar:5.6.9]
  274.     at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) ~[elasticsearch-5.6.9.jar:5.6.9]
  275.     at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) ~[elasticsearch-5.6.9.jar:5.6.9]
  276.     at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) ~[elasticsearch-5.6.9.jar:5.6.9]
  277.     at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) ~[elasticsearch-5.6.9.jar:5.6.9]
  278.     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_181]
  279.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_181]
  280.     at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  281. Caused by: org.elasticsearch.transport.NodeNotConnectedException: [wAVZzky][10.231.83.146:9302] Node not connected
  282.     at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:640) ~[elasticsearch-5.6.9.jar:5.6.9]
  283.     at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:117) ~[elasticsearch-5.6.9.jar:5.6.9]
  284.     at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:540) ~[elasticsearch-5.6.9.jar:5.6.9]
  285.     at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:516) ~[elasticsearch-5.6.9.jar:5.6.9]
  286.     ... 30 more
  287. [2019-01-25T10:43:22,593][WARN ][o.e.g.GatewayAllocator$InternalReplicaShardAllocator] [yRj9xlW] [device][0]: failed to list shard for shard_store on node [wAVZzkyrSfyBmuGRw0Bltw]
  288. org.elasticsearch.action.FailedNodeException: Failed node [wAVZzkyrSfyBmuGRw0Bltw]
  289.     at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:239) ~[elasticsearch-5.6.9.jar:5.6.9]
  290.     at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$200(TransportNodesAction.java:153) ~[elasticsearch-5.6.9.jar:5.6.9]
  291.     at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:211) ~[elasticsearch-5.6.9.jar:5.6.9]
  292.     at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:520) ~[elasticsearch-5.6.9.jar:5.6.9]
  293.     at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.start(TransportNodesAction.java:197) ~[elasticsearch-5.6.9.jar:5.6.9]
  294.     at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:89) ~[elasticsearch-5.6.9.jar:5.6.9]
  295.     at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:52) ~[elasticsearch-5.6.9.jar:5.6.9]
  296.     at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) ~[elasticsearch-5.6.9.jar:5.6.9]
  297.     at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) ~[elasticsearch-5.6.9.jar:5.6.9]
  298.     at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84) ~[elasticsearch-5.6.9.jar:5.6.9]
  299.     at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.list(TransportNodesListShardStoreMetaData.java:90) ~[elasticsearch-5.6.9.jar:5.6.9]
  300.     at org.elasticsearch.gateway.AsyncShardFetch.asyncFetch(AsyncShardFetch.java:289) ~[elasticsearch-5.6.9.jar:5.6.9]
  301.     at org.elasticsearch.gateway.AsyncShardFetch.fetchData(AsyncShardFetch.java:127) ~[elasticsearch-5.6.9.jar:5.6.9]
  302.     at org.elasticsearch.gateway.GatewayAllocator$InternalReplicaShardAllocator.fetchData(GatewayAllocator.java:188) ~[elasticsearch-5.6.9.jar:5.6.9]
  303.     at org.elasticsearch.gateway.ReplicaShardAllocator.makeAllocationDecision(ReplicaShardAllocator.java:172) ~[elasticsearch-5.6.9.jar:5.6.9]
  304.     at org.elasticsearch.gateway.BaseGatewayShardAllocator.allocateUnassigned(BaseGatewayShardAllocator.java:62) ~[elasticsearch-5.6.9.jar:5.6.9]
  305.     at org.elasticsearch.gateway.GatewayAllocator.innerAllocatedUnassigned(GatewayAllocator.java:123) ~[elasticsearch-5.6.9.jar:5.6.9]
  306.     at org.elasticsearch.gateway.GatewayAllocator.allocateUnassigned(GatewayAllocator.java:111) ~[elasticsearch-5.6.9.jar:5.6.9]
  307.     at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:326) ~[elasticsearch-5.6.9.jar:5.6.9]
  308.     at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:296) ~[elasticsearch-5.6.9.jar:5.6.9]
  309.     at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:281) ~[elasticsearch-5.6.9.jar:5.6.9]
  310.     at org.elasticsearch.discovery.zen.NodeJoinController$JoinTaskExecutor.execute(NodeJoinController.java:461) ~[elasticsearch-5.6.9.jar:5.6.9]
  311.     at org.elasticsearch.cluster.service.ClusterService.executeTasks(ClusterService.java:634) ~[elasticsearch-5.6.9.jar:5.6.9]
  312.     at org.elasticsearch.cluster.service.ClusterService.calculateTaskOutputs(ClusterService.java:612) ~[elasticsearch-5.6.9.jar:5.6.9]
  313.     at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:571) ~[elasticsearch-5.6.9.jar:5.6.9]
  314.     at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) ~[elasticsearch-5.6.9.jar:5.6.9]
  315.     at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) ~[elasticsearch-5.6.9.jar:5.6.9]
  316.     at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) ~[elasticsearch-5.6.9.jar:5.6.9]
  317.     at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) ~[elasticsearch-5.6.9.jar:5.6.9]
  318.     at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) ~[elasticsearch-5.6.9.jar:5.6.9]
  319.     at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) ~[elasticsearch-5.6.9.jar:5.6.9]
  320.     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_181]
  321.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_181]
  322.     at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  323. Caused by: org.elasticsearch.transport.NodeNotConnectedException: [wAVZzky][10.231.83.146:9302] Node not connected
  324.     at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:640) ~[elasticsearch-5.6.9.jar:5.6.9]
  325.     at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:117) ~[elasticsearch-5.6.9.jar:5.6.9]
  326.     at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:540) ~[elasticsearch-5.6.9.jar:5.6.9]
  327.     at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:516) ~[elasticsearch-5.6.9.jar:5.6.9]
  328.     ... 30 more
  329. [2019-01-25T10:43:22,596][WARN ][o.e.g.GatewayAllocator$InternalReplicaShardAllocator] [yRj9xlW] [vanmessagetopic][3]: failed to list shard for shard_store on node [wAVZzkyrSfyBmuGRw0Bltw]
  330. org.elasticsearch.action.FailedNodeException: Failed node [wAVZzkyrSfyBmuGRw0Bltw]
  331.     at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:239) ~[elasticsearch-5.6.9.jar:5.6.9]
  332.     at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$200(TransportNodesAction.java:153) ~[elasticsearch-5.6.9.jar:5.6.9]
  333.     at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:211) ~[elasticsearch-5.6.9.jar:5.6.9]
  334.     at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:520) ~[elasticsearch-5.6.9.jar:5.6.9]
  335.     at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.start(TransportNodesAction.java:197) ~[elasticsearch-5.6.9.jar:5.6.9]
  336.     at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:89) ~[elasticsearch-5.6.9.jar:5.6.9]
  337.     at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:52) ~[elasticsearch-5.6.9.jar:5.6.9]
  338.     at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) ~[elasticsearch-5.6.9.jar:5.6.9]
  339.     at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) ~[elasticsearch-5.6.9.jar:5.6.9]
  340.     at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84) ~[elasticsearch-5.6.9.jar:5.6.9]
  341.     at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.list(TransportNodesListShardStoreMetaData.java:90) ~[elasticsearch-5.6.9.jar:5.6.9]
  342.     at org.elasticsearch.gateway.AsyncShardFetch.asyncFetch(AsyncShardFetch.java:289) ~[elasticsearch-5.6.9.jar:5.6.9]
  343.     at org.elasticsearch.gateway.AsyncShardFetch.fetchData(AsyncShardFetch.java:127) ~[elasticsearch-5.6.9.jar:5.6.9]
  344.     at org.elasticsearch.gateway.GatewayAllocator$InternalReplicaShardAllocator.fetchData(GatewayAllocator.java:188) ~[elasticsearch-5.6.9.jar:5.6.9]
  345.     at org.elasticsearch.gateway.ReplicaShardAllocator.makeAllocationDecision(ReplicaShardAllocator.java:172) ~[elasticsearch-5.6.9.jar:5.6.9]
  346.     at org.elasticsearch.gateway.BaseGatewayShardAllocator.allocateUnassigned(BaseGatewayShardAllocator.java:62) ~[elasticsearch-5.6.9.jar:5.6.9]
  347.     at org.elasticsearch.gateway.GatewayAllocator.innerAllocatedUnassigned(GatewayAllocator.java:123) ~[elasticsearch-5.6.9.jar:5.6.9]
  348.     at org.elasticsearch.gateway.GatewayAllocator.allocateUnassigned(GatewayAllocator.java:111) ~[elasticsearch-5.6.9.jar:5.6.9]
  349.     at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:326) ~[elasticsearch-5.6.9.jar:5.6.9]
  350.     at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:296) ~[elasticsearch-5.6.9.jar:5.6.9]
  351.     at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:281) ~[elasticsearch-5.6.9.jar:5.6.9]
  352.     at org.elasticsearch.discovery.zen.NodeJoinController$JoinTaskExecutor.execute(NodeJoinController.java:461) ~[elasticsearch-5.6.9.jar:5.6.9]
  353.     at org.elasticsearch.cluster.service.ClusterService.executeTasks(ClusterService.java:634) ~[elasticsearch-5.6.9.jar:5.6.9]
  354.     at org.elasticsearch.cluster.service.ClusterService.calculateTaskOutputs(ClusterService.java:612) ~[elasticsearch-5.6.9.jar:5.6.9]
  355.     at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:571) ~[elasticsearch-5.6.9.jar:5.6.9]
  356.     at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) ~[elasticsearch-5.6.9.jar:5.6.9]
  357.     at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) ~[elasticsearch-5.6.9.jar:5.6.9]
  358.     at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) ~[elasticsearch-5.6.9.jar:5.6.9]
  359.     at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) ~[elasticsearch-5.6.9.jar:5.6.9]
  360.     at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) ~[elasticsearch-5.6.9.jar:5.6.9]
  361.     at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) ~[elasticsearch-5.6.9.jar:5.6.9]
  362.     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_181]
  363.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_181]
  364.     at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  365. Caused by: org.elasticsearch.transport.NodeNotConnectedException: [wAVZzky][10.231.83.146:9302] Node not connected
  366.     at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:640) ~[elasticsearch-5.6.9.jar:5.6.9]
  367.     at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:117) ~[elasticsearch-5.6.9.jar:5.6.9]
  368.     at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:540) ~[elasticsearch-5.6.9.jar:5.6.9]
  369.     at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:516) ~[elasticsearch-5.6.9.jar:5.6.9]
  370.     ... 30 more
  371. [2019-01-25T10:43:22,597][WARN ][o.e.g.GatewayAllocator$InternalReplicaShardAllocator] [yRj9xlW] [vanmessagetopic][4]: failed to list shard for shard_store on node [wAVZzkyrSfyBmuGRw0Bltw]
  372. org.elasticsearch.action.FailedNodeException: Failed node [wAVZzkyrSfyBmuGRw0Bltw]
  373.     at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:239) ~[elasticsearch-5.6.9.jar:5.6.9]
  374.     at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$200(TransportNodesAction.java:153) ~[elasticsearch-5.6.9.jar:5.6.9]
  375.     at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:211) ~[elasticsearch-5.6.9.jar:5.6.9]
  376.     at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:520) ~[elasticsearch-5.6.9.jar:5.6.9]
  377.     at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.start(TransportNodesAction.java:197) ~[elasticsearch-5.6.9.jar:5.6.9]
  378.     at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:89) ~[elasticsearch-5.6.9.jar:5.6.9]
  379.     at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:52) ~[elasticsearch-5.6.9.jar:5.6.9]
  380.     at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) ~[elasticsearch-5.6.9.jar:5.6.9]
  381.     at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) ~[elasticsearch-5.6.9.jar:5.6.9]
  382.     at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84) ~[elasticsearch-5.6.9.jar:5.6.9]
  383.     at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.list(TransportNodesListShardStoreMetaData.java:90) ~[elasticsearch-5.6.9.jar:5.6.9]
  384.     at org.elasticsearch.gateway.AsyncShardFetch.asyncFetch(AsyncShardFetch.java:289) ~[elasticsearch-5.6.9.jar:5.6.9]
  385.     at org.elasticsearch.gateway.AsyncShardFetch.fetchData(AsyncShardFetch.java:127) ~[elasticsearch-5.6.9.jar:5.6.9]
  386.     at org.elasticsearch.gateway.GatewayAllocator$InternalReplicaShardAllocator.fetchData(GatewayAllocator.java:188) ~[elasticsearch-5.6.9.jar:5.6.9]
  387.     at org.elasticsearch.gateway.ReplicaShardAllocator.makeAllocationDecision(ReplicaShardAllocator.java:172) ~[elasticsearch-5.6.9.jar:5.6.9]
  388.     at org.elasticsearch.gateway.BaseGatewayShardAllocator.allocateUnassigned(BaseGatewayShardAllocator.java:62) ~[elasticsearch-5.6.9.jar:5.6.9]
  389.     at org.elasticsearch.gateway.GatewayAllocator.innerAllocatedUnassigned(GatewayAllocator.java:123) ~[elasticsearch-5.6.9.jar:5.6.9]
  390.     at org.elasticsearch.gateway.GatewayAllocator.allocateUnassigned(GatewayAllocator.java:111) ~[elasticsearch-5.6.9.jar:5.6.9]
  391.     at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:326) ~[elasticsearch-5.6.9.jar:5.6.9]
  392.     at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:296) ~[elasticsearch-5.6.9.jar:5.6.9]
  393.     at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:281) ~[elasticsearch-5.6.9.jar:5.6.9]
  394.     at org.elasticsearch.discovery.zen.NodeJoinController$JoinTaskExecutor.execute(NodeJoinController.java:461) ~[elasticsearch-5.6.9.jar:5.6.9]
  395.     at org.elasticsearch.cluster.service.ClusterService.executeTasks(ClusterService.java:634) ~[elasticsearch-5.6.9.jar:5.6.9]
  396.     at org.elasticsearch.cluster.service.ClusterService.calculateTaskOutputs(ClusterService.java:612) ~[elasticsearch-5.6.9.jar:5.6.9]
  397.     at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:571) ~[elasticsearch-5.6.9.jar:5.6.9]
  398.     at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) ~[elasticsearch-5.6.9.jar:5.6.9]
  399.     at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) ~[elasticsearch-5.6.9.jar:5.6.9]
  400.     at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) ~[elasticsearch-5.6.9.jar:5.6.9]
  401.     at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) ~[elasticsearch-5.6.9.jar:5.6.9]
  402.     at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) ~[elasticsearch-5.6.9.jar:5.6.9]
  403.     at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) ~[elasticsearch-5.6.9.jar:5.6.9]
  404.     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_181]
  405.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_181]
  406.     at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  407. Caused by: org.elasticsearch.transport.NodeNotConnectedException: [wAVZzky][10.231.83.146:9302] Node not connected
  408.     at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:640) ~[elasticsearch-5.6.9.jar:5.6.9]
  409.     at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:117) ~[elasticsearch-5.6.9.jar:5.6.9]
  410.     at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:540) ~[elasticsearch-5.6.9.jar:5.6.9]
  411.     at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:516) ~[elasticsearch-5.6.9.jar:5.6.9]
  412.     ... 30 more
  413. [2019-01-25T10:43:22,989][INFO ][o.e.c.r.DelayedAllocationService] [yRj9xlW] scheduling reroute for delayed shards in [52.3s] (7 delayed shards)
  414. [2019-01-25T10:43:22,993][DEBUG][o.e.a.a.c.n.s.TransportNodesStatsAction] [yRj9xlW] failed to execute on node [wAVZzkyrSfyBmuGRw0Bltw]
  415. org.elasticsearch.transport.NodeNotConnectedException: [wAVZzky][10.231.83.146:9302] Node not connected
  416.     at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:640) ~[elasticsearch-5.6.9.jar:5.6.9]
  417.     at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:117) ~[elasticsearch-5.6.9.jar:5.6.9]
  418.     at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:540) ~[elasticsearch-5.6.9.jar:5.6.9]
  419.     at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:516) ~[elasticsearch-5.6.9.jar:5.6.9]
  420.     at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.start(TransportNodesAction.java:197) ~[elasticsearch-5.6.9.jar:5.6.9]
  421.     at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:89) ~[elasticsearch-5.6.9.jar:5.6.9]
  422.     at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:52) ~[elasticsearch-5.6.9.jar:5.6.9]
  423.     at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) ~[elasticsearch-5.6.9.jar:5.6.9]
  424.     at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) ~[elasticsearch-5.6.9.jar:5.6.9]
  425.     at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84) ~[elasticsearch-5.6.9.jar:5.6.9]
  426.     at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) ~[elasticsearch-5.6.9.jar:5.6.9]
  427.     at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) ~[elasticsearch-5.6.9.jar:5.6.9]
  428.     at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408) ~[elasticsearch-5.6.9.jar:5.6.9]
  429.     at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:730) ~[elasticsearch-5.6.9.jar:5.6.9]
  430.     at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.nodesStats(AbstractClient.java:826) ~[elasticsearch-5.6.9.jar:5.6.9]
  431.     at org.elasticsearch.cluster.InternalClusterInfoService.updateNodeStats(InternalClusterInfoService.java:256) ~[elasticsearch-5.6.9.jar:5.6.9]
  432.     at org.elasticsearch.cluster.InternalClusterInfoService.refresh(InternalClusterInfoService.java:292) ~[elasticsearch-5.6.9.jar:5.6.9]
  433.     at org.elasticsearch.cluster.InternalClusterInfoService.maybeRefresh(InternalClusterInfoService.java:277) ~[elasticsearch-5.6.9.jar:5.6.9]
  434.     at org.elasticsearch.cluster.InternalClusterInfoService.lambda$onMaster$0(InternalClusterInfoService.java:137) ~[elasticsearch-5.6.9.jar:5.6.9]
  435.     at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.9.jar:5.6.9]
  436.     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
  437.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
  438.     at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  439. [2019-01-25T10:43:23,020][DEBUG][o.e.a.a.i.s.TransportIndicesStatsAction] [yRj9xlW] failed to execute [indices:monitor/stats] on node [wAVZzkyrSfyBmuGRw0Bltw]
  440. org.elasticsearch.transport.NodeNotConnectedException: [wAVZzky][10.231.83.146:9302] Node not connected
  441.     at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:640) ~[elasticsearch-5.6.9.jar:5.6.9]
  442.     at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:117) ~[elasticsearch-5.6.9.jar:5.6.9]
  443.     at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:540) ~[elasticsearch-5.6.9.jar:5.6.9]
  444.     at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:503) ~[elasticsearch-5.6.9.jar:5.6.9]
  445.     at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.sendNodeRequest(TransportBroadcastByNodeAction.java:322) ~[elasticsearch-5.6.9.jar:5.6.9]
  446.     at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.start(TransportBroadcastByNodeAction.java:311) ~[elasticsearch-5.6.9.jar:5.6.9]
  447.     at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:234) ~[elasticsearch-5.6.9.jar:5.6.9]
  448.     at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:79) ~[elasticsearch-5.6.9.jar:5.6.9]
  449.     at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) ~[elasticsearch-5.6.9.jar:5.6.9]
  450.     at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) ~[elasticsearch-5.6.9.jar:5.6.9]
  451.     at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84) ~[elasticsearch-5.6.9.jar:5.6.9]
  452.     at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) ~[elasticsearch-5.6.9.jar:5.6.9]
  453.     at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) ~[elasticsearch-5.6.9.jar:5.6.9]
  454.     at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408) ~[elasticsearch-5.6.9.jar:5.6.9]
  455.     at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1256) ~[elasticsearch-5.6.9.jar:5.6.9]
  456.     at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.stats(AbstractClient.java:1577) ~[elasticsearch-5.6.9.jar:5.6.9]
  457.     at org.elasticsearch.cluster.InternalClusterInfoService.updateIndicesStats(InternalClusterInfoService.java:270) ~[elasticsearch-5.6.9.jar:5.6.9]
  458.     at org.elasticsearch.cluster.InternalClusterInfoService.refresh(InternalClusterInfoService.java:321) ~[elasticsearch-5.6.9.jar:5.6.9]
  459.     at org.elasticsearch.cluster.InternalClusterInfoService.maybeRefresh(InternalClusterInfoService.java:277) ~[elasticsearch-5.6.9.jar:5.6.9]
  460.     at org.elasticsearch.cluster.InternalClusterInfoService.lambda$onMaster$0(InternalClusterInfoService.java:137) ~[elasticsearch-5.6.9.jar:5.6.9]
  461.     at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.9.jar:5.6.9]
  462.     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
  463.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
  464.     at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  465. [2019-01-25T10:43:23,034][INFO ][o.e.c.r.a.AllocationService] [yRj9xlW] Cluster health status changed from [YELLOW] to [RED] (reason: []).
  466. [2019-01-25T10:43:23,035][INFO ][o.e.c.s.ClusterService   ] [yRj9xlW] removed {{wAVZzky}{wAVZzkyrSfyBmuGRw0Bltw}{r1L7UXpESgiM_Kxq_jFUjw}{10.231.83.146}{10.231.83.146:9302},}, reason: zen-disco-node-failed({wAVZzky}{wAVZzkyrSfyBmuGRw0Bltw}{r1L7UXpESgiM_Kxq_jFUjw}{10.231.83.146}{10.231.83.146:9302}), reason(transport disconnected)
  467. [2019-01-25T10:43:24,112][INFO ][o.e.c.r.a.AllocationService] [yRj9xlW] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[vanmessagetopic][4], [vanmessagetopic][3]] ...]).
  468. [2019-01-25T10:43:24,484][INFO ][o.e.c.s.ClusterService   ] [yRj9xlW] added {{wAVZzky}{wAVZzkyrSfyBmuGRw0Bltw}{Qc8kODytQreZFj_24t5O7g}{10.231.83.146}{10.231.83.146:9302},}, reason: zen-disco-node-join
  469. [2019-01-25T10:43:29,088][INFO ][o.e.c.r.a.AllocationService] [yRj9xlW] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[device][0]] ...]).
  470. [2019-01-25T10:44:30,199][INFO ][o.e.c.r.a.AllocationService] [yRj9xlW] Cluster health status changed from [GREEN] to [YELLOW] (reason: []).
  471. [2019-01-25T10:44:30,200][INFO ][o.e.c.s.ClusterService   ] [yRj9xlW] removed {{lJgJwYh}{lJgJwYhmQA-fucaOBHD5Vg}{lbN7QcyiSWCOBYNL4jNkdg}{10.231.83.146}{10.231.83.146:9301},}, reason: zen-disco-node-failed({lJgJwYh}{lJgJwYhmQA-fucaOBHD5Vg}{lbN7QcyiSWCOBYNL4jNkdg}{10.231.83.146}{10.231.83.146:9301}), reason(failed to ping, tried [3] times, each with maximum [4s] timeout)
  472. [2019-01-25T10:44:38,008][DEBUG][o.e.a.a.c.n.s.TransportNodesStatsAction] [yRj9xlW] failed to execute on node [lJgJwYhmQA-fucaOBHD5Vg]
  473. org.elasticsearch.transport.ReceiveTimeoutTransportException: [lJgJwYh][10.231.83.146:9301][cluster:monitor/nodes/stats[n]] request_id [328] timed out after [15000ms]
  474.     at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:961) [elasticsearch-5.6.9.jar:5.6.9]
  475.     at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.9.jar:5.6.9]
  476.     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
  477.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
  478.     at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  479. [2019-01-25T10:44:38,007][DEBUG][o.e.a.a.c.n.s.TransportNodesStatsAction] [yRj9xlW] failed to execute on node [wAVZzkyrSfyBmuGRw0Bltw]
  480. org.elasticsearch.transport.ReceiveTimeoutTransportException: [wAVZzky][10.231.83.146:9302][cluster:monitor/nodes/stats[n]] request_id [329] timed out after [15000ms]
  481.     at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:961) [elasticsearch-5.6.9.jar:5.6.9]
  482.     at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.9.jar:5.6.9]
  483.     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
  484.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
  485.     at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  486. [2019-01-25T10:45:00,202][WARN ][o.e.c.s.ClusterService   ] [yRj9xlW] failing [zen-disco-node-failed({lJgJwYh}{lJgJwYhmQA-fucaOBHD5Vg}{lbN7QcyiSWCOBYNL4jNkdg}{10.231.83.146}{10.231.83.146:9301}), reason(failed to ping, tried [3] times, each with maximum [4s] timeout)]: failed to commit cluster state version [200]
  487. org.elasticsearch.discovery.Discovery$FailedToCommitClusterStateException: timed out while waiting for enough masters to ack sent cluster state. [1] left
  488.     at org.elasticsearch.discovery.zen.PublishClusterStateAction$SendingController.waitForCommit(PublishClusterStateAction.java:574) ~[elasticsearch-5.6.9.jar:5.6.9]
  489.     at org.elasticsearch.discovery.zen.PublishClusterStateAction.innerPublish(PublishClusterStateAction.java:202) ~[elasticsearch-5.6.9.jar:5.6.9]
  490.     at org.elasticsearch.discovery.zen.PublishClusterStateAction.publish(PublishClusterStateAction.java:167) ~[elasticsearch-5.6.9.jar:5.6.9]
  491.     at org.elasticsearch.discovery.zen.ZenDiscovery.publish(ZenDiscovery.java:310) ~[elasticsearch-5.6.9.jar:5.6.9]
  492.     at org.elasticsearch.cluster.service.ClusterService.publishAndApplyChanges(ClusterService.java:741) [elasticsearch-5.6.9.jar:5.6.9]
  493.     at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:587) [elasticsearch-5.6.9.jar:5.6.9]
  494.     at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) [elasticsearch-5.6.9.jar:5.6.9]
  495.     at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-5.6.9.jar:5.6.9]
  496.     at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-5.6.9.jar:5.6.9]
  497.     at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.9.jar:5.6.9]
  498.     at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) [elasticsearch-5.6.9.jar:5.6.9]
  499.     at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) [elasticsearch-5.6.9.jar:5.6.9]
  500.     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
  501.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
  502.     at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  503. [2019-01-25T10:45:00,204][ERROR][o.e.d.z.ZenDiscovery     ] [yRj9xlW] unexpected failure during [zen-disco-node-failed({lJgJwYh}{lJgJwYhmQA-fucaOBHD5Vg}{lbN7QcyiSWCOBYNL4jNkdg}{10.231.83.146}{10.231.83.146:9301}), reason(failed to ping, tried [3] times, each with maximum [4s] timeout)]
  504. org.elasticsearch.discovery.Discovery$FailedToCommitClusterStateException: timed out while waiting for enough masters to ack sent cluster state. [1] left
  505.     at org.elasticsearch.discovery.zen.PublishClusterStateAction$SendingController.waitForCommit(PublishClusterStateAction.java:574) ~[elasticsearch-5.6.9.jar:5.6.9]
  506.     at org.elasticsearch.discovery.zen.PublishClusterStateAction.innerPublish(PublishClusterStateAction.java:202) ~[elasticsearch-5.6.9.jar:5.6.9]
  507.     at org.elasticsearch.discovery.zen.PublishClusterStateAction.publish(PublishClusterStateAction.java:167) ~[elasticsearch-5.6.9.jar:5.6.9]
  508.     at org.elasticsearch.discovery.zen.ZenDiscovery.publish(ZenDiscovery.java:310) ~[elasticsearch-5.6.9.jar:5.6.9]
  509.     at org.elasticsearch.cluster.service.ClusterService.publishAndApplyChanges(ClusterService.java:741) [elasticsearch-5.6.9.jar:5.6.9]
  510.     at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:587) [elasticsearch-5.6.9.jar:5.6.9]
  511.     at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) [elasticsearch-5.6.9.jar:5.6.9]
  512.     at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-5.6.9.jar:5.6.9]
  513.     at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-5.6.9.jar:5.6.9]
  514.     at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.9.jar:5.6.9]
  515.     at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) [elasticsearch-5.6.9.jar:5.6.9]
  516.     at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) [elasticsearch-5.6.9.jar:5.6.9]
  517.     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
  518.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
  519.     at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  520. [2019-01-25T10:45:00,204][WARN ][o.e.c.s.ClusterService   ] [yRj9xlW] cluster state update task [zen-disco-node-failed({lJgJwYh}{lJgJwYhmQA-fucaOBHD5Vg}{lbN7QcyiSWCOBYNL4jNkdg}{10.231.83.146}{10.231.83.146:9301}), reason(failed to ping, tried [3] times, each with maximum [4s] timeout)] took [30s] above the warn threshold of 30s
  521. [2019-01-25T10:45:00,208][INFO ][o.e.c.r.a.AllocationService] [yRj9xlW] Cluster health status changed from [GREEN] to [YELLOW] (reason: []).
  522. [2019-01-25T10:45:00,209][INFO ][o.e.c.s.ClusterService   ] [yRj9xlW] removed {{wAVZzky}{wAVZzkyrSfyBmuGRw0Bltw}{Qc8kODytQreZFj_24t5O7g}{10.231.83.146}{10.231.83.146:9302},}, reason: zen-disco-node-failed({wAVZzky}{wAVZzkyrSfyBmuGRw0Bltw}{Qc8kODytQreZFj_24t5O7g}{10.231.83.146}{10.231.83.146:9302}), reason(failed to ping, tried [3] times, each with maximum [4s] timeout)
  523. [2019-01-25T10:45:21,837][DEBUG][o.e.a.a.i.s.TransportIndicesStatsAction] [yRj9xlW] failed to execute [indices:monitor/stats] on node [wAVZzkyrSfyBmuGRw0Bltw]
  524. org.elasticsearch.transport.NodeDisconnectedException: [wAVZzky][10.231.83.146:9302][indices:monitor/stats[n]] disconnected
  525. [2019-01-25T10:45:21,840][WARN ][o.e.g.GatewayAllocator$InternalReplicaShardAllocator] [yRj9xlW] [vanmessagetopic][1]: failed to list shard for shard_store on node [wAVZzkyrSfyBmuGRw0Bltw]
  526. org.elasticsearch.action.FailedNodeException: Failed node [wAVZzkyrSfyBmuGRw0Bltw]
  527.     at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:239) ~[elasticsearch-5.6.9.jar:5.6.9]
  528.     at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$200(TransportNodesAction.java:153) ~[elasticsearch-5.6.9.jar:5.6.9]
  529.     at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:211) ~[elasticsearch-5.6.9.jar:5.6.9]
  530.     at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1077) ~[elasticsearch-5.6.9.jar:5.6.9]
  531.     at org.elasticsearch.transport.TransportService$Adapter.lambda$onConnectionClosed$6(TransportService.java:903) ~[elasticsearch-5.6.9.jar:5.6.9]
  532.     at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.9.jar:5.6.9]
  533.     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
  534.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
  535.     at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  536. Caused by: org.elasticsearch.transport.NodeDisconnectedException: [wAVZzky][10.231.83.146:9302][internal:cluster/nodes/indices/shard/store[n]] disconnected
  537. [2019-01-25T10:45:21,947][WARN ][o.e.t.n.Netty4Transport  ] [yRj9xlW] write and flush on the network layer failed (channel: [id: 0x237db2db, L:/10.231.83.238:9300 ! R:/10.231.83.146:51160])
  538. java.io.IOException: Broken pipe
  539.     at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[?:?]
  540.     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[?:?]
  541.     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[?:?]
  542.     at sun.nio.ch.IOUtil.write(IOUtil.java:51) ~[?:?]
  543.     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) ~[?:?]
  544.     at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:417) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  545.     at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:856) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  546.     at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:362) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  547.     at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:823) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  548.     at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1296) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  549.     at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  550.     at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  551.     at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  552.     at io.netty.handler.logging.LoggingHandler.flush(LoggingHandler.java:265) ~[netty-handler-4.1.13.Final.jar:4.1.13.Final]
  553.     at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  554.     at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  555.     at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  556.     at io.netty.channel.ChannelDuplexHandler.flush(ChannelDuplexHandler.java:117) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  557.     at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  558.     at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  559.     at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  560.     at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  561.     at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  562.     at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1032) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  563.     at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:296) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  564.     at org.elasticsearch.transport.netty4.Netty4Transport.sendMessage(Netty4Transport.java:400) [transport-netty4-5.6.9.jar:5.6.9]
  565.     at org.elasticsearch.transport.netty4.Netty4Transport.sendMessage(Netty4Transport.java:96) [transport-netty4-5.6.9.jar:5.6.9]
  566.     at org.elasticsearch.transport.TcpTransport.internalSendMessage(TcpTransport.java:1138) [elasticsearch-5.6.9.jar:5.6.9]
  567.     at org.elasticsearch.transport.TcpTransport.sendResponse(TcpTransport.java:1205) [elasticsearch-5.6.9.jar:5.6.9]
  568.     at org.elasticsearch.transport.TcpTransport.sendResponse(TcpTransport.java:1181) [elasticsearch-5.6.9.jar:5.6.9]
  569.     at org.elasticsearch.transport.TcpTransportChannel.sendResponse(TcpTransportChannel.java:67) [elasticsearch-5.6.9.jar:5.6.9]
  570.     at org.elasticsearch.transport.TcpTransportChannel.sendResponse(TcpTransportChannel.java:61) [elasticsearch-5.6.9.jar:5.6.9]
  571.     at org.elasticsearch.transport.DelegatingTransportChannel.sendResponse(DelegatingTransportChannel.java:60) [elasticsearch-5.6.9.jar:5.6.9]
  572.     at org.elasticsearch.transport.RequestHandlerRegistry$TransportChannelWrapper.sendResponse(RequestHandlerRegistry.java:111) [elasticsearch-5.6.9.jar:5.6.9]
  573.     at org.elasticsearch.discovery.zen.UnicastZenPing$UnicastPingRequestHandler.messageReceived(UnicastZenPing.java:616) [elasticsearch-5.6.9.jar:5.6.9]
  574.     at org.elasticsearch.discovery.zen.UnicastZenPing$UnicastPingRequestHandler.messageReceived(UnicastZenPing.java:608) [elasticsearch-5.6.9.jar:5.6.9]
  575.     at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33) [elasticsearch-5.6.9.jar:5.6.9]
  576.     at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) [elasticsearch-5.6.9.jar:5.6.9]
  577.     at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1556) [elasticsearch-5.6.9.jar:5.6.9]
  578.     at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.9.jar:5.6.9]
  579.     at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:110) [elasticsearch-5.6.9.jar:5.6.9]
  580.     at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1513) [elasticsearch-5.6.9.jar:5.6.9]
  581.     at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1396) [elasticsearch-5.6.9.jar:5.6.9]
  582.     at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:75) [transport-netty4-5.6.9.jar:5.6.9]
  583.     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  584.     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  585.     at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  586.     at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.13.Final.jar:4.1.13.Final]
  587.     at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297) [netty-codec-4.1.13.Final.jar:4.1.13.Final]
  588.     at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413) [netty-codec-4.1.13.Final.jar:4.1.13.Final]
  589.     at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) [netty-codec-4.1.13.Final.jar:4.1.13.Final]
  590.     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  591.     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  592.     at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  593.     at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  594.     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  595.     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  596.     at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  597.     at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) [netty-handler-4.1.13.Final.jar:4.1.13.Final]
  598.     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  599.     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  600.     at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  601.     at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  602.     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  603.     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  604.     at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  605.     at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  606.     at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  607.     at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  608.     at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  609.     at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  610.     at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.13.Final.jar:4.1.13.Final]
  611.     at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  612. [2019-01-25T10:45:23,010][DEBUG][o.e.a.a.c.n.s.TransportNodesStatsAction] [yRj9xlW] failed to execute on node [wAVZzkyrSfyBmuGRw0Bltw]
  613. org.elasticsearch.transport.NodeNotConnectedException: [wAVZzky][10.231.83.146:9302] Node not connected
  614.     at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:640) ~[elasticsearch-5.6.9.jar:5.6.9]
  615.     at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:117) ~[elasticsearch-5.6.9.jar:5.6.9]
  616.     at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:540) ~[elasticsearch-5.6.9.jar:5.6.9]
  617.     at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:516) ~[elasticsearch-5.6.9.jar:5.6.9]
  618.     at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.start(TransportNodesAction.java:197) ~[elasticsearch-5.6.9.jar:5.6.9]
  619.     at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:89) ~[elasticsearch-5.6.9.jar:5.6.9]
  620.     at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:52) ~[elasticsearch-5.6.9.jar:5.6.9]
  621.     at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) ~[elasticsearch-5.6.9.jar:5.6.9]
  622.     at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) ~[elasticsearch-5.6.9.jar:5.6.9]
  623.     at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84) ~[elasticsearch-5.6.9.jar:5.6.9]
  624.     at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) ~[elasticsearch-5.6.9.jar:5.6.9]
  625.     at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) ~[elasticsearch-5.6.9.jar:5.6.9]
  626.     at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408) ~[elasticsearch-5.6.9.jar:5.6.9]
  627.     at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:730) ~[elasticsearch-5.6.9.jar:5.6.9]
  628.     at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.nodesStats(AbstractClient.java:826) ~[elasticsearch-5.6.9.jar:5.6.9]
  629.     at org.elasticsearch.cluster.InternalClusterInfoService.updateNodeStats(InternalClusterInfoService.java:256) ~[elasticsearch-5.6.9.jar:5.6.9]
  630.     at org.elasticsearch.cluster.InternalClusterInfoService.refresh(InternalClusterInfoService.java:292) ~[elasticsearch-5.6.9.jar:5.6.9]
  631.     at org.elasticsearch.cluster.InternalClusterInfoService.maybeRefresh(InternalClusterInfoService.java:277) ~[elasticsearch-5.6.9.jar:5.6.9]
  632.     at org.elasticsearch.cluster.InternalClusterInfoService.access$500(InternalClusterInfoService.java:67) ~[elasticsearch-5.6.9.jar:5.6.9]
  633.     at org.elasticsearch.cluster.InternalClusterInfoService$SubmitReschedulingClusterInfoUpdatedJob.lambda$run$0(InternalClusterInfoService.java:224) ~[elasticsearch-5.6.9.jar:5.6.9]
  634.     at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.9.jar:5.6.9]
  635.     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
  636.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
  637.     at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  638. [2019-01-25T10:45:23,014][DEBUG][o.e.a.a.i.s.TransportIndicesStatsAction] [yRj9xlW] failed to execute [indices:monitor/stats] on node [wAVZzkyrSfyBmuGRw0Bltw]
  639. org.elasticsearch.transport.NodeNotConnectedException: [wAVZzky][10.231.83.146:9302] Node not connected
  640.     at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:640) ~[elasticsearch-5.6.9.jar:5.6.9]
  641.     at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:117) ~[elasticsearch-5.6.9.jar:5.6.9]
  642.     at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:540) ~[elasticsearch-5.6.9.jar:5.6.9]
  643.     at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:503) ~[elasticsearch-5.6.9.jar:5.6.9]
  644.     at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.sendNodeRequest(TransportBroadcastByNodeAction.java:322) ~[elasticsearch-5.6.9.jar:5.6.9]
  645.     at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.start(TransportBroadcastByNodeAction.java:311) ~[elasticsearch-5.6.9.jar:5.6.9]
  646.     at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:234) ~[elasticsearch-5.6.9.jar:5.6.9]
  647.     at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:79) ~[elasticsearch-5.6.9.jar:5.6.9]
  648.     at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) ~[elasticsearch-5.6.9.jar:5.6.9]
  649.     at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) ~[elasticsearch-5.6.9.jar:5.6.9]
  650.     at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84) ~[elasticsearch-5.6.9.jar:5.6.9]
  651.     at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) ~[elasticsearch-5.6.9.jar:5.6.9]
  652.     at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) ~[elasticsearch-5.6.9.jar:5.6.9]
  653.     at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408) ~[elasticsearch-5.6.9.jar:5.6.9]
  654.     at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1256) ~[elasticsearch-5.6.9.jar:5.6.9]
  655.     at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.stats(AbstractClient.java:1577) ~[elasticsearch-5.6.9.jar:5.6.9]
  656.     at org.elasticsearch.cluster.InternalClusterInfoService.updateIndicesStats(InternalClusterInfoService.java:270) ~[elasticsearch-5.6.9.jar:5.6.9]
  657.     at org.elasticsearch.cluster.InternalClusterInfoService.refresh(InternalClusterInfoService.java:321) ~[elasticsearch-5.6.9.jar:5.6.9]
  658.     at org.elasticsearch.cluster.InternalClusterInfoService.maybeRefresh(InternalClusterInfoService.java:277) ~[elasticsearch-5.6.9.jar:5.6.9]
  659.     at org.elasticsearch.cluster.InternalClusterInfoService.access$500(InternalClusterInfoService.java:67) ~[elasticsearch-5.6.9.jar:5.6.9]
  660.     at org.elasticsearch.cluster.InternalClusterInfoService$SubmitReschedulingClusterInfoUpdatedJob.lambda$run$0(InternalClusterInfoService.java:224) ~[elasticsearch-5.6.9.jar:5.6.9]
  661.     at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.9.jar:5.6.9]
  662.     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
  663.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
  664.     at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  665. [2019-01-25T10:45:23,026][WARN ][o.e.t.TransportService   ] [yRj9xlW] Received response for a request that has timed out, sent [60019ms] ago, timed out [45019ms] ago, action [cluster:monitor/nodes/stats[n]], node [{lJgJwYh}{lJgJwYhmQA-fucaOBHD5Vg}{lbN7QcyiSWCOBYNL4jNkdg}{10.231.83.146}{10.231.83.146:9301}], id [328]
  666. ^C[2019-01-25T10:45:24,254][INFO ][o.e.n.Node               ] [yRj9xlW] stopping ...
  667. [2019-01-25T10:45:24,515][DEBUG][o.e.a.a.i.s.TransportIndicesStatsAction] [yRj9xlW] failed to execute [indices:monitor/stats] on node [lJgJwYhmQA-fucaOBHD5Vg]
  668. org.elasticsearch.transport.NodeDisconnectedException: [lJgJwYh][10.231.83.146:9301][indices:monitor/stats[n]] disconnected
  669. [2019-01-25T10:45:24,518][WARN ][o.e.t.n.Netty4Transport  ] [yRj9xlW] write and flush on the network layer failed (channel: [id: 0xdc753be3, L:/10.231.83.238:9300 ! R:/10.231.83.146:51098])
  670. java.nio.channels.ClosedChannelException: null
  671.     at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
  672. [2019-01-25T10:45:24,515][WARN ][o.e.g.GatewayAllocator$InternalReplicaShardAllocator] [yRj9xlW] [vanmessagetopic][3]: failed to list shard for shard_store on node [lJgJwYhmQA-fucaOBHD5Vg]
  673. org.elasticsearch.action.FailedNodeException: Failed node [lJgJwYhmQA-fucaOBHD5Vg]
  674.     at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:239) ~[elasticsearch-5.6.9.jar:5.6.9]
  675.     at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$200(TransportNodesAction.java:153) ~[elasticsearch-5.6.9.jar:5.6.9]
  676.     at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:211) ~[elasticsearch-5.6.9.jar:5.6.9]
  677.     at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1077) ~[elasticsearch-5.6.9.jar:5.6.9]
  678.     at org.elasticsearch.transport.TransportService$Adapter.lambda$onConnectionClosed$6(TransportService.java:903) ~[elasticsearch-5.6.9.jar:5.6.9]
  679.     at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.9.jar:5.6.9]
  680.     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
  681.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
  682.     at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  683. Caused by: org.elasticsearch.transport.NodeDisconnectedException: [lJgJwYh][10.231.83.146:9301][internal:cluster/nodes/indices/shard/store[n]] disconnected
  684. [2019-01-25T10:45:24,519][WARN ][o.e.t.n.Netty4Transport  ] [yRj9xlW] write and flush on the network layer failed (channel: [id: 0x3fc1eeb3, L:/10.231.83.238:9300 ! R:/10.231.83.146:51100])
  685. java.nio.channels.ClosedChannelException: null
  686.     at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
  687. [2019-01-25T10:45:24,522][WARN ][o.e.g.GatewayAllocator$InternalReplicaShardAllocator] [yRj9xlW] [vanmessagetopic][2]: failed to list shard for shard_store on node [lJgJwYhmQA-fucaOBHD5Vg]
  688. org.elasticsearch.action.FailedNodeException: Failed node [lJgJwYhmQA-fucaOBHD5Vg]
  689.     at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:239) ~[elasticsearch-5.6.9.jar:5.6.9]
  690.     at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$200(TransportNodesAction.java:153) ~[elasticsearch-5.6.9.jar:5.6.9]
  691.     at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:211) ~[elasticsearch-5.6.9.jar:5.6.9]
  692.     at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1077) ~[elasticsearch-5.6.9.jar:5.6.9]
  693.     at org.elasticsearch.transport.TransportService$Adapter.lambda$onConnectionClosed$6(TransportService.java:903) ~[elasticsearch-5.6.9.jar:5.6.9]
  694.     at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.9.jar:5.6.9]
  695.     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
  696.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
  697.     at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  698. Caused by: org.elasticsearch.transport.NodeDisconnectedException: [lJgJwYh][10.231.83.146:9301][internal:cluster/nodes/indices/shard/store[n]] disconnected
  699. [2019-01-25T10:45:24,525][WARN ][o.e.c.s.ClusterService   ] [yRj9xlW] failing [zen-disco-node-failed({wAVZzky}{wAVZzkyrSfyBmuGRw0Bltw}{Qc8kODytQreZFj_24t5O7g}{10.231.83.146}{10.231.83.146:9302}), reason(failed to ping, tried [3] times, each with maximum [4s] timeout)]: failed to commit cluster state version [200]
  700. org.elasticsearch.discovery.Discovery$FailedToCommitClusterStateException: failed to get enough masters to ack sent cluster state. [1] left
  701.     at org.elasticsearch.discovery.zen.PublishClusterStateAction$SendingController.waitForCommit(PublishClusterStateAction.java:574) ~[elasticsearch-5.6.9.jar:5.6.9]
  702.     at org.elasticsearch.discovery.zen.PublishClusterStateAction.innerPublish(PublishClusterStateAction.java:202) ~[elasticsearch-5.6.9.jar:5.6.9]
  703.     at org.elasticsearch.discovery.zen.PublishClusterStateAction.publish(PublishClusterStateAction.java:167) ~[elasticsearch-5.6.9.jar:5.6.9]
  704.     at org.elasticsearch.discovery.zen.ZenDiscovery.publish(ZenDiscovery.java:310) ~[elasticsearch-5.6.9.jar:5.6.9]
  705.     at org.elasticsearch.cluster.service.ClusterService.publishAndApplyChanges(ClusterService.java:741) [elasticsearch-5.6.9.jar:5.6.9]
  706.     at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:587) [elasticsearch-5.6.9.jar:5.6.9]
  707.     at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) [elasticsearch-5.6.9.jar:5.6.9]
  708.     at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-5.6.9.jar:5.6.9]
  709.     at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-5.6.9.jar:5.6.9]
  710.     at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.9.jar:5.6.9]
  711.     at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) [elasticsearch-5.6.9.jar:5.6.9]
  712.     at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) [elasticsearch-5.6.9.jar:5.6.9]
  713.     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
  714.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
  715.     at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  716. [2019-01-25T10:45:24,526][WARN ][o.e.g.GatewayAllocator$InternalReplicaShardAllocator] [yRj9xlW] [vanmessagetopic][0]: failed to list shard for shard_store on node [lJgJwYhmQA-fucaOBHD5Vg]
  717. org.elasticsearch.action.FailedNodeException: Failed node [lJgJwYhmQA-fucaOBHD5Vg]
  718.     at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:239) ~[elasticsearch-5.6.9.jar:5.6.9]
  719.     at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$200(TransportNodesAction.java:153) ~[elasticsearch-5.6.9.jar:5.6.9]
  720.     at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:211) ~[elasticsearch-5.6.9.jar:5.6.9]
  721.     at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1077) ~[elasticsearch-5.6.9.jar:5.6.9]
  722.     at org.elasticsearch.transport.TransportService$Adapter.lambda$onConnectionClosed$6(TransportService.java:903) ~[elasticsearch-5.6.9.jar:5.6.9]
  723.     at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.9.jar:5.6.9]
  724.     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
  725.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
  726.     at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  727. Caused by: org.elasticsearch.transport.NodeDisconnectedException: [lJgJwYh][10.231.83.146:9301][internal:cluster/nodes/indices/shard/store[n]] disconnected
  728. [2019-01-25T10:45:24,521][WARN ][o.e.t.n.Netty4Transport  ] [yRj9xlW] write and flush on the network layer failed (channel: [id: 0x80fc56a7, L:/10.231.83.238:9300 ! R:/10.231.83.146:51090])
  729. java.io.IOException: Broken pipe
  730.     at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[?:?]
  731.     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[?:?]
  732.     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[?:?]
  733.     at sun.nio.ch.IOUtil.write(IOUtil.java:51) ~[?:?]
  734.     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) ~[?:?]
  735.     at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:417) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  736.     at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:856) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  737.     at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:362) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  738.     at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:823) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  739.     at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1296) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  740.     at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  741.     at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  742.     at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  743.     at io.netty.handler.logging.LoggingHandler.flush(LoggingHandler.java:265) ~[netty-handler-4.1.13.Final.jar:4.1.13.Final]
  744.     at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  745.     at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  746.     at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  747.     at io.netty.channel.ChannelDuplexHandler.flush(ChannelDuplexHandler.java:117) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  748.     at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  749.     at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  750.     at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  751.     at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  752.     at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  753.     at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1032) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  754.     at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:296) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  755.     at org.elasticsearch.transport.netty4.Netty4Transport.sendMessage(Netty4Transport.java:400) [transport-netty4-5.6.9.jar:5.6.9]
  756.     at org.elasticsearch.transport.netty4.Netty4Transport.sendMessage(Netty4Transport.java:96) [transport-netty4-5.6.9.jar:5.6.9]
  757.     at org.elasticsearch.transport.TcpTransport.internalSendMessage(TcpTransport.java:1138) [elasticsearch-5.6.9.jar:5.6.9]
  758.     at org.elasticsearch.transport.TcpTransport.sendErrorResponse(TcpTransport.java:1170) [elasticsearch-5.6.9.jar:5.6.9]
  759.     at org.elasticsearch.transport.TcpTransportChannel.sendResponse(TcpTransportChannel.java:76) [elasticsearch-5.6.9.jar:5.6.9]
  760.     at org.elasticsearch.transport.TcpTransport$RequestHandler.onFailure(TcpTransport.java:1569) [elasticsearch-5.6.9.jar:5.6.9]
  761.     at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39) [elasticsearch-5.6.9.jar:5.6.9]
  762.     at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:110) [elasticsearch-5.6.9.jar:5.6.9]
  763.     at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1513) [elasticsearch-5.6.9.jar:5.6.9]
  764.     at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1396) [elasticsearch-5.6.9.jar:5.6.9]
  765.     at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:75) [transport-netty4-5.6.9.jar:5.6.9]
  766.     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  767.     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  768.     at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  769.     at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.13.Final.jar:4.1.13.Final]
  770.     at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297) [netty-codec-4.1.13.Final.jar:4.1.13.Final]
  771.     at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413) [netty-codec-4.1.13.Final.jar:4.1.13.Final]
  772.     at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) [netty-codec-4.1.13.Final.jar:4.1.13.Final]
  773.     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  774.     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  775.     at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  776.     at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  777.     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  778.     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  779.     at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  780.     at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) [netty-handler-4.1.13.Final.jar:4.1.13.Final]
  781.     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  782.     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  783.     at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  784.     at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  785.     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  786.     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  787.     at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  788.     at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  789.     at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  790.     at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  791.     at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  792.     at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  793.     at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.13.Final.jar:4.1.13.Final]
  794.     at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  795. [2019-01-25T10:45:24,536][WARN ][o.e.c.NodeConnectionsService] [yRj9xlW] failed to connect to node {wAVZzky}{wAVZzkyrSfyBmuGRw0Bltw}{Qc8kODytQreZFj_24t5O7g}{10.231.83.146}{10.231.83.146:9302} (tried [1] times)
  796. org.elasticsearch.transport.ConnectTransportException: [wAVZzky][10.231.83.146:9302] connect_timeout[30s]
  797.     at org.elasticsearch.transport.netty4.Netty4Transport.connectToChannels(Netty4Transport.java:363) ~[?:?]
  798.     at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:570) ~[elasticsearch-5.6.9.jar:5.6.9]
  799.     at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:473) ~[elasticsearch-5.6.9.jar:5.6.9]
  800.     at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:342) ~[elasticsearch-5.6.9.jar:5.6.9]
  801.     at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:329) ~[elasticsearch-5.6.9.jar:5.6.9]
  802.     at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:154) [elasticsearch-5.6.9.jar:5.6.9]
  803.     at org.elasticsearch.cluster.NodeConnectionsService$1.doRun(NodeConnectionsService.java:107) [elasticsearch-5.6.9.jar:5.6.9]
  804.     at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:674) [elasticsearch-5.6.9.jar:5.6.9]
  805.     at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.9.jar:5.6.9]
  806.     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
  807.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
  808.     at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  809. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: 10.231.83.146/10.231.83.146:9302
  810.     at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  811.     at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
  812.     at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352) ~[?:?]
  813.     at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  814.     at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?]
  815.     at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?]
  816.     at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?]
  817.     at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?]
  818.     at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
  819.     ... 1 more
  820. Caused by: java.net.ConnectException: Connection refused
  821.     at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  822.     at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
  823.     at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352) ~[?:?]
  824.     at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  825.     at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?]
  826.     at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?]
  827.     at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?]
  828.     at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?]
  829.     at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
  830.     ... 1 more
  831. [2019-01-25T10:45:24,557][WARN ][o.e.t.n.Netty4Transport  ] [yRj9xlW] write and flush on the network layer failed (channel: [id: 0xe0ce28dc, L:/10.231.83.238:9300 ! R:/10.231.83.146:51106])
  832. java.io.IOException: Broken pipe
  833.     at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[?:?]
  834.     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[?:?]
  835.     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[?:?]
  836.     at sun.nio.ch.IOUtil.write(IOUtil.java:51) ~[?:?]
  837.     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) ~[?:?]
  838.     at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:417) ~[netty-transport-4.1.13.Final.jar:4.1.13.Final]
  839.     at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:856) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  840.     at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  841.     at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:823) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  842.     at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1296) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  843.     at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  844.     at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  845.     at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  846.     at io.netty.handler.logging.LoggingHandler.flush(LoggingHandler.java:265) [netty-handler-4.1.13.Final.jar:4.1.13.Final]
  847.     at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  848.     at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  849.     at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  850.     at io.netty.channel.ChannelDuplexHandler.flush(ChannelDuplexHandler.java:117) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  851.     at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  852.     at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  853.     at io.netty.channel.AbstractChannelHandlerContext.access$1500(AbstractChannelHandlerContext.java:38) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  854.     at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1137) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  855.     at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:1078) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  856.     at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [netty-common-4.1.13.Final.jar:4.1.13.Final]
  857.     at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403) [netty-common-4.1.13.Final.jar:4.1.13.Final]
  858.     at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  859.     at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.13.Final.jar:4.1.13.Final]
  860.     at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  861. [2019-01-25T10:45:24,543][WARN ][o.e.c.NodeConnectionsService] [yRj9xlW] failed to connect to node {lJgJwYh}{lJgJwYhmQA-fucaOBHD5Vg}{lbN7QcyiSWCOBYNL4jNkdg}{10.231.83.146}{10.231.83.146:9301} (tried [1] times)
  862. org.elasticsearch.transport.ConnectTransportException: [lJgJwYh][10.231.83.146:9301] connect_timeout[30s]
  863.     at org.elasticsearch.transport.netty4.Netty4Transport.connectToChannels(Netty4Transport.java:363) ~[?:?]
  864.     at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:570) ~[elasticsearch-5.6.9.jar:5.6.9]
  865.     at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:473) ~[elasticsearch-5.6.9.jar:5.6.9]
  866.     at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:342) ~[elasticsearch-5.6.9.jar:5.6.9]
  867.     at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:329) ~[elasticsearch-5.6.9.jar:5.6.9]
  868.     at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:154) [elasticsearch-5.6.9.jar:5.6.9]
  869.     at org.elasticsearch.cluster.NodeConnectionsService$1.doRun(NodeConnectionsService.java:107) [elasticsearch-5.6.9.jar:5.6.9]
  870.     at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:674) [elasticsearch-5.6.9.jar:5.6.9]
  871.     at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.9.jar:5.6.9]
  872.     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
  873.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
  874.     at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  875. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: 10.231.83.146/10.231.83.146:9301
  876.     at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  877.     at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
  878.     at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352) ~[?:?]
  879.     at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  880.     at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?]
  881.     at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?]
  882.     at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?]
  883.     at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?]
  884.     at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
  885.     ... 1 more
  886. Caused by: java.net.ConnectException: Connection refused
  887.     at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  888.     at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
  889.     at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352) ~[?:?]
  890.     at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  891.     at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?]
  892.     at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?]
  893.     at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?]
  894.     at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?]
  895.     at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
  896.     ... 1 more
  897. [2019-01-25T10:45:24,562][ERROR][o.e.d.z.ZenDiscovery     ] [yRj9xlW] unexpected failure during [zen-disco-node-failed({wAVZzky}{wAVZzkyrSfyBmuGRw0Bltw}{Qc8kODytQreZFj_24t5O7g}{10.231.83.146}{10.231.83.146:9302}), reason(failed to ping, tried [3] times, each with maximum [4s] timeout)]
  898. org.elasticsearch.discovery.Discovery$FailedToCommitClusterStateException: failed to get enough masters to ack sent cluster state. [1] left
  899.     at org.elasticsearch.discovery.zen.PublishClusterStateAction$SendingController.waitForCommit(PublishClusterStateAction.java:574) ~[elasticsearch-5.6.9.jar:5.6.9]
  900.     at org.elasticsearch.discovery.zen.PublishClusterStateAction.innerPublish(PublishClusterStateAction.java:202) ~[elasticsearch-5.6.9.jar:5.6.9]
  901.     at org.elasticsearch.discovery.zen.PublishClusterStateAction.publish(PublishClusterStateAction.java:167) ~[elasticsearch-5.6.9.jar:5.6.9]
  902.     at org.elasticsearch.discovery.zen.ZenDiscovery.publish(ZenDiscovery.java:310) ~[elasticsearch-5.6.9.jar:5.6.9]
  903.     at org.elasticsearch.cluster.service.ClusterService.publishAndApplyChanges(ClusterService.java:741) [elasticsearch-5.6.9.jar:5.6.9]
  904.     at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:587) [elasticsearch-5.6.9.jar:5.6.9]
  905.     at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) [elasticsearch-5.6.9.jar:5.6.9]
  906.     at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-5.6.9.jar:5.6.9]
  907.     at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-5.6.9.jar:5.6.9]
  908.     at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.9.jar:5.6.9]
  909.     at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) [elasticsearch-5.6.9.jar:5.6.9]
  910.     at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) [elasticsearch-5.6.9.jar:5.6.9]
  911.     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
  912.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
  913.     at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
  914. [2019-01-25T10:45:24,586][INFO ][o.e.n.Node               ] [yRj9xlW] stopped
  915. [2019-01-25T10:45:24,586][INFO ][o.e.n.Node               ] [yRj9xlW] closing ...
  916. [2019-01-25T10:45:24,592][INFO ][o.e.n.Node               ] [yRj9xlW] closed
RAW Paste Data
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy. OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!
 
Top