SHARE
TWEET

Untitled

a guest Jan 25th, 2019 71 Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
  1. [2019-01-25T10:43:14,316][INFO ][o.e.n.Node               ] [] initializing ...
  2. [2019-01-25T10:43:14,424][INFO ][o.e.e.NodeEnvironment    ] [lJgJwYh] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [1.5tb], net total_space [1.7tb], spins? [possibly], types [ext4]
  3. [2019-01-25T10:43:14,425][INFO ][o.e.e.NodeEnvironment    ] [lJgJwYh] heap size [1.9gb], compressed ordinary object pointers [true]
  4. [2019-01-25T10:43:14,442][INFO ][o.e.n.Node               ] node name [lJgJwYh] derived from node ID [lJgJwYhmQA-fucaOBHD5Vg]; set [node.name] to override
  5. [2019-01-25T10:43:14,442][INFO ][o.e.n.Node               ] version[5.6.9], pid[8804], build[877a590/2018-04-12T16:25:14.838Z], OS[Linux/4.9.0-8-amd64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_131/25.131-b11]
  6. [2019-01-25T10:43:14,442][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/home/christian/elasticsearch-5.6.9-2]
  7. [2019-01-25T10:43:14,912][INFO ][o.e.p.PluginsService     ] [lJgJwYh] loaded module [aggs-matrix-stats]
  8. [2019-01-25T10:43:14,913][INFO ][o.e.p.PluginsService     ] [lJgJwYh] loaded module [ingest-common]
  9. [2019-01-25T10:43:14,913][INFO ][o.e.p.PluginsService     ] [lJgJwYh] loaded module [lang-expression]
  10. [2019-01-25T10:43:14,913][INFO ][o.e.p.PluginsService     ] [lJgJwYh] loaded module [lang-groovy]
  11. [2019-01-25T10:43:14,913][INFO ][o.e.p.PluginsService     ] [lJgJwYh] loaded module [lang-mustache]
  12. [2019-01-25T10:43:14,913][INFO ][o.e.p.PluginsService     ] [lJgJwYh] loaded module [lang-painless]
  13. [2019-01-25T10:43:14,913][INFO ][o.e.p.PluginsService     ] [lJgJwYh] loaded module [parent-join]
  14. [2019-01-25T10:43:14,913][INFO ][o.e.p.PluginsService     ] [lJgJwYh] loaded module [percolator]
  15. [2019-01-25T10:43:14,913][INFO ][o.e.p.PluginsService     ] [lJgJwYh] loaded module [reindex]
  16. [2019-01-25T10:43:14,913][INFO ][o.e.p.PluginsService     ] [lJgJwYh] loaded module [transport-netty3]
  17. [2019-01-25T10:43:14,913][INFO ][o.e.p.PluginsService     ] [lJgJwYh] loaded module [transport-netty4]
  18. [2019-01-25T10:43:14,913][INFO ][o.e.p.PluginsService     ] [lJgJwYh] no plugins loaded
  19. [2019-01-25T10:43:15,793][INFO ][o.e.d.DiscoveryModule    ] [lJgJwYh] using discovery type [zen]
  20. [2019-01-25T10:43:16,265][INFO ][o.e.n.Node               ] initialized
  21. [2019-01-25T10:43:16,266][INFO ][o.e.n.Node               ] [lJgJwYh] starting ...
  22. [2019-01-25T10:43:16,363][INFO ][o.e.t.TransportService   ] [lJgJwYh] publish_address {10.231.83.146:9301}, bound_addresses {10.231.83.146:9301}
  23. [2019-01-25T10:43:16,372][INFO ][o.e.b.BootstrapChecks    ] [lJgJwYh] bound or publishing to a non-loopback address, enforcing bootstrap checks
  24. [2019-01-25T10:43:19,403][WARN ][o.e.d.z.ZenDiscovery     ] [lJgJwYh] failed to connect to master [{wAVZzky}{wAVZzkyrSfyBmuGRw0Bltw}{r1L7UXpESgiM_Kxq_jFUjw}{10.231.83.146}{10.231.83.146:9302}], retrying...
  25. org.elasticsearch.transport.ConnectTransportException: [wAVZzky][10.231.83.146:9302] connect_timeout[30s]
  26.     at org.elasticsearch.transport.netty4.Netty4Transport.connectToChannels(Netty4Transport.java:363) ~[?:?]
  27.     at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:570) ~[elasticsearch-5.6.9.jar:5.6.9]
  28.     at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:473) ~[elasticsearch-5.6.9.jar:5.6.9]
  29.     at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:342) ~[elasticsearch-5.6.9.jar:5.6.9]
  30.     at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:329) ~[elasticsearch-5.6.9.jar:5.6.9]
  31.     at org.elasticsearch.discovery.zen.ZenDiscovery.joinElectedMaster(ZenDiscovery.java:458) [elasticsearch-5.6.9.jar:5.6.9]
  32.     at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:410) [elasticsearch-5.6.9.jar:5.6.9]
  33.     at org.elasticsearch.discovery.zen.ZenDiscovery.access$4100(ZenDiscovery.java:82) [elasticsearch-5.6.9.jar:5.6.9]
  34.     at org.elasticsearch.discovery.zen.ZenDiscovery$JoinThreadControl$1.run(ZenDiscovery.java:1188) [elasticsearch-5.6.9.jar:5.6.9]
  35.     at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.9.jar:5.6.9]
  36.     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
  37.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
  38.     at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
  39. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: 10.231.83.146/10.231.83.146:9302
  40.     at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  41.     at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
  42.     at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352) ~[?:?]
  43.     at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  44.     at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?]
  45.     at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?]
  46.     at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?]
  47.     at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?]
  48.     at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
  49.     ... 1 more
  50. Caused by: java.net.ConnectException: Connection refused
  51.     at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  52.     at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
  53.     at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352) ~[?:?]
  54.     at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  55.     at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?]
  56.     at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?]
  57.     at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?]
  58.     at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?]
  59.     at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
  60.     ... 1 more
  61. [2019-01-25T10:43:22,583][INFO ][o.e.c.s.ClusterService   ] [lJgJwYh] detected_master {yRj9xlW}{yRj9xlW8Q-WgQ0j6H4DbOQ}{SAn1BH7ES42OhsAknxVe0Q}{10.231.83.238}{10.231.83.238:9300}, added {{yRj9xlW}{yRj9xlW8Q-WgQ0j6H4DbOQ}{SAn1BH7ES42OhsAknxVe0Q}{10.231.83.238}{10.231.83.238:9300},{wAVZzky}{wAVZzkyrSfyBmuGRw0Bltw}{r1L7UXpESgiM_Kxq_jFUjw}{10.231.83.146}{10.231.83.146:9302},}, reason: zen-disco-receive(from master [master {yRj9xlW}{yRj9xlW8Q-WgQ0j6H4DbOQ}{SAn1BH7ES42OhsAknxVe0Q}{10.231.83.238}{10.231.83.238:9300} committed version [183]])
  62. [2019-01-25T10:43:22,604][WARN ][o.e.c.NodeConnectionsService] [lJgJwYh] failed to connect to node {wAVZzky}{wAVZzkyrSfyBmuGRw0Bltw}{r1L7UXpESgiM_Kxq_jFUjw}{10.231.83.146}{10.231.83.146:9302} (tried [1] times)
  63. org.elasticsearch.transport.ConnectTransportException: [wAVZzky][10.231.83.146:9302] handshake failed. unexpected remote node {wAVZzky}{wAVZzkyrSfyBmuGRw0Bltw}{Qc8kODytQreZFj_24t5O7g}{10.231.83.146}{10.231.83.146:9302}
  64.     at org.elasticsearch.transport.TransportService.lambda$connectToNode$3(TransportService.java:346) ~[elasticsearch-5.6.9.jar:5.6.9]
  65.     at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:474) ~[elasticsearch-5.6.9.jar:5.6.9]
  66.     at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:342) ~[elasticsearch-5.6.9.jar:5.6.9]
  67.     at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:329) ~[elasticsearch-5.6.9.jar:5.6.9]
  68.     at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:154) [elasticsearch-5.6.9.jar:5.6.9]
  69.     at org.elasticsearch.cluster.NodeConnectionsService$1.doRun(NodeConnectionsService.java:107) [elasticsearch-5.6.9.jar:5.6.9]
  70.     at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:674) [elasticsearch-5.6.9.jar:5.6.9]
  71.     at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.9.jar:5.6.9]
  72.     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
  73.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
  74.     at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
  75. [2019-01-25T10:43:23,023][INFO ][o.e.h.n.Netty4HttpServerTransport] [lJgJwYh] publish_address {10.231.83.146:9201}, bound_addresses {10.231.83.146:9201}
  76. [2019-01-25T10:43:23,023][INFO ][o.e.n.Node               ] [lJgJwYh] started
  77. [2019-01-25T10:43:23,057][INFO ][o.e.c.s.ClusterService   ] [lJgJwYh] removed {{wAVZzky}{wAVZzkyrSfyBmuGRw0Bltw}{r1L7UXpESgiM_Kxq_jFUjw}{10.231.83.146}{10.231.83.146:9302},}, reason: zen-disco-receive(from master [master {yRj9xlW}{yRj9xlW8Q-WgQ0j6H4DbOQ}{SAn1BH7ES42OhsAknxVe0Q}{10.231.83.238}{10.231.83.238:9300} committed version [184]])
  78. [2019-01-25T10:43:24,494][INFO ][o.e.c.s.ClusterService   ] [lJgJwYh] added {{wAVZzky}{wAVZzkyrSfyBmuGRw0Bltw}{Qc8kODytQreZFj_24t5O7g}{10.231.83.146}{10.231.83.146:9302},}, reason: zen-disco-receive(from master [master {yRj9xlW}{yRj9xlW8Q-WgQ0j6H4DbOQ}{SAn1BH7ES42OhsAknxVe0Q}{10.231.83.238}{10.231.83.238:9300} committed version [188]])
  79. [2019-01-25T10:44:30,153][INFO ][o.e.d.z.ZenDiscovery     ] [lJgJwYh] master_left [{yRj9xlW}{yRj9xlW8Q-WgQ0j6H4DbOQ}{SAn1BH7ES42OhsAknxVe0Q}{10.231.83.238}{10.231.83.238:9300}], reason [failed to ping, tried [3] times, each with  maximum [4s] timeout]
  80. [2019-01-25T10:44:30,156][WARN ][o.e.d.z.ZenDiscovery     ] [lJgJwYh] master left (reason = failed to ping, tried [3] times, each with  maximum [4s] timeout), current nodes: nodes:
  81.    {yRj9xlW}{yRj9xlW8Q-WgQ0j6H4DbOQ}{SAn1BH7ES42OhsAknxVe0Q}{10.231.83.238}{10.231.83.238:9300}, master
  82.    {lJgJwYh}{lJgJwYhmQA-fucaOBHD5Vg}{lbN7QcyiSWCOBYNL4jNkdg}{10.231.83.146}{10.231.83.146:9301}, local
  83.    {wAVZzky}{wAVZzkyrSfyBmuGRw0Bltw}{Qc8kODytQreZFj_24t5O7g}{10.231.83.146}{10.231.83.146:9302}
  84.  
  85. [2019-01-25T10:44:33,319][INFO ][o.e.c.s.ClusterService   ] [lJgJwYh] new_master {lJgJwYh}{lJgJwYhmQA-fucaOBHD5Vg}{lbN7QcyiSWCOBYNL4jNkdg}{10.231.83.146}{10.231.83.146:9301}, reason: zen-disco-elected-as-master ([1] nodes joined)[, ]
  86. [2019-01-25T10:44:33,327][DEBUG][o.e.a.a.c.h.TransportClusterHealthAction] [lJgJwYh] no known master node, scheduling a retry
  87. [2019-01-25T10:44:33,327][DEBUG][o.e.a.a.c.h.TransportClusterHealthAction] [lJgJwYh] no known master node, scheduling a retry
  88. [2019-01-25T10:44:33,910][WARN ][o.e.d.z.UnicastZenPing   ] [lJgJwYh] failed to send ping to [{yRj9xlW}{yRj9xlW8Q-WgQ0j6H4DbOQ}{SAn1BH7ES42OhsAknxVe0Q}{10.231.83.238}{10.231.83.238:9300}]
  89. org.elasticsearch.transport.ReceiveTimeoutTransportException: [yRj9xlW][10.231.83.238:9300][internal:discovery/zen/unicast] request_id [111] timed out after [3750ms]
  90.     at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:961) [elasticsearch-5.6.9.jar:5.6.9]
  91.     at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.9.jar:5.6.9]
  92.     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
  93.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
  94.     at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
  95. [2019-01-25T10:44:34,717][DEBUG][o.e.a.a.c.h.TransportClusterHealthAction] [lJgJwYh] no known master node, scheduling a retry
  96. [2019-01-25T10:44:34,910][WARN ][o.e.d.z.UnicastZenPing   ] [lJgJwYh] failed to send ping to [{yRj9xlW}{yRj9xlW8Q-WgQ0j6H4DbOQ}{SAn1BH7ES42OhsAknxVe0Q}{10.231.83.238}{10.231.83.238:9300}]
  97. org.elasticsearch.transport.ReceiveTimeoutTransportException: [yRj9xlW][10.231.83.238:9300][internal:discovery/zen/unicast] request_id [114] timed out after [3751ms]
  98.     at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:961) [elasticsearch-5.6.9.jar:5.6.9]
  99.     at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.9.jar:5.6.9]
  100.     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
  101.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
  102.     at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
  103. [2019-01-25T10:44:35,910][WARN ][o.e.d.z.UnicastZenPing   ] [lJgJwYh] failed to send ping to [{yRj9xlW}{yRj9xlW8Q-WgQ0j6H4DbOQ}{SAn1BH7ES42OhsAknxVe0Q}{10.231.83.238}{10.231.83.238:9300}]
  104. org.elasticsearch.transport.ReceiveTimeoutTransportException: [yRj9xlW][10.231.83.238:9300][internal:discovery/zen/unicast] request_id [118] timed out after [3750ms]
  105.     at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:961) [elasticsearch-5.6.9.jar:5.6.9]
  106.     at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.9.jar:5.6.9]
  107.     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
  108.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
  109.     at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
  110. [2019-01-25T10:44:36,727][DEBUG][o.e.a.a.c.h.TransportClusterHealthAction] [lJgJwYh] no known master node, scheduling a retry
  111. [2019-01-25T10:44:38,738][DEBUG][o.e.a.a.c.h.TransportClusterHealthAction] [lJgJwYh] no known master node, scheduling a retry
  112. [2019-01-25T10:44:40,748][DEBUG][o.e.a.a.c.h.TransportClusterHealthAction] [lJgJwYh] no known master node, scheduling a retry
  113. [2019-01-25T10:44:42,758][DEBUG][o.e.a.a.c.h.TransportClusterHealthAction] [lJgJwYh] no known master node, scheduling a retry
  114. [2019-01-25T10:44:44,767][DEBUG][o.e.a.a.c.h.TransportClusterHealthAction] [lJgJwYh] no known master node, scheduling a retry
  115. [2019-01-25T10:45:03,322][WARN ][o.e.d.z.PublishClusterStateAction] [lJgJwYh] timed out waiting for all nodes to process published state [200] (timeout [30s], pending nodes: [{yRj9xlW}{yRj9xlW8Q-WgQ0j6H4DbOQ}{SAn1BH7ES42OhsAknxVe0Q}{10.231.83.238}{10.231.83.238:9300}])
  116. [2019-01-25T10:45:03,327][WARN ][o.e.c.s.ClusterService   ] [lJgJwYh] cluster state update task [zen-disco-elected-as-master ([1] nodes joined)[, ]] took [30s] above the warn threshold of 30s
  117. ^C[2019-01-25T10:45:14,479][INFO ][o.e.n.Node               ] [lJgJwYh] stopping ...
  118. [2019-01-25T10:45:16,277][WARN ][o.e.c.NodeConnectionsService] [lJgJwYh] failed to connect to node {wAVZzky}{wAVZzkyrSfyBmuGRw0Bltw}{Qc8kODytQreZFj_24t5O7g}{10.231.83.146}{10.231.83.146:9302} (tried [1] times)
  119. org.elasticsearch.transport.ConnectTransportException: [wAVZzky][10.231.83.146:9302] connect_timeout[30s]
  120.     at org.elasticsearch.transport.netty4.Netty4Transport.connectToChannels(Netty4Transport.java:363) ~[?:?]
  121.     at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:570) ~[elasticsearch-5.6.9.jar:5.6.9]
  122.     at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:473) ~[elasticsearch-5.6.9.jar:5.6.9]
  123.     at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:342) ~[elasticsearch-5.6.9.jar:5.6.9]
  124.     at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:329) ~[elasticsearch-5.6.9.jar:5.6.9]
  125.     at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:154) [elasticsearch-5.6.9.jar:5.6.9]
  126.     at org.elasticsearch.cluster.NodeConnectionsService$ConnectionChecker.doRun(NodeConnectionsService.java:183) [elasticsearch-5.6.9.jar:5.6.9]
  127.     at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:674) [elasticsearch-5.6.9.jar:5.6.9]
  128.     at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.9.jar:5.6.9]
  129.     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
  130.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
  131.     at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
  132. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: 10.231.83.146/10.231.83.146:9302
  133.     at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  134.     at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
  135.     at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352) ~[?:?]
  136.     at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  137.     at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?]
  138.     at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?]
  139.     at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?]
  140.     at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?]
  141.     at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
  142.     ... 1 more
  143. Caused by: java.net.ConnectException: Connection refused
  144.     at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  145.     at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
  146.     at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352) ~[?:?]
  147.     at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  148.     at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?]
  149.     at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?]
  150.     at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?]
  151.     at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?]
  152.     at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
  153.     ... 1 more
  154. [2019-01-25T10:45:18,333][DEBUG][o.e.a.a.c.n.s.TransportNodesStatsAction] [lJgJwYh] failed to execute on node [yRj9xlW8Q-WgQ0j6H4DbOQ]
  155. org.elasticsearch.transport.ReceiveTimeoutTransportException: [yRj9xlW][10.231.83.238:9300][cluster:monitor/nodes/stats[n]] request_id [128] timed out after [15000ms]
  156.     at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:961) [elasticsearch-5.6.9.jar:5.6.9]
  157.     at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.9.jar:5.6.9]
  158.     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
  159.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
  160.     at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
  161. [2019-01-25T10:45:24,502][WARN ][o.e.e.NodeEnvironment    ] [lJgJwYh] lock assertion failed
  162. java.nio.channels.ClosedByInterruptException: null
  163.     at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202) ~[?:1.8.0_131]
  164.     at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:315) ~[?:?]
  165.     at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:170) ~[lucene-core-6.6.1.jar:6.6.1 9aa465a89b64ff2dabe7b4d50c472de32c298683 - varunthacker - 2017-08-29 21:54:39]
  166.     at org.elasticsearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:941) ~[elasticsearch-5.6.9.jar:5.6.9]
  167.     at org.elasticsearch.env.NodeEnvironment.availableIndexFolders(NodeEnvironment.java:820) ~[elasticsearch-5.6.9.jar:5.6.9]
  168.     at org.elasticsearch.gateway.MetaStateService.loadIndicesStates(MetaStateService.java:89) ~[elasticsearch-5.6.9.jar:5.6.9]
  169.     at org.elasticsearch.gateway.DanglingIndicesState.findNewDanglingIndices(DanglingIndicesState.java:131) ~[elasticsearch-5.6.9.jar:5.6.9]
  170.     at org.elasticsearch.gateway.DanglingIndicesState.findNewAndAddDanglingIndices(DanglingIndicesState.java:116) ~[elasticsearch-5.6.9.jar:5.6.9]
  171.     at org.elasticsearch.gateway.DanglingIndicesState.processDanglingIndices(DanglingIndicesState.java:81) ~[elasticsearch-5.6.9.jar:5.6.9]
  172.     at org.elasticsearch.gateway.DanglingIndicesState.clusterChanged(DanglingIndicesState.java:185) ~[elasticsearch-5.6.9.jar:5.6.9]
  173.     at org.elasticsearch.cluster.service.ClusterService.lambda$publishAndApplyChanges$7(ClusterService.java:777) ~[elasticsearch-5.6.9.jar:5.6.9]
  174.     at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) [?:1.8.0_131]
  175.     at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742) [?:1.8.0_131]
  176.     at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580) [?:1.8.0_131]
  177.     at org.elasticsearch.cluster.service.ClusterService.publishAndApplyChanges(ClusterService.java:774) [elasticsearch-5.6.9.jar:5.6.9]
  178.     at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:587) [elasticsearch-5.6.9.jar:5.6.9]
  179.     at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) [elasticsearch-5.6.9.jar:5.6.9]
  180.     at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-5.6.9.jar:5.6.9]
  181.     at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-5.6.9.jar:5.6.9]
  182.     at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.9.jar:5.6.9]
  183.     at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) [elasticsearch-5.6.9.jar:5.6.9]
  184.     at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) [elasticsearch-5.6.9.jar:5.6.9]
  185.     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
  186.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
  187.     at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
  188. [2019-01-25T10:45:24,505][WARN ][o.e.c.s.ClusterService   ] [lJgJwYh] failed to notify ClusterStateListener
  189. java.lang.IllegalStateException: environment is not locked
  190.     at org.elasticsearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:944) ~[elasticsearch-5.6.9.jar:5.6.9]
  191.     at org.elasticsearch.env.NodeEnvironment.availableIndexFolders(NodeEnvironment.java:820) ~[elasticsearch-5.6.9.jar:5.6.9]
  192.     at org.elasticsearch.gateway.MetaStateService.loadIndicesStates(MetaStateService.java:89) ~[elasticsearch-5.6.9.jar:5.6.9]
  193.     at org.elasticsearch.gateway.DanglingIndicesState.findNewDanglingIndices(DanglingIndicesState.java:131) ~[elasticsearch-5.6.9.jar:5.6.9]
  194.     at org.elasticsearch.gateway.DanglingIndicesState.findNewAndAddDanglingIndices(DanglingIndicesState.java:116) ~[elasticsearch-5.6.9.jar:5.6.9]
  195.     at org.elasticsearch.gateway.DanglingIndicesState.processDanglingIndices(DanglingIndicesState.java:81) ~[elasticsearch-5.6.9.jar:5.6.9]
  196.     at org.elasticsearch.gateway.DanglingIndicesState.clusterChanged(DanglingIndicesState.java:185) ~[elasticsearch-5.6.9.jar:5.6.9]
  197.     at org.elasticsearch.cluster.service.ClusterService.lambda$publishAndApplyChanges$7(ClusterService.java:777) ~[elasticsearch-5.6.9.jar:5.6.9]
  198.     at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) [?:1.8.0_131]
  199.     at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742) [?:1.8.0_131]
  200.     at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580) [?:1.8.0_131]
  201.     at org.elasticsearch.cluster.service.ClusterService.publishAndApplyChanges(ClusterService.java:774) [elasticsearch-5.6.9.jar:5.6.9]
  202.     at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:587) [elasticsearch-5.6.9.jar:5.6.9]
  203.     at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) [elasticsearch-5.6.9.jar:5.6.9]
  204.     at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-5.6.9.jar:5.6.9]
  205.     at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-5.6.9.jar:5.6.9]
  206.     at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575) [elasticsearch-5.6.9.jar:5.6.9]
  207.     at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) [elasticsearch-5.6.9.jar:5.6.9]
  208.     at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) [elasticsearch-5.6.9.jar:5.6.9]
  209.     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
  210.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
  211.     at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
  212. Caused by: java.nio.channels.ClosedByInterruptException
  213.     at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202) ~[?:1.8.0_131]
  214.     at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:315) ~[?:?]
  215.     at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:170) ~[lucene-core-6.6.1.jar:6.6.1 9aa465a89b64ff2dabe7b4d50c472de32c298683 - varunthacker - 2017-08-29 21:54:39]
  216.     at org.elasticsearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:941) ~[elasticsearch-5.6.9.jar:5.6.9]
  217.     ... 21 more
  218. [2019-01-25T10:45:24,509][DEBUG][o.e.a.a.i.s.TransportIndicesStatsAction] [lJgJwYh] failed to execute [indices:monitor/stats] on node [yRj9xlW8Q-WgQ0j6H4DbOQ]
  219. org.elasticsearch.transport.NodeDisconnectedException: [yRj9xlW][10.231.83.238:9300][indices:monitor/stats[n]] disconnected
  220. [2019-01-25T10:45:24,518][WARN ][o.e.t.n.Netty4Transport  ] [lJgJwYh] write and flush on the network layer failed (channel: [id: 0xa0aaf523, L:0.0.0.0/0.0.0.0:9301 ! R:/10.231.83.146:44348])
  221. java.nio.channels.ClosedChannelException: null
  222.     at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
  223. [2019-01-25T10:45:24,528][INFO ][o.e.n.Node               ] [lJgJwYh] stopped
  224. [2019-01-25T10:45:24,528][INFO ][o.e.n.Node               ] [lJgJwYh] closing ...
  225. [2019-01-25T10:45:24,534][INFO ][o.e.n.Node               ] [lJgJwYh] closed
RAW Paste Data
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy. OK, I Understand
 
Top