API tools faq
paste
Advertisement
Guest User

Untitled

a guest
Apr 19th, 2019
201
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 47.15 KB | None | 0 0
raw download clone embed print report
  1. [2019-04-19T20:23:18,609][INFO ][o.e.c.s.ClusterApplierService] [dev-efk-backend05] master node changed {previous [{dev-efk-backend01}{9-ePQkHuS_6IAyGpYSCyDg}{Ba7wi43ZS6irFH-87BMpqQ}{192.168.0.202}{192.168.0.202:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true}], current []}, term: 9, version: 57, reason: becoming candidate: onLeaderFailure
  2. [2019-04-19T20:23:18,658][WARN ][o.e.c.NodeConnectionsService] [dev-efk-backend05] failed to connect to node {dev-efk-backend01}{9-ePQkHuS_6IAyGpYSCyDg}{Ba7wi43ZS6irFH-87BMpqQ}{192.168.0.202}{192.168.0.202:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true} (tried [1] times)
  3. org.elasticsearch.transport.ConnectTransportException: [dev-efk-backend01][192.168.0.202:9300] connect_exception
  4. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.0.0.jar:7.0.0]
  5. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.0.0.jar:7.0.0]
  6. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.0.0.jar:7.0.0]
  7. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  8. at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:883) ~[?:?]
  9. at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2322) ~[?:?]
  10. at org.elasticsearch.common.concurrent.CompletableContext.addListener(CompletableContext.java:45) ~[elasticsearch-core-7.0.0.jar:7.0.0]
  11. at org.elasticsearch.transport.netty4.Netty4TcpChannel.addConnectListener(Netty4TcpChannel.java:100) ~[?:?]
  12. at org.elasticsearch.transport.TcpTransport.initiateConnection(TcpTransport.java:325) ~[elasticsearch-7.0.0.jar:7.0.0]
  13. at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:292) ~[elasticsearch-7.0.0.jar:7.0.0]
  14. at org.elasticsearch.transport.ConnectionManager.internalOpenConnection(ConnectionManager.java:206) ~[elasticsearch-7.0.0.jar:7.0.0]
  15. at org.elasticsearch.transport.ConnectionManager.connectToNode(ConnectionManager.java:104) ~[elasticsearch-7.0.0.jar:7.0.0]
  16. at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:343) ~[elasticsearch-7.0.0.jar:7.0.0]
  17. at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:330) ~[elasticsearch-7.0.0.jar:7.0.0]
  18. at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:153) [elasticsearch-7.0.0.jar:7.0.0]
  19. at org.elasticsearch.cluster.NodeConnectionsService$1.doRun(NodeConnectionsService.java:106) [elasticsearch-7.0.0.jar:7.0.0]
  20. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [elasticsearch-7.0.0.jar:7.0.0]
  21. at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.0.jar:7.0.0]
  22. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
  23. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
  24. at java.lang.Thread.run(Thread.java:835) [?:?]
  25. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: 192.168.0.202/192.168.0.202:9300
  26. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  27. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  28. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  29. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  30. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  31. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  32. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  33. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  34. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  35. ... 1 more
  36. Caused by: java.net.ConnectException: Connection refused
  37. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  38. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  39. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  40. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  41. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  42. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  43. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  44. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  45. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  46. ... 1 more
  47. [2019-04-19T20:23:18,705][INFO ][o.e.c.c.JoinHelper ] [dev-efk-backend05] failed to join {dev-efk-backend05}{gKWS3gxIRs-P5FFlvMTm4A}{yAp4wKi8SJ6BPzEQo4rGcQ}{192.168.0.239}{192.168.0.239:9300}{ml.machine_memory=16171651072, xpack.installed=true, ml.max_open_jobs=20} with JoinRequest{sourceNode={dev-efk-backend05}{gKWS3gxIRs-P5FFlvMTm4A}{yAp4wKi8SJ6BPzEQo4rGcQ}{192.168.0.239}{192.168.0.239:9300}{ml.machine_memory=16171651072, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=10, lastAcceptedTerm=9, lastAcceptedVersion=57, sourceNode={dev-efk-backend05}{gKWS3gxIRs-P5FFlvMTm4A}{yAp4wKi8SJ6BPzEQo4rGcQ}{192.168.0.239}{192.168.0.239:9300}{ml.machine_memory=16171651072, xpack.installed=true, ml.max_open_jobs=20}, targetNode={dev-efk-backend05}{gKWS3gxIRs-P5FFlvMTm4A}{yAp4wKi8SJ6BPzEQo4rGcQ}{192.168.0.239}{192.168.0.239:9300}{ml.machine_memory=16171651072, xpack.installed=true, ml.max_open_jobs=20}}]}
  48. org.elasticsearch.transport.RemoteTransportException: [dev-efk-backend05][192.168.0.239:9300][internal:cluster/coordination/join]
  49. Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 10 does not match current term 11
  50. at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:218) ~[elasticsearch-7.0.0.jar:7.0.0]
  51. at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:940) ~[elasticsearch-7.0.0.jar:7.0.0]
  52. at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
  53. at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:512) ~[elasticsearch-7.0.0.jar:7.0.0]
  54. at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:478) ~[elasticsearch-7.0.0.jar:7.0.0]
  55. at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:124) ~[elasticsearch-7.0.0.jar:7.0.0]
  56. at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:251) [x-pack-security-7.0.0.jar:7.0.0]
  57. at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.0.jar:7.0.0]
  58. at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:309) [x-pack-security-7.0.0.jar:7.0.0]
  59. at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [elasticsearch-7.0.0.jar:7.0.0]
  60. at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:687) [elasticsearch-7.0.0.jar:7.0.0]
  61. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [elasticsearch-7.0.0.jar:7.0.0]
  62. at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.0.jar:7.0.0]
  63. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
  64. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
  65. at java.lang.Thread.run(Thread.java:835) [?:?]
  66. [2019-04-19T20:23:18,775][INFO ][o.e.c.c.JoinHelper ] [dev-efk-backend05] failed to join {dev-efk-backend03}{xYDB_57RTQaUmCzxQVMm-A}{CAfs0z4VT0G78Ttk0unpOg}{192.168.0.142}{192.168.0.142:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={dev-efk-backend05}{gKWS3gxIRs-P5FFlvMTm4A}{yAp4wKi8SJ6BPzEQo4rGcQ}{192.168.0.239}{192.168.0.239:9300}{ml.machine_memory=16171651072, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=12, lastAcceptedTerm=9, lastAcceptedVersion=57, sourceNode={dev-efk-backend05}{gKWS3gxIRs-P5FFlvMTm4A}{yAp4wKi8SJ6BPzEQo4rGcQ}{192.168.0.239}{192.168.0.239:9300}{ml.machine_memory=16171651072, xpack.installed=true, ml.max_open_jobs=20}, targetNode={dev-efk-backend03}{xYDB_57RTQaUmCzxQVMm-A}{CAfs0z4VT0G78Ttk0unpOg}{192.168.0.142}{192.168.0.142:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true}}]}
  67. org.elasticsearch.transport.RemoteTransportException: [dev-efk-backend03][192.168.0.142:9300][internal:cluster/coordination/join]
  68. Caused by: org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: node is no longer master for term 12 while handling publication
  69. at org.elasticsearch.cluster.coordination.Coordinator.publish(Coordinator.java:1010) ~[elasticsearch-7.0.0.jar:7.0.0]
  70. at org.elasticsearch.cluster.service.MasterService.publish(MasterService.java:252) ~[elasticsearch-7.0.0.jar:7.0.0]
  71. at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:238) ~[elasticsearch-7.0.0.jar:7.0.0]
  72. at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:142) ~[elasticsearch-7.0.0.jar:7.0.0]
  73. at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) ~[elasticsearch-7.0.0.jar:7.0.0]
  74. at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) ~[elasticsearch-7.0.0.jar:7.0.0]
  75. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) ~[elasticsearch-7.0.0.jar:7.0.0]
  76. at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) ~[elasticsearch-7.0.0.jar:7.0.0]
  77. at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) ~[elasticsearch-7.0.0.jar:7.0.0]
  78. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
  79. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
  80. at java.lang.Thread.run(Thread.java:835) [?:?]
  81. [2019-04-19T20:23:18,875][INFO ][o.e.c.c.JoinHelper ] [dev-efk-backend05] failed to join {dev-efk-backend04}{WPeuO5I1SDWZkVTdpVGqlQ}{_Da2zkZCQGa62XDKg5ej4w}{192.168.0.21}{192.168.0.21:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={dev-efk-backend05}{gKWS3gxIRs-P5FFlvMTm4A}{yAp4wKi8SJ6BPzEQo4rGcQ}{192.168.0.239}{192.168.0.239:9300}{ml.machine_memory=16171651072, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=11, lastAcceptedTerm=9, lastAcceptedVersion=57, sourceNode={dev-efk-backend05}{gKWS3gxIRs-P5FFlvMTm4A}{yAp4wKi8SJ6BPzEQo4rGcQ}{192.168.0.239}{192.168.0.239:9300}{ml.machine_memory=16171651072, xpack.installed=true, ml.max_open_jobs=20}, targetNode={dev-efk-backend04}{WPeuO5I1SDWZkVTdpVGqlQ}{_Da2zkZCQGa62XDKg5ej4w}{192.168.0.21}{192.168.0.21:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true}}]}
  82. org.elasticsearch.transport.RemoteTransportException: [dev-efk-backend04][192.168.0.21:9300][internal:cluster/coordination/join]
  83. Caused by: org.elasticsearch.cluster.NotMasterException: Node [{dev-efk-backend04}{WPeuO5I1SDWZkVTdpVGqlQ}{_Da2zkZCQGa62XDKg5ej4w}{192.168.0.21}{192.168.0.21:9300}{ml.machine_memory=16171651072, xpack.installed=true, ml.max_open_jobs=20}] not master for join request
  84. [2019-04-19T20:23:18,890][INFO ][o.e.c.s.ClusterApplierService] [dev-efk-backend05] master node changed {previous [], current [{dev-efk-backend02}{iO6O8mpUQaGtCk8NWO5X9w}{AvVXSLOrRuGKTZeJe18gYQ}{192.168.0.104}{192.168.0.104:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true}]}, term: 13, version: 58, reason: ApplyCommitRequest{term=13, version=58, sourceNode={dev-efk-backend02}{iO6O8mpUQaGtCk8NWO5X9w}{AvVXSLOrRuGKTZeJe18gYQ}{192.168.0.104}{192.168.0.104:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true}}
  85. [2019-04-19T20:23:18,979][INFO ][o.e.c.s.ClusterApplierService] [dev-efk-backend05] removed {{dev-efk-backend01}{9-ePQkHuS_6IAyGpYSCyDg}{Ba7wi43ZS6irFH-87BMpqQ}{192.168.0.202}{192.168.0.202:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true},}, term: 13, version: 59, reason: ApplyCommitRequest{term=13, version=59, sourceNode={dev-efk-backend02}{iO6O8mpUQaGtCk8NWO5X9w}{AvVXSLOrRuGKTZeJe18gYQ}{192.168.0.104}{192.168.0.104:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true}}
  86. [2019-04-19T20:23:23,217][INFO ][o.e.c.s.ClusterApplierService] [dev-efk-backend05] master node changed {previous [{dev-efk-backend02}{iO6O8mpUQaGtCk8NWO5X9w}{AvVXSLOrRuGKTZeJe18gYQ}{192.168.0.104}{192.168.0.104:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true}], current []}, term: 13, version: 61, reason: becoming candidate: onLeaderFailure
  87. [2019-04-19T20:23:23,227][WARN ][o.e.c.NodeConnectionsService] [dev-efk-backend05] failed to connect to node {dev-efk-backend02}{iO6O8mpUQaGtCk8NWO5X9w}{AvVXSLOrRuGKTZeJe18gYQ}{192.168.0.104}{192.168.0.104:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true} (tried [1] times)
  88. org.elasticsearch.transport.ConnectTransportException: [dev-efk-backend02][192.168.0.104:9300] connect_exception
  89. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.0.0.jar:7.0.0]
  90. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.0.0.jar:7.0.0]
  91. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.0.0.jar:7.0.0]
  92. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  93. at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:883) ~[?:?]
  94. at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2322) ~[?:?]
  95. at org.elasticsearch.common.concurrent.CompletableContext.addListener(CompletableContext.java:45) ~[elasticsearch-core-7.0.0.jar:7.0.0]
  96. at org.elasticsearch.transport.netty4.Netty4TcpChannel.addConnectListener(Netty4TcpChannel.java:100) ~[?:?]
  97. at org.elasticsearch.transport.TcpTransport.initiateConnection(TcpTransport.java:325) ~[elasticsearch-7.0.0.jar:7.0.0]
  98. at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:292) ~[elasticsearch-7.0.0.jar:7.0.0]
  99. at org.elasticsearch.transport.ConnectionManager.internalOpenConnection(ConnectionManager.java:206) ~[elasticsearch-7.0.0.jar:7.0.0]
  100. at org.elasticsearch.transport.ConnectionManager.connectToNode(ConnectionManager.java:104) ~[elasticsearch-7.0.0.jar:7.0.0]
  101. at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:343) ~[elasticsearch-7.0.0.jar:7.0.0]
  102. at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:330) ~[elasticsearch-7.0.0.jar:7.0.0]
  103. at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:153) [elasticsearch-7.0.0.jar:7.0.0]
  104. at org.elasticsearch.cluster.NodeConnectionsService$1.doRun(NodeConnectionsService.java:106) [elasticsearch-7.0.0.jar:7.0.0]
  105. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [elasticsearch-7.0.0.jar:7.0.0]
  106. at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.0.jar:7.0.0]
  107. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
  108. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
  109. at java.lang.Thread.run(Thread.java:835) [?:?]
  110. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: 192.168.0.104/192.168.0.104:9300
  111. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  112. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  113. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  114. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  115. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  116. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  117. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  118. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  119. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  120. ... 1 more
  121. Caused by: java.net.ConnectException: Connection refused
  122. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  123. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  124. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  125. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  126. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  127. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  128. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  129. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  130. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  131. ... 1 more
  132. [2019-04-19T20:23:23,301][INFO ][o.e.c.c.JoinHelper ] [dev-efk-backend05] failed to join {dev-efk-backend04}{WPeuO5I1SDWZkVTdpVGqlQ}{_Da2zkZCQGa62XDKg5ej4w}{192.168.0.21}{192.168.0.21:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={dev-efk-backend05}{gKWS3gxIRs-P5FFlvMTm4A}{yAp4wKi8SJ6BPzEQo4rGcQ}{192.168.0.239}{192.168.0.239:9300}{ml.machine_memory=16171651072, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=14, lastAcceptedTerm=13, lastAcceptedVersion=61, sourceNode={dev-efk-backend05}{gKWS3gxIRs-P5FFlvMTm4A}{yAp4wKi8SJ6BPzEQo4rGcQ}{192.168.0.239}{192.168.0.239:9300}{ml.machine_memory=16171651072, xpack.installed=true, ml.max_open_jobs=20}, targetNode={dev-efk-backend04}{WPeuO5I1SDWZkVTdpVGqlQ}{_Da2zkZCQGa62XDKg5ej4w}{192.168.0.21}{192.168.0.21:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true}}]}
  133. org.elasticsearch.transport.RemoteTransportException: [dev-efk-backend04][192.168.0.21:9300][internal:cluster/coordination/join]
  134. Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 14 does not match current term 15
  135. at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:218) ~[elasticsearch-7.0.0.jar:7.0.0]
  136. at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:940) ~[elasticsearch-7.0.0.jar:7.0.0]
  137. at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
  138. at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:512) ~[elasticsearch-7.0.0.jar:7.0.0]
  139. at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:478) ~[elasticsearch-7.0.0.jar:7.0.0]
  140. at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:124) ~[elasticsearch-7.0.0.jar:7.0.0]
  141. at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:251) ~[?:?]
  142. at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.0.0.jar:7.0.0]
  143. at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:309) ~[?:?]
  144. at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.0.0.jar:7.0.0]
  145. at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[elasticsearch-7.0.0.jar:7.0.0]
  146. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[elasticsearch-7.0.0.jar:7.0.0]
  147. at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.0.0.jar:7.0.0]
  148. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
  149. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
  150. at java.lang.Thread.run(Thread.java:835) [?:?]
  151. [2019-04-19T20:23:23,339][INFO ][o.e.c.s.MasterService ] [dev-efk-backend05] elected-as-master ([3] nodes joined)[{dev-efk-backend03}{xYDB_57RTQaUmCzxQVMm-A}{CAfs0z4VT0G78Ttk0unpOg}{192.168.0.142}{192.168.0.142:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true} elect leader, {dev-efk-backend05}{gKWS3gxIRs-P5FFlvMTm4A}{yAp4wKi8SJ6BPzEQo4rGcQ}{192.168.0.239}{192.168.0.239:9300}{ml.machine_memory=16171651072, xpack.installed=true, ml.max_open_jobs=20} elect leader, {dev-efk-backend04}{WPeuO5I1SDWZkVTdpVGqlQ}{_Da2zkZCQGa62XDKg5ej4w}{192.168.0.21}{192.168.0.21:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 15, version: 62, reason: master node changed {previous [], current [{dev-efk-backend05}{gKWS3gxIRs-P5FFlvMTm4A}{yAp4wKi8SJ6BPzEQo4rGcQ}{192.168.0.239}{192.168.0.239:9300}{ml.machine_memory=16171651072, xpack.installed=true, ml.max_open_jobs=20}]}
  152. [2019-04-19T20:23:23,460][INFO ][o.e.c.s.ClusterApplierService] [dev-efk-backend05] master node changed {previous [], current [{dev-efk-backend05}{gKWS3gxIRs-P5FFlvMTm4A}{yAp4wKi8SJ6BPzEQo4rGcQ}{192.168.0.239}{192.168.0.239:9300}{ml.machine_memory=16171651072, xpack.installed=true, ml.max_open_jobs=20}]}, term: 15, version: 62, reason: Publication{term=15, version=62}
  153. [2019-04-19T20:23:23,525][INFO ][o.e.c.s.MasterService ] [dev-efk-backend05] node-left[{dev-efk-backend02}{iO6O8mpUQaGtCk8NWO5X9w}{AvVXSLOrRuGKTZeJe18gYQ}{192.168.0.104}{192.168.0.104:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true} disconnected], term: 15, version: 63, reason: removed {{dev-efk-backend02}{iO6O8mpUQaGtCk8NWO5X9w}{AvVXSLOrRuGKTZeJe18gYQ}{192.168.0.104}{192.168.0.104:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true},}
  154. [2019-04-19T20:23:23,525][DEBUG][o.e.a.a.c.n.s.TransportNodesStatsAction] [dev-efk-backend05] failed to execute on node [iO6O8mpUQaGtCk8NWO5X9w]
  155. org.elasticsearch.transport.NodeNotConnectedException: [dev-efk-backend02][192.168.0.104:9300] Node not connected
  156. at org.elasticsearch.transport.ConnectionManager.getConnection(ConnectionManager.java:151) ~[elasticsearch-7.0.0.jar:7.0.0]
  157. at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:557) ~[elasticsearch-7.0.0.jar:7.0.0]
  158. at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:529) [elasticsearch-7.0.0.jar:7.0.0]
  159. at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.start(TransportNodesAction.java:182) [elasticsearch-7.0.0.jar:7.0.0]
  160. at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:82) [elasticsearch-7.0.0.jar:7.0.0]
  161. at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:51) [elasticsearch-7.0.0.jar:7.0.0]
  162. at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:145) [elasticsearch-7.0.0.jar:7.0.0]
  163. at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:122) [x-pack-security-7.0.0.jar:7.0.0]
  164. at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:143) [elasticsearch-7.0.0.jar:7.0.0]
  165. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:121) [elasticsearch-7.0.0.jar:7.0.0]
  166. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:64) [elasticsearch-7.0.0.jar:7.0.0]
  167. at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) [elasticsearch-7.0.0.jar:7.0.0]
  168. at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) [elasticsearch-7.0.0.jar:7.0.0]
  169. at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:393) [elasticsearch-7.0.0.jar:7.0.0]
  170. at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:682) [elasticsearch-7.0.0.jar:7.0.0]
  171. at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.nodesStats(AbstractClient.java:778) [elasticsearch-7.0.0.jar:7.0.0]
  172. at org.elasticsearch.cluster.InternalClusterInfoService.updateNodeStats(InternalClusterInfoService.java:248) [elasticsearch-7.0.0.jar:7.0.0]
  173. at org.elasticsearch.cluster.InternalClusterInfoService.refresh(InternalClusterInfoService.java:284) [elasticsearch-7.0.0.jar:7.0.0]
  174. at org.elasticsearch.cluster.InternalClusterInfoService.maybeRefresh(InternalClusterInfoService.java:269) [elasticsearch-7.0.0.jar:7.0.0]
  175. at org.elasticsearch.cluster.InternalClusterInfoService.lambda$onMaster$0(InternalClusterInfoService.java:140) [elasticsearch-7.0.0.jar:7.0.0]
  176. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-7.0.0.jar:7.0.0]
  177. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
  178. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
  179. at java.lang.Thread.run(Thread.java:835) [?:?]
  180. [2019-04-19T20:23:23,542][INFO ][o.e.c.s.ClusterApplierService] [dev-efk-backend05] removed {{dev-efk-backend02}{iO6O8mpUQaGtCk8NWO5X9w}{AvVXSLOrRuGKTZeJe18gYQ}{192.168.0.104}{192.168.0.104:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true},}, term: 15, version: 63, reason: Publication{term=15, version=63}
  181. [2019-04-19T20:23:25,953][INFO ][o.e.c.s.MasterService ] [dev-efk-backend05] node-left[{dev-efk-backend03}{xYDB_57RTQaUmCzxQVMm-A}{CAfs0z4VT0G78Ttk0unpOg}{192.168.0.142}{192.168.0.142:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true} disconnected], term: 15, version: 65, reason: removed {{dev-efk-backend03}{xYDB_57RTQaUmCzxQVMm-A}{CAfs0z4VT0G78Ttk0unpOg}{192.168.0.142}{192.168.0.142:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true},}
  182. [2019-04-19T20:23:25,960][INFO ][o.e.c.s.ClusterApplierService] [dev-efk-backend05] removed {{dev-efk-backend03}{xYDB_57RTQaUmCzxQVMm-A}{CAfs0z4VT0G78Ttk0unpOg}{192.168.0.142}{192.168.0.142:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true},}, term: 15, version: 65, reason: Publication{term=15, version=65}
  183. [2019-04-19T20:23:28,522][INFO ][o.e.c.s.MasterService ] [dev-efk-backend05] node-left[{dev-efk-backend04}{WPeuO5I1SDWZkVTdpVGqlQ}{_Da2zkZCQGa62XDKg5ej4w}{192.168.0.21}{192.168.0.21:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true} disconnected], term: 15, version: 66, reason: removed {{dev-efk-backend04}{WPeuO5I1SDWZkVTdpVGqlQ}{_Da2zkZCQGa62XDKg5ej4w}{192.168.0.21}{192.168.0.21:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true},}
  184. [2019-04-19T20:23:28,529][WARN ][o.e.c.s.MasterService ] [dev-efk-backend05] failing [node-left[{dev-efk-backend04}{WPeuO5I1SDWZkVTdpVGqlQ}{_Da2zkZCQGa62XDKg5ej4w}{192.168.0.21}{192.168.0.21:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true} disconnected]]: failed to commit cluster state version [66]
  185. org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: publication failed
  186. at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication$3.onFailure(Coordinator.java:1350) ~[elasticsearch-7.0.0.jar:7.0.0]
  187. at org.elasticsearch.common.util.concurrent.ListenableFuture$1.run(ListenableFuture.java:101) ~[elasticsearch-7.0.0.jar:7.0.0]
  188. at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[elasticsearch-7.0.0.jar:7.0.0]
  189. at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-7.0.0.jar:7.0.0]
  190. at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:54) ~[elasticsearch-7.0.0.jar:7.0.0]
  191. at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication.onCompletion(Coordinator.java:1290) ~[elasticsearch-7.0.0.jar:7.0.0]
  192. at org.elasticsearch.cluster.coordination.Publication.onPossibleCompletion(Publication.java:124) ~[elasticsearch-7.0.0.jar:7.0.0]
  193. at org.elasticsearch.cluster.coordination.Publication.onPossibleCommitFailure(Publication.java:172) ~[elasticsearch-7.0.0.jar:7.0.0]
  194. at org.elasticsearch.cluster.coordination.Publication.start(Publication.java:71) ~[elasticsearch-7.0.0.jar:7.0.0]
  195. at org.elasticsearch.cluster.coordination.Coordinator.publish(Coordinator.java:1041) ~[elasticsearch-7.0.0.jar:7.0.0]
  196. at org.elasticsearch.cluster.service.MasterService.publish(MasterService.java:252) [elasticsearch-7.0.0.jar:7.0.0]
  197. at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:238) [elasticsearch-7.0.0.jar:7.0.0]
  198. at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:142) [elasticsearch-7.0.0.jar:7.0.0]
  199. at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.0.0.jar:7.0.0]
  200. at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.0.0.jar:7.0.0]
  201. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-7.0.0.jar:7.0.0]
  202. at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.0.0.jar:7.0.0]
  203. at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.0.0.jar:7.0.0]
  204. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
  205. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
  206. at java.lang.Thread.run(Thread.java:835) [?:?]
  207. Caused by: org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: non-failed nodes do not form a quorum
  208. at org.elasticsearch.cluster.coordination.Publication.onPossibleCommitFailure(Publication.java:170) ~[elasticsearch-7.0.0.jar:7.0.0]
  209. ... 13 more
  210. [2019-04-19T20:23:28,530][ERROR][o.e.c.c.Coordinator ] [dev-efk-backend05] unexpected failure during [node-left]
  211. org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: publication failed
  212. at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication$3.onFailure(Coordinator.java:1350) ~[elasticsearch-7.0.0.jar:7.0.0]
  213. at org.elasticsearch.common.util.concurrent.ListenableFuture$1.run(ListenableFuture.java:101) ~[elasticsearch-7.0.0.jar:7.0.0]
  214. at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[elasticsearch-7.0.0.jar:7.0.0]
  215. at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-7.0.0.jar:7.0.0]
  216. at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:54) ~[elasticsearch-7.0.0.jar:7.0.0]
  217. at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication.onCompletion(Coordinator.java:1290) ~[elasticsearch-7.0.0.jar:7.0.0]
  218. at org.elasticsearch.cluster.coordination.Publication.onPossibleCompletion(Publication.java:124) ~[elasticsearch-7.0.0.jar:7.0.0]
  219. at org.elasticsearch.cluster.coordination.Publication.onPossibleCommitFailure(Publication.java:172) ~[elasticsearch-7.0.0.jar:7.0.0]
  220. at org.elasticsearch.cluster.coordination.Publication.start(Publication.java:71) ~[elasticsearch-7.0.0.jar:7.0.0]
  221. at org.elasticsearch.cluster.coordination.Coordinator.publish(Coordinator.java:1041) ~[elasticsearch-7.0.0.jar:7.0.0]
  222. at org.elasticsearch.cluster.service.MasterService.publish(MasterService.java:252) [elasticsearch-7.0.0.jar:7.0.0]
  223. at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:238) [elasticsearch-7.0.0.jar:7.0.0]
  224. at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:142) [elasticsearch-7.0.0.jar:7.0.0]
  225. at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.0.0.jar:7.0.0]
  226. at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.0.0.jar:7.0.0]
  227. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-7.0.0.jar:7.0.0]
  228. at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.0.0.jar:7.0.0]
  229. at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.0.0.jar:7.0.0]
  230. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
  231. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
  232. at java.lang.Thread.run(Thread.java:835) [?:?]
  233. Caused by: org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: non-failed nodes do not form a quorum
  234. at org.elasticsearch.cluster.coordination.Publication.onPossibleCommitFailure(Publication.java:170) ~[elasticsearch-7.0.0.jar:7.0.0]
  235. ... 13 more
  236. [2019-04-19T20:23:28,533][INFO ][o.e.c.s.ClusterApplierService] [dev-efk-backend05] master node changed {previous [{dev-efk-backend05}{gKWS3gxIRs-P5FFlvMTm4A}{yAp4wKi8SJ6BPzEQo4rGcQ}{192.168.0.239}{192.168.0.239:9300}{ml.machine_memory=16171651072, xpack.installed=true, ml.max_open_jobs=20}], current []}, term: 15, version: 65, reason: becoming candidate: Publication.onCompletion(false)
  237. [2019-04-19T20:23:28,537][WARN ][o.e.c.NodeConnectionsService] [dev-efk-backend05] failed to connect to node {dev-efk-backend04}{WPeuO5I1SDWZkVTdpVGqlQ}{_Da2zkZCQGa62XDKg5ej4w}{192.168.0.21}{192.168.0.21:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true} (tried [1] times)
  238. org.elasticsearch.transport.ConnectTransportException: [dev-efk-backend04][192.168.0.21:9300] connect_exception
  239. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.0.0.jar:7.0.0]
  240. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.0.0.jar:7.0.0]
  241. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.0.0.jar:7.0.0]
  242. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  243. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  244. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  245. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  246. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.0.0.jar:7.0.0]
  247. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  248. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  249. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  250. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  251. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  252. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  253. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  254. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  255. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  256. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  257. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  258. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  259. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  260. at java.lang.Thread.run(Thread.java:835) [?:?]
  261. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: 192.168.0.21/192.168.0.21:9300
  262. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  263. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  264. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  265. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  266. ... 6 more
  267. Caused by: java.net.ConnectException: Connection refused
  268. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  269. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  270. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  271. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  272. ... 6 more
  273. [2019-04-19T20:23:31,097][INFO ][o.e.n.Node ] [dev-efk-backend05] stopping ...
  274. [2019-04-19T20:23:31,109][INFO ][o.e.x.w.WatcherService ] [dev-efk-backend05] stopping watch service, reason [shutdown initiated]
  275. [2019-04-19T20:23:31,393][INFO ][o.e.n.Node ] [dev-efk-backend05] stopped
  276. [2019-04-19T20:23:31,394][INFO ][o.e.n.Node ] [dev-efk-backend05] closing ...
  277. [2019-04-19T20:23:31,406][INFO ][o.e.n.Node ] [dev-efk-backend05] closed
  278. [2019-04-19T20:23:31,408][INFO ][o.e.x.m.p.NativeController] [dev-efk-backend05] Native controller process has stopped - no new native processes can be started
  279. [2019-04-19T20:24:16,977][WARN ][o.e.c.l.LogConfigurator ] [dev-efk-backend05] Some logging configurations have %marker but don't have %node_name. We will automatically add %node_name to the pattern to ease the migration for users who customize log4j2.properties but will stop this behavior in 7.0. You should manually replace `%node_name` with `[%node_name]%marker ` in these locations:
  280. /etc/elasticsearch/dev/log4j2.properties
  281. [2019-04-19T20:24:17,983][INFO ][o.e.e.NodeEnvironment ] [dev-efk-backend05] using [1] data paths, mounts [[/var/lib/elasticsearch (/dev/nvme1n1)]], net usable_space [999.4gb], net total_space [999.5gb], types [xfs]
  282. [2019-04-19T20:24:17,983][INFO ][o.e.e.NodeEnvironment ] [dev-efk-backend05] heap size [12gb], compressed ordinary object pointers [true]
  283. [2019-04-19T20:24:17,986][INFO ][o.e.n.Node ] [dev-efk-backend05] node name [dev-efk-backend05], node ID [gKWS3gxIRs-P5FFlvMTm4A]
  284. [2019-04-19T20:24:17,986][INFO ][o.e.n.Node ] [dev-efk-backend05] version[7.0.0], pid[24705], build[default/rpm/b7e28a7/2019-04-05T22:55:32.697037Z], OS[Linux/3.10.0-693.11.6.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/12/12+33]
  285. [2019-04-19T20:24:17,986][INFO ][o.e.n.Node ] [dev-efk-backend05] JVM home [/usr/share/elasticsearch/jdk]
  286. [2019-04-19T20:24:17,986][INFO ][o.e.n.Node ] [dev-efk-backend05] JVM arguments [-Xms12338m, -Xmx12338m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch/dev, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
  287. [2019-04-19T20:24:21,006][INFO ][o.e.p.PluginsService ] [dev-efk-backend05] loaded module [aggs-matrix-stats]
  288. [2019-04-19T20:24:21,007][INFO ][o.e.p.PluginsService ] [dev-efk-backend05] loaded module [analysis-common]
  289. [2019-04-19T20:24:21,007][INFO ][o.e.p.PluginsService ] [dev-efk-backend05] loaded module [ingest-common]
  290. [2019-04-19T20:24:21,007][INFO ][o.e.p.PluginsService ] [dev-efk-backend05] loaded module [ingest-geoip]
  291. [2019-04-19T20:24:21,007][INFO ][o.e.p.PluginsService ] [dev-efk-backend05] loaded module [ingest-user-agent]
  292. [2019-04-19T20:24:21,007][INFO ][o.e.p.PluginsService ] [dev-efk-backend05] loaded module [lang-expression]
  293. [2019-04-19T20:24:21,007][INFO ][o.e.p.PluginsService ] [dev-efk-backend05] loaded module [lang-mustache]
  294. [2019-04-19T20:24:21,007][INFO ][o.e.p.PluginsService ] [dev-efk-backend05] loaded module [lang-painless]
  295. [2019-04-19T20:24:21,007][INFO ][o.e.p.PluginsService ] [dev-efk-backend05] loaded module [mapper-extras]
  296. [2019-04-19T20:24:21,007][INFO ][o.e.p.PluginsService ] [dev-efk-backend05] loaded module [parent-join]
  297. [2019-04-19T20:24:21,007][INFO ][o.e.p.PluginsService ] [dev-efk-backend05] loaded module [percolator]
  298. [2019-04-19T20:24:21,007][INFO ][o.e.p.PluginsService ] [dev-efk-backend05] loaded module [rank-eval]
  299. [2019-04-19T20:24:21,008][INFO ][o.e.p.PluginsService ] [dev-efk-backend05] loaded module [reindex]
  300. [2019-04-19T20:24:21,008][INFO ][o.e.p.PluginsService ] [dev-efk-backend05] loaded module [repository-url]
  301. [2019-04-19T20:24:21,008][INFO ][o.e.p.PluginsService ] [dev-efk-backend05] loaded module [transport-netty4]
  302. [2019-04-19T20:24:21,008][INFO ][o.e.p.PluginsService ] [dev-efk-backend05] loaded module [x-pack-ccr]
  303. [2019-04-19T20:24:21,008][INFO ][o.e.p.PluginsService ] [dev-efk-backend05] loaded module [x-pack-core]
  304. [2019-04-19T20:24:21,008][INFO ][o.e.p.PluginsService ] [dev-efk-backend05] loaded module [x-pack-deprecation]
  305. [2019-04-19T20:24:21,008][INFO ][o.e.p.PluginsService ] [dev-efk-backend05] loaded module [x-pack-graph]
  306. [2019-04-19T20:24:21,008][INFO ][o.e.p.PluginsService ] [dev-efk-backend05] loaded module [x-pack-ilm]
  307. [2019-04-19T20:24:21,008][INFO ][o.e.p.PluginsService ] [dev-efk-backend05] loaded module [x-pack-logstash]
  308. [2019-04-19T20:24:21,008][INFO ][o.e.p.PluginsService ] [dev-efk-backend05] loaded module [x-pack-ml]
  309. [2019-04-19T20:24:21,008][INFO ][o.e.p.PluginsService ] [dev-efk-backend05] loaded module [x-pack-monitoring]
  310. [2019-04-19T20:24:21,008][INFO ][o.e.p.PluginsService ] [dev-efk-backend05] loaded module [x-pack-rollup]
  311. [2019-04-19T20:24:21,009][INFO ][o.e.p.PluginsService ] [dev-efk-backend05] loaded module [x-pack-security]
  312. [2019-04-19T20:24:21,009][INFO ][o.e.p.PluginsService ] [dev-efk-backend05] loaded module [x-pack-sql]
  313. [2019-04-19T20:24:21,009][INFO ][o.e.p.PluginsService ] [dev-efk-backend05] loaded module [x-pack-watcher]
  314. [2019-04-19T20:24:21,009][INFO ][o.e.p.PluginsService ] [dev-efk-backend05] no plugins loaded
  315. [2019-04-19T20:24:27,391][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [dev-efk-backend05] [controller/24873] [Main.cc@109] controller (64 bit): Version 7.0.0 (Build cdaa022645f38d) Copyright (c) 2019 Elasticsearch BV
  316. [2019-04-19T20:24:28,088][DEBUG][o.e.a.ActionModule ] [dev-efk-backend05] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
  317. [2019-04-19T20:24:28,458][INFO ][o.e.d.DiscoveryModule ] [dev-efk-backend05] using discovery type [zen] and seed hosts providers [settings]
  318. [2019-04-19T20:24:29,430][INFO ][o.e.n.Node ] [dev-efk-backend05] initialized
  319. [2019-04-19T20:24:29,430][INFO ][o.e.n.Node ] [dev-efk-backend05] starting ...
  320. [2019-04-19T20:24:29,690][INFO ][o.e.t.TransportService ] [dev-efk-backend05] publish_address {192.168.0.239:9300}, bound_addresses {[::]:9300}
  321. [2019-04-19T20:24:29,698][INFO ][o.e.b.BootstrapChecks ] [dev-efk-backend05] bound or publishing to a non-loopback address, enforcing bootstrap checks
  322. [2019-04-19T20:24:30,413][INFO ][o.e.c.s.ClusterApplierService] [dev-efk-backend05] master node changed {previous [], current [{dev-efk-backend04}{WPeuO5I1SDWZkVTdpVGqlQ}{TqQgfsm_RfWyZtyyq-VJCw}{192.168.0.21}{192.168.0.21:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true}]}, added {{dev-efk-backend03}{xYDB_57RTQaUmCzxQVMm-A}{7DyVRWQQRZO4DaQThEOKVA}{192.168.0.142}{192.168.0.142:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true},{dev-efk-backend04}{WPeuO5I1SDWZkVTdpVGqlQ}{TqQgfsm_RfWyZtyyq-VJCw}{192.168.0.21}{192.168.0.21:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true},{dev-efk-backend01}{9-ePQkHuS_6IAyGpYSCyDg}{rIyhgOV9TPCZc2TIR-nZnA}{192.168.0.202}{192.168.0.202:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true},}, term: 16, version: 71, reason: ApplyCommitRequest{term=16, version=71, sourceNode={dev-efk-backend04}{WPeuO5I1SDWZkVTdpVGqlQ}{TqQgfsm_RfWyZtyyq-VJCw}{192.168.0.21}{192.168.0.21:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true}}
  323. [2019-04-19T20:24:30,569][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [dev-efk-backend05] Failed to clear cache for realms [[]]
  324. [2019-04-19T20:24:30,571][INFO ][o.e.x.s.a.TokenService ] [dev-efk-backend05] refresh keys
  325. [2019-04-19T20:24:30,816][INFO ][o.e.x.s.a.TokenService ] [dev-efk-backend05] refreshed keys
  326. [2019-04-19T20:24:30,854][INFO ][o.e.l.LicenseService ] [dev-efk-backend05] license [790c1d0b-d017-4809-9846-3616d274a2cb] mode [basic] - valid
  327. [2019-04-19T20:24:30,887][INFO ][o.e.h.AbstractHttpServerTransport] [dev-efk-backend05] publish_address {192.168.0.239:9200}, bound_addresses {[::]:9200}
  328. [2019-04-19T20:24:30,887][INFO ][o.e.n.Node ] [dev-efk-backend05] started
  329. [2019-04-19T20:24:47,691][INFO ][o.e.c.s.ClusterApplierService] [dev-efk-backend05] added {{dev-efk-backend02}{iO6O8mpUQaGtCk8NWO5X9w}{mKdecz4bRcKqFWRMAARzZw}{192.168.0.104}{192.168.0.104:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true},}, term: 16, version: 72, reason: ApplyCommitRequest{term=16, version=72, sourceNode={dev-efk-backend04}{WPeuO5I1SDWZkVTdpVGqlQ}{TqQgfsm_RfWyZtyyq-VJCw}{192.168.0.21}{192.168.0.21:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true}}
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!