API tools faq
paste
Advertisement
Guest User

Untitled

a guest
Apr 19th, 2019
351
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 32.42 KB | None | 0 0
raw download clone embed print report
  1. [2019-04-19T20:23:18,601][INFO ][o.e.c.s.ClusterApplierService] [dev-efk-backend03] master node changed {previous [{dev-efk-backend01}{9-ePQkHuS_6IAyGpYSCyDg}{Ba7wi43ZS6irFH-87BMpqQ}{192.168.0.202}{192.168.0.202:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true}], current []}, term: 9, version: 57, reason: becoming candidate: onLeaderFailure
  2. [2019-04-19T20:23:18,676][WARN ][o.e.c.NodeConnectionsService] [dev-efk-backend03] failed to connect to node {dev-efk-backend01}{9-ePQkHuS_6IAyGpYSCyDg}{Ba7wi43ZS6irFH-87BMpqQ}{192.168.0.202}{192.168.0.202:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true} (tried [1] times)
  3. org.elasticsearch.transport.ConnectTransportException: [dev-efk-backend01][192.168.0.202:9300] connect_exception
  4. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.0.0.jar:7.0.0]
  5. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.0.0.jar:7.0.0]
  6. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.0.0.jar:7.0.0]
  7. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  8. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  9. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  10. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  11. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.0.0.jar:7.0.0]
  12. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  13. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  14. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  15. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  16. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  17. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  18. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  19. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  20. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  21. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  22. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  23. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  24. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  25. at java.lang.Thread.run(Thread.java:835) [?:?]
  26. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: 192.168.0.202/192.168.0.202:9300
  27. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  28. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  29. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  30. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  31. ... 6 more
  32. Caused by: java.net.ConnectException: Connection refused
  33. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  34. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  35. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  36. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  37. ... 6 more
  38. [2019-04-19T20:23:18,707][INFO ][o.e.c.c.JoinHelper ] [dev-efk-backend03] failed to join {dev-efk-backend05}{gKWS3gxIRs-P5FFlvMTm4A}{yAp4wKi8SJ6BPzEQo4rGcQ}{192.168.0.239}{192.168.0.239:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={dev-efk-backend03}{xYDB_57RTQaUmCzxQVMm-A}{CAfs0z4VT0G78Ttk0unpOg}{192.168.0.142}{192.168.0.142:9300}{ml.machine_memory=16171651072, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=10, lastAcceptedTerm=9, lastAcceptedVersion=57, sourceNode={dev-efk-backend03}{xYDB_57RTQaUmCzxQVMm-A}{CAfs0z4VT0G78Ttk0unpOg}{192.168.0.142}{192.168.0.142:9300}{ml.machine_memory=16171651072, xpack.installed=true, ml.max_open_jobs=20}, targetNode={dev-efk-backend05}{gKWS3gxIRs-P5FFlvMTm4A}{yAp4wKi8SJ6BPzEQo4rGcQ}{192.168.0.239}{192.168.0.239:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true}}]}
  39. org.elasticsearch.transport.RemoteTransportException: [dev-efk-backend05][192.168.0.239:9300][internal:cluster/coordination/join]
  40. Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 10 does not match current term 11
  41. at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:218) ~[elasticsearch-7.0.0.jar:7.0.0]
  42. at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:940) ~[elasticsearch-7.0.0.jar:7.0.0]
  43. at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
  44. at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:512) ~[elasticsearch-7.0.0.jar:7.0.0]
  45. at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:478) ~[elasticsearch-7.0.0.jar:7.0.0]
  46. at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:124) ~[elasticsearch-7.0.0.jar:7.0.0]
  47. at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:251) ~[?:?]
  48. at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.0.0.jar:7.0.0]
  49. at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:309) ~[?:?]
  50. at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.0.0.jar:7.0.0]
  51. at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[elasticsearch-7.0.0.jar:7.0.0]
  52. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[elasticsearch-7.0.0.jar:7.0.0]
  53. at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.0.0.jar:7.0.0]
  54. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
  55. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
  56. at java.lang.Thread.run(Thread.java:835) [?:?]
  57. [2019-04-19T20:23:18,769][INFO ][o.e.c.s.MasterService ] [dev-efk-backend03] elected-as-master ([2] nodes joined)[{dev-efk-backend02}{iO6O8mpUQaGtCk8NWO5X9w}{AvVXSLOrRuGKTZeJe18gYQ}{192.168.0.104}{192.168.0.104:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true} elect leader, {dev-efk-backend05}{gKWS3gxIRs-P5FFlvMTm4A}{yAp4wKi8SJ6BPzEQo4rGcQ}{192.168.0.239}{192.168.0.239:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 12, version: 58, reason: master node changed {previous [], current [{dev-efk-backend03}{xYDB_57RTQaUmCzxQVMm-A}{CAfs0z4VT0G78Ttk0unpOg}{192.168.0.142}{192.168.0.142:9300}{ml.machine_memory=16171651072, xpack.installed=true, ml.max_open_jobs=20}]}
  58. [2019-04-19T20:23:18,772][WARN ][o.e.c.s.MasterService ] [dev-efk-backend03] failing [elected-as-master ([2] nodes joined)[{dev-efk-backend02}{iO6O8mpUQaGtCk8NWO5X9w}{AvVXSLOrRuGKTZeJe18gYQ}{192.168.0.104}{192.168.0.104:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true} elect leader, {dev-efk-backend05}{gKWS3gxIRs-P5FFlvMTm4A}{yAp4wKi8SJ6BPzEQo4rGcQ}{192.168.0.239}{192.168.0.239:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_]]: failed to commit cluster state version [58]
  59. org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: node is no longer master for term 12 while handling publication
  60. at org.elasticsearch.cluster.coordination.Coordinator.publish(Coordinator.java:1010) ~[elasticsearch-7.0.0.jar:7.0.0]
  61. at org.elasticsearch.cluster.service.MasterService.publish(MasterService.java:252) [elasticsearch-7.0.0.jar:7.0.0]
  62. at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:238) [elasticsearch-7.0.0.jar:7.0.0]
  63. at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:142) [elasticsearch-7.0.0.jar:7.0.0]
  64. at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.0.0.jar:7.0.0]
  65. at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.0.0.jar:7.0.0]
  66. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-7.0.0.jar:7.0.0]
  67. at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.0.0.jar:7.0.0]
  68. at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.0.0.jar:7.0.0]
  69. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
  70. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
  71. at java.lang.Thread.run(Thread.java:835) [?:?]
  72. [2019-04-19T20:23:18,871][INFO ][o.e.c.c.JoinHelper ] [dev-efk-backend03] failed to join {dev-efk-backend04}{WPeuO5I1SDWZkVTdpVGqlQ}{_Da2zkZCQGa62XDKg5ej4w}{192.168.0.21}{192.168.0.21:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={dev-efk-backend03}{xYDB_57RTQaUmCzxQVMm-A}{CAfs0z4VT0G78Ttk0unpOg}{192.168.0.142}{192.168.0.142:9300}{ml.machine_memory=16171651072, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=11, lastAcceptedTerm=9, lastAcceptedVersion=57, sourceNode={dev-efk-backend03}{xYDB_57RTQaUmCzxQVMm-A}{CAfs0z4VT0G78Ttk0unpOg}{192.168.0.142}{192.168.0.142:9300}{ml.machine_memory=16171651072, xpack.installed=true, ml.max_open_jobs=20}, targetNode={dev-efk-backend04}{WPeuO5I1SDWZkVTdpVGqlQ}{_Da2zkZCQGa62XDKg5ej4w}{192.168.0.21}{192.168.0.21:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true}}]}
  73. org.elasticsearch.transport.RemoteTransportException: [dev-efk-backend04][192.168.0.21:9300][internal:cluster/coordination/join]
  74. Caused by: org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: node is no longer master for term 12 while handling publication
  75. at org.elasticsearch.cluster.coordination.Coordinator.publish(Coordinator.java:1010) ~[elasticsearch-7.0.0.jar:7.0.0]
  76. at org.elasticsearch.cluster.service.MasterService.publish(MasterService.java:252) ~[elasticsearch-7.0.0.jar:7.0.0]
  77. at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:238) ~[elasticsearch-7.0.0.jar:7.0.0]
  78. at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:142) ~[elasticsearch-7.0.0.jar:7.0.0]
  79. at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) ~[elasticsearch-7.0.0.jar:7.0.0]
  80. at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) ~[elasticsearch-7.0.0.jar:7.0.0]
  81. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) ~[elasticsearch-7.0.0.jar:7.0.0]
  82. at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) ~[elasticsearch-7.0.0.jar:7.0.0]
  83. at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) ~[elasticsearch-7.0.0.jar:7.0.0]
  84. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
  85. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
  86. at java.lang.Thread.run(Thread.java:835) [?:?]
  87. [2019-04-19T20:23:18,890][INFO ][o.e.c.s.ClusterApplierService] [dev-efk-backend03] master node changed {previous [], current [{dev-efk-backend02}{iO6O8mpUQaGtCk8NWO5X9w}{AvVXSLOrRuGKTZeJe18gYQ}{192.168.0.104}{192.168.0.104:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true}]}, term: 13, version: 58, reason: ApplyCommitRequest{term=13, version=58, sourceNode={dev-efk-backend02}{iO6O8mpUQaGtCk8NWO5X9w}{AvVXSLOrRuGKTZeJe18gYQ}{192.168.0.104}{192.168.0.104:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true}}
  88. [2019-04-19T20:23:18,975][INFO ][o.e.c.s.ClusterApplierService] [dev-efk-backend03] removed {{dev-efk-backend01}{9-ePQkHuS_6IAyGpYSCyDg}{Ba7wi43ZS6irFH-87BMpqQ}{192.168.0.202}{192.168.0.202:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true},}, term: 13, version: 59, reason: ApplyCommitRequest{term=13, version=59, sourceNode={dev-efk-backend02}{iO6O8mpUQaGtCk8NWO5X9w}{AvVXSLOrRuGKTZeJe18gYQ}{192.168.0.104}{192.168.0.104:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true}}
  89. [2019-04-19T20:23:23,220][INFO ][o.e.c.s.ClusterApplierService] [dev-efk-backend03] master node changed {previous [{dev-efk-backend02}{iO6O8mpUQaGtCk8NWO5X9w}{AvVXSLOrRuGKTZeJe18gYQ}{192.168.0.104}{192.168.0.104:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true}], current []}, term: 13, version: 61, reason: becoming candidate: onLeaderFailure
  90. [2019-04-19T20:23:23,233][WARN ][o.e.c.NodeConnectionsService] [dev-efk-backend03] failed to connect to node {dev-efk-backend02}{iO6O8mpUQaGtCk8NWO5X9w}{AvVXSLOrRuGKTZeJe18gYQ}{192.168.0.104}{192.168.0.104:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true} (tried [1] times)
  91. org.elasticsearch.transport.ConnectTransportException: [dev-efk-backend02][192.168.0.104:9300] connect_exception
  92. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.0.0.jar:7.0.0]
  93. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.0.0.jar:7.0.0]
  94. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.0.0.jar:7.0.0]
  95. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  96. at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:883) ~[?:?]
  97. at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2322) ~[?:?]
  98. at org.elasticsearch.common.concurrent.CompletableContext.addListener(CompletableContext.java:45) ~[elasticsearch-core-7.0.0.jar:7.0.0]
  99. at org.elasticsearch.transport.netty4.Netty4TcpChannel.addConnectListener(Netty4TcpChannel.java:100) ~[?:?]
  100. at org.elasticsearch.transport.TcpTransport.initiateConnection(TcpTransport.java:325) ~[elasticsearch-7.0.0.jar:7.0.0]
  101. at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:292) ~[elasticsearch-7.0.0.jar:7.0.0]
  102. at org.elasticsearch.transport.ConnectionManager.internalOpenConnection(ConnectionManager.java:206) ~[elasticsearch-7.0.0.jar:7.0.0]
  103. at org.elasticsearch.transport.ConnectionManager.connectToNode(ConnectionManager.java:104) ~[elasticsearch-7.0.0.jar:7.0.0]
  104. at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:343) ~[elasticsearch-7.0.0.jar:7.0.0]
  105. at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:330) ~[elasticsearch-7.0.0.jar:7.0.0]
  106. at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:153) [elasticsearch-7.0.0.jar:7.0.0]
  107. at org.elasticsearch.cluster.NodeConnectionsService$1.doRun(NodeConnectionsService.java:106) [elasticsearch-7.0.0.jar:7.0.0]
  108. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [elasticsearch-7.0.0.jar:7.0.0]
  109. at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.0.jar:7.0.0]
  110. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
  111. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
  112. at java.lang.Thread.run(Thread.java:835) [?:?]
  113. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: 192.168.0.104/192.168.0.104:9300
  114. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  115. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  116. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  117. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  118. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  119. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  120. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  121. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  122. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  123. ... 1 more
  124. Caused by: java.net.ConnectException: Connection refused
  125. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  126. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  127. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  128. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  129. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  130. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  131. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  132. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  133. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  134. ... 1 more
  135. [2019-04-19T20:23:23,356][INFO ][o.e.c.c.JoinHelper ] [dev-efk-backend03] failed to join {dev-efk-backend04}{WPeuO5I1SDWZkVTdpVGqlQ}{_Da2zkZCQGa62XDKg5ej4w}{192.168.0.21}{192.168.0.21:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={dev-efk-backend03}{xYDB_57RTQaUmCzxQVMm-A}{CAfs0z4VT0G78Ttk0unpOg}{192.168.0.142}{192.168.0.142:9300}{ml.machine_memory=16171651072, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=14, lastAcceptedTerm=13, lastAcceptedVersion=61, sourceNode={dev-efk-backend03}{xYDB_57RTQaUmCzxQVMm-A}{CAfs0z4VT0G78Ttk0unpOg}{192.168.0.142}{192.168.0.142:9300}{ml.machine_memory=16171651072, xpack.installed=true, ml.max_open_jobs=20}, targetNode={dev-efk-backend04}{WPeuO5I1SDWZkVTdpVGqlQ}{_Da2zkZCQGa62XDKg5ej4w}{192.168.0.21}{192.168.0.21:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true}}]}
  136. org.elasticsearch.transport.RemoteTransportException: [dev-efk-backend04][192.168.0.21:9300][internal:cluster/coordination/join]
  137. Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: became follower
  138. at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.lambda$close$3(JoinHelper.java:468) ~[elasticsearch-7.0.0.jar:7.0.0]
  139. at java.util.HashMap$Values.forEach(HashMap.java:976) ~[?:?]
  140. at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.close(JoinHelper.java:468) ~[elasticsearch-7.0.0.jar:7.0.0]
  141. at org.elasticsearch.cluster.coordination.Coordinator.becomeFollower(Coordinator.java:601) ~[elasticsearch-7.0.0.jar:7.0.0]
  142. at org.elasticsearch.cluster.coordination.Coordinator.onFollowerCheckRequest(Coordinator.java:237) ~[elasticsearch-7.0.0.jar:7.0.0]
  143. at org.elasticsearch.cluster.coordination.FollowersChecker$2.doRun(FollowersChecker.java:187) ~[elasticsearch-7.0.0.jar:7.0.0]
  144. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[elasticsearch-7.0.0.jar:7.0.0]
  145. at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.0.0.jar:7.0.0]
  146. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
  147. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
  148. at java.lang.Thread.run(Thread.java:835) [?:?]
  149. [2019-04-19T20:23:23,434][INFO ][o.e.c.s.ClusterApplierService] [dev-efk-backend03] master node changed {previous [], current [{dev-efk-backend05}{gKWS3gxIRs-P5FFlvMTm4A}{yAp4wKi8SJ6BPzEQo4rGcQ}{192.168.0.239}{192.168.0.239:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true}]}, term: 15, version: 62, reason: ApplyCommitRequest{term=15, version=62, sourceNode={dev-efk-backend05}{gKWS3gxIRs-P5FFlvMTm4A}{yAp4wKi8SJ6BPzEQo4rGcQ}{192.168.0.239}{192.168.0.239:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true}}
  150. [2019-04-19T20:23:23,541][INFO ][o.e.c.s.ClusterApplierService] [dev-efk-backend03] removed {{dev-efk-backend02}{iO6O8mpUQaGtCk8NWO5X9w}{AvVXSLOrRuGKTZeJe18gYQ}{192.168.0.104}{192.168.0.104:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true},}, term: 15, version: 63, reason: ApplyCommitRequest{term=15, version=63, sourceNode={dev-efk-backend05}{gKWS3gxIRs-P5FFlvMTm4A}{yAp4wKi8SJ6BPzEQo4rGcQ}{192.168.0.239}{192.168.0.239:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true}}
  151. [2019-04-19T20:23:25,629][INFO ][o.e.n.Node ] [dev-efk-backend03] stopping ...
  152. [2019-04-19T20:23:25,639][INFO ][o.e.x.w.WatcherService ] [dev-efk-backend03] stopping watch service, reason [shutdown initiated]
  153. [2019-04-19T20:23:25,957][INFO ][o.e.n.Node ] [dev-efk-backend03] stopped
  154. [2019-04-19T20:23:25,957][INFO ][o.e.n.Node ] [dev-efk-backend03] closing ...
  155. [2019-04-19T20:23:25,970][INFO ][o.e.n.Node ] [dev-efk-backend03] closed
  156. [2019-04-19T20:23:25,971][INFO ][o.e.x.m.p.NativeController] [dev-efk-backend03] Native controller process has stopped - no new native processes can be started
  157. [2019-04-19T20:24:10,884][WARN ][o.e.c.l.LogConfigurator ] [dev-efk-backend03] Some logging configurations have %marker but don't have %node_name. We will automatically add %node_name to the pattern to ease the migration for users who customize log4j2.properties but will stop this behavior in 7.0. You should manually replace `%node_name` with `[%node_name]%marker ` in these locations:
  158. /etc/elasticsearch/dev/log4j2.properties
  159. [2019-04-19T20:24:11,583][INFO ][o.e.e.NodeEnvironment ] [dev-efk-backend03] using [1] data paths, mounts [[/var/lib/elasticsearch (/dev/nvme1n1)]], net usable_space [999.4gb], net total_space [999.5gb], types [xfs]
  160. [2019-04-19T20:24:11,583][INFO ][o.e.e.NodeEnvironment ] [dev-efk-backend03] heap size [12gb], compressed ordinary object pointers [true]
  161. [2019-04-19T20:24:11,586][INFO ][o.e.n.Node ] [dev-efk-backend03] node name [dev-efk-backend03], node ID [xYDB_57RTQaUmCzxQVMm-A]
  162. [2019-04-19T20:24:11,586][INFO ][o.e.n.Node ] [dev-efk-backend03] version[7.0.0], pid[24126], build[default/rpm/b7e28a7/2019-04-05T22:55:32.697037Z], OS[Linux/3.10.0-693.11.6.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/12/12+33]
  163. [2019-04-19T20:24:11,587][INFO ][o.e.n.Node ] [dev-efk-backend03] JVM home [/usr/share/elasticsearch/jdk]
  164. [2019-04-19T20:24:11,587][INFO ][o.e.n.Node ] [dev-efk-backend03] JVM arguments [-Xms12338m, -Xmx12338m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch/dev, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
  165. [2019-04-19T20:24:14,360][INFO ][o.e.p.PluginsService ] [dev-efk-backend03] loaded module [aggs-matrix-stats]
  166. [2019-04-19T20:24:14,361][INFO ][o.e.p.PluginsService ] [dev-efk-backend03] loaded module [analysis-common]
  167. [2019-04-19T20:24:14,361][INFO ][o.e.p.PluginsService ] [dev-efk-backend03] loaded module [ingest-common]
  168. [2019-04-19T20:24:14,361][INFO ][o.e.p.PluginsService ] [dev-efk-backend03] loaded module [ingest-geoip]
  169. [2019-04-19T20:24:14,361][INFO ][o.e.p.PluginsService ] [dev-efk-backend03] loaded module [ingest-user-agent]
  170. [2019-04-19T20:24:14,361][INFO ][o.e.p.PluginsService ] [dev-efk-backend03] loaded module [lang-expression]
  171. [2019-04-19T20:24:14,361][INFO ][o.e.p.PluginsService ] [dev-efk-backend03] loaded module [lang-mustache]
  172. [2019-04-19T20:24:14,361][INFO ][o.e.p.PluginsService ] [dev-efk-backend03] loaded module [lang-painless]
  173. [2019-04-19T20:24:14,361][INFO ][o.e.p.PluginsService ] [dev-efk-backend03] loaded module [mapper-extras]
  174. [2019-04-19T20:24:14,361][INFO ][o.e.p.PluginsService ] [dev-efk-backend03] loaded module [parent-join]
  175. [2019-04-19T20:24:14,361][INFO ][o.e.p.PluginsService ] [dev-efk-backend03] loaded module [percolator]
  176. [2019-04-19T20:24:14,362][INFO ][o.e.p.PluginsService ] [dev-efk-backend03] loaded module [rank-eval]
  177. [2019-04-19T20:24:14,362][INFO ][o.e.p.PluginsService ] [dev-efk-backend03] loaded module [reindex]
  178. [2019-04-19T20:24:14,362][INFO ][o.e.p.PluginsService ] [dev-efk-backend03] loaded module [repository-url]
  179. [2019-04-19T20:24:14,362][INFO ][o.e.p.PluginsService ] [dev-efk-backend03] loaded module [transport-netty4]
  180. [2019-04-19T20:24:14,362][INFO ][o.e.p.PluginsService ] [dev-efk-backend03] loaded module [x-pack-ccr]
  181. [2019-04-19T20:24:14,362][INFO ][o.e.p.PluginsService ] [dev-efk-backend03] loaded module [x-pack-core]
  182. [2019-04-19T20:24:14,362][INFO ][o.e.p.PluginsService ] [dev-efk-backend03] loaded module [x-pack-deprecation]
  183. [2019-04-19T20:24:14,362][INFO ][o.e.p.PluginsService ] [dev-efk-backend03] loaded module [x-pack-graph]
  184. [2019-04-19T20:24:14,362][INFO ][o.e.p.PluginsService ] [dev-efk-backend03] loaded module [x-pack-ilm]
  185. [2019-04-19T20:24:14,362][INFO ][o.e.p.PluginsService ] [dev-efk-backend03] loaded module [x-pack-logstash]
  186. [2019-04-19T20:24:14,362][INFO ][o.e.p.PluginsService ] [dev-efk-backend03] loaded module [x-pack-ml]
  187. [2019-04-19T20:24:14,363][INFO ][o.e.p.PluginsService ] [dev-efk-backend03] loaded module [x-pack-monitoring]
  188. [2019-04-19T20:24:14,363][INFO ][o.e.p.PluginsService ] [dev-efk-backend03] loaded module [x-pack-rollup]
  189. [2019-04-19T20:24:14,363][INFO ][o.e.p.PluginsService ] [dev-efk-backend03] loaded module [x-pack-security]
  190. [2019-04-19T20:24:14,363][INFO ][o.e.p.PluginsService ] [dev-efk-backend03] loaded module [x-pack-sql]
  191. [2019-04-19T20:24:14,363][INFO ][o.e.p.PluginsService ] [dev-efk-backend03] loaded module [x-pack-watcher]
  192. [2019-04-19T20:24:14,363][INFO ][o.e.p.PluginsService ] [dev-efk-backend03] no plugins loaded
  193. [2019-04-19T20:24:20,557][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [dev-efk-backend03] [controller/24301] [Main.cc@109] controller (64 bit): Version 7.0.0 (Build cdaa022645f38d) Copyright (c) 2019 Elasticsearch BV
  194. [2019-04-19T20:24:21,281][DEBUG][o.e.a.ActionModule ] [dev-efk-backend03] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
  195. [2019-04-19T20:24:21,714][INFO ][o.e.d.DiscoveryModule ] [dev-efk-backend03] using discovery type [zen] and seed hosts providers [settings]
  196. [2019-04-19T20:24:22,712][INFO ][o.e.n.Node ] [dev-efk-backend03] initialized
  197. [2019-04-19T20:24:22,712][INFO ][o.e.n.Node ] [dev-efk-backend03] starting ...
  198. [2019-04-19T20:24:22,905][INFO ][o.e.t.TransportService ] [dev-efk-backend03] publish_address {192.168.0.142:9300}, bound_addresses {[::]:9300}
  199. [2019-04-19T20:24:22,911][INFO ][o.e.b.BootstrapChecks ] [dev-efk-backend03] bound or publishing to a non-loopback address, enforcing bootstrap checks
  200. [2019-04-19T20:24:26,997][INFO ][o.e.c.s.ClusterApplierService] [dev-efk-backend03] master node changed {previous [], current [{dev-efk-backend04}{WPeuO5I1SDWZkVTdpVGqlQ}{TqQgfsm_RfWyZtyyq-VJCw}{192.168.0.21}{192.168.0.21:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true}]}, added {{dev-efk-backend04}{WPeuO5I1SDWZkVTdpVGqlQ}{TqQgfsm_RfWyZtyyq-VJCw}{192.168.0.21}{192.168.0.21:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true},{dev-efk-backend01}{9-ePQkHuS_6IAyGpYSCyDg}{rIyhgOV9TPCZc2TIR-nZnA}{192.168.0.202}{192.168.0.202:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true},{dev-efk-backend02}{iO6O8mpUQaGtCk8NWO5X9w}{83WWsic7TXOTr4yq9j7mOA}{192.168.0.104}{192.168.0.104:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true},}, term: 16, version: 66, reason: ApplyCommitRequest{term=16, version=66, sourceNode={dev-efk-backend04}{WPeuO5I1SDWZkVTdpVGqlQ}{TqQgfsm_RfWyZtyyq-VJCw}{192.168.0.21}{192.168.0.21:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true}}
  201. [2019-04-19T20:24:27,025][INFO ][o.e.h.AbstractHttpServerTransport] [dev-efk-backend03] publish_address {192.168.0.142:9200}, bound_addresses {[::]:9200}
  202. [2019-04-19T20:24:27,025][INFO ][o.e.n.Node ] [dev-efk-backend03] started
  203. [2019-04-19T20:24:27,153][INFO ][o.e.x.m.e.l.LocalExporter] [dev-efk-backend03] waiting for elected master node [{dev-efk-backend04}{WPeuO5I1SDWZkVTdpVGqlQ}{TqQgfsm_RfWyZtyyq-VJCw}{192.168.0.21}{192.168.0.21:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true}] to setup local exporter [default_local] (does it have x-pack installed?)
  204. [2019-04-19T20:24:27,316][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [dev-efk-backend03] Failed to clear cache for realms [[]]
  205. [2019-04-19T20:24:27,340][INFO ][o.e.l.LicenseService ] [dev-efk-backend03] license [790c1d0b-d017-4809-9846-3616d274a2cb] mode [basic] - valid
  206. [2019-04-19T20:24:27,629][INFO ][o.e.x.s.a.TokenService ] [dev-efk-backend03] refresh keys
  207. [2019-04-19T20:24:27,813][INFO ][o.e.x.s.a.TokenService ] [dev-efk-backend03] refreshed keys
  208. [2019-04-19T20:24:29,005][INFO ][o.e.c.s.ClusterApplierService] [dev-efk-backend03] removed {{dev-efk-backend02}{iO6O8mpUQaGtCk8NWO5X9w}{83WWsic7TXOTr4yq9j7mOA}{192.168.0.104}{192.168.0.104:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true},}, term: 16, version: 70, reason: ApplyCommitRequest{term=16, version=70, sourceNode={dev-efk-backend04}{WPeuO5I1SDWZkVTdpVGqlQ}{TqQgfsm_RfWyZtyyq-VJCw}{192.168.0.21}{192.168.0.21:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true}}
  209. [2019-04-19T20:24:30,314][INFO ][o.e.c.s.ClusterApplierService] [dev-efk-backend03] added {{dev-efk-backend05}{gKWS3gxIRs-P5FFlvMTm4A}{6OIwyAcbRx64idykQ2h8LQ}{192.168.0.239}{192.168.0.239:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true},}, term: 16, version: 71, reason: ApplyCommitRequest{term=16, version=71, sourceNode={dev-efk-backend04}{WPeuO5I1SDWZkVTdpVGqlQ}{TqQgfsm_RfWyZtyyq-VJCw}{192.168.0.21}{192.168.0.21:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true}}
  210. [2019-04-19T20:24:47,685][INFO ][o.e.c.s.ClusterApplierService] [dev-efk-backend03] added {{dev-efk-backend02}{iO6O8mpUQaGtCk8NWO5X9w}{mKdecz4bRcKqFWRMAARzZw}{192.168.0.104}{192.168.0.104:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true},}, term: 16, version: 72, reason: ApplyCommitRequest{term=16, version=72, sourceNode={dev-efk-backend04}{WPeuO5I1SDWZkVTdpVGqlQ}{TqQgfsm_RfWyZtyyq-VJCw}{192.168.0.21}{192.168.0.21:9300}{ml.machine_memory=16171651072, ml.max_open_jobs=20, xpack.installed=true}}
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!