API tools faq
paste
Advertisement
KO_1984

N2 Elasticsearch.log

KO_1984
Apr 5th, 2017
71
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 89.13 KB | None | 0 0
raw download clone embed print report
  1. [2017-04-04 14:06:58,268][INFO ][cluster.routing.allocation] [node-2] Cluster health status changed from [GREEN] to [RED] (reason: [{node-1}{sG50XWssTbalcYDKzAr9gA}{192.168.37.52}{192.168.37.52:9300} left]).
  2. [2017-04-04 14:06:58,270][INFO ][cluster.service ] [node-2] removed {{node-1}{sG50XWssTbalcYDKzAr9gA}{192.168.37.52}{192.168.37.52:9300},}, reason: zen-disco-node-left({node-1}{sG50XWssTbalcYDKzAr9gA}{192.168.37.52}{192.168.37.52:9300}), reason(left)
  3. [2017-04-04 14:06:58,306][INFO ][cluster.routing ] [node-2] delaying allocation for [190] unassigned shards, next check in [1m]
  4. [2017-04-04 14:07:01,641][INFO ][node ] [node-2] stopping ...
  5. [2017-04-04 14:07:09,117][INFO ][node ] [node-2] stopped
  6. [2017-04-04 14:07:09,117][INFO ][node ] [node-2] closing ...
  7. [2017-04-04 14:07:09,139][INFO ][node ] [node-2] closed
  8. [2017-04-04 14:10:02,034][INFO ][node ] [node-2] version[2.4.4], pid[1379], build[fcbb46d/2017-01-03T11:33:16Z]
  9. [2017-04-04 14:10:02,044][INFO ][node ] [node-2] initializing ...
  10. [2017-04-04 14:10:03,110][INFO ][plugins ] [node-2] modules [reindex, lang-expression, lang-groovy], plugins [], sites []
  11. [2017-04-04 14:10:03,162][INFO ][env ] [node-2] using [1] data paths, mounts [[/ (/dev/mapper/GraySrv2--vg-root)]], net usable_space [2.4tb], net total_space [2.6tb], spins? [possibly], types [ext4]
  12. [2017-04-04 14:10:03,162][INFO ][env ] [node-2] heap size [989.8mb], compressed ordinary object pointers [true]
  13. [2017-04-04 14:10:06,231][INFO ][node ] [node-2] initialized
  14. [2017-04-04 14:10:06,231][INFO ][node ] [node-2] starting ...
  15. [2017-04-04 14:10:06,403][INFO ][transport ] [node-2] publish_address {192.168.37.55:9300}, bound_addresses {127.0.1.1:9300}, {192.168.37.55:9300}
  16. [2017-04-04 14:10:06,408][INFO ][discovery ] [node-2] GRAY-graylog/VqPRyepFShaGl3rn3DRZ9A
  17. [2017-04-04 14:10:11,572][INFO ][cluster.service ] [node-2] detected_master {node-1}{f8NgS62qTl6WBiO_CNXtZQ}{192.168.37.52}{192.168.37.52:9300}, added {{node-1}{f8NgS62qTl6WBiO_CNXtZQ}{192.168.37.52}{192.168.37.52:9300},}, reason: zen-disco-receive(from master [{node-1}{f8NgS62qTl6WBiO_CNXtZQ}{192.168.37.52}{192.168.37.52:9300}])
  18. [2017-04-04 14:10:13,951][INFO ][http ] [node-2] publish_address {192.168.37.55:9200}, bound_addresses {127.0.1.1:9200}, {192.168.37.55:9200}
  19. [2017-04-04 14:10:13,951][INFO ][node ] [node-2] started
  20. [2017-04-04 14:10:29,438][INFO ][cluster.service ] [node-2] added {{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{MlFlkzUDRQuKR6cuWXi5oQ}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false},{GRAY-73692d2f-f8ff-4c4a-be14-db8f93eeec8c}{s-IUwEO8TD24buAIGbC0Iw}{192.168.37.52}{192.168.37.52:9350}{client=true, data=false, master=false},}, reason: zen-disco-receive(from master [{node-1}{f8NgS62qTl6WBiO_CNXtZQ}{192.168.37.52}{192.168.37.52:9300}])
  21. [2017-04-04 14:42:00,045][INFO ][discovery.zen ] [node-2] master_left [{node-1}{f8NgS62qTl6WBiO_CNXtZQ}{192.168.37.52}{192.168.37.52:9300}], reason [shut_down]
  22. [2017-04-04 14:42:00,047][WARN ][discovery.zen ] [node-2] master left (reason = shut_down), current nodes: {{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{MlFlkzUDRQuKR6cuWXi5oQ}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false},{GRAY-73692d2f-f8ff-4c4a-be14-db8f93eeec8c}{s-IUwEO8TD24buAIGbC0Iw}{192.168.37.52}{192.168.37.52:9350}{client=true, data=false, master=false},{node-2}{VqPRyepFShaGl3rn3DRZ9A}{192.168.37.55}{192.168.37.55:9300},}
  23. [2017-04-04 14:42:00,049][INFO ][cluster.service ] [node-2] removed {{node-1}{f8NgS62qTl6WBiO_CNXtZQ}{192.168.37.52}{192.168.37.52:9300},}, reason: zen-disco-master_failed ({node-1}{f8NgS62qTl6WBiO_CNXtZQ}{192.168.37.52}{192.168.37.52:9300})
  24. [2017-04-04 14:42:00,066][WARN ][discovery.zen.ping.unicast] [node-2] failed to send ping to [{node-1}{f8NgS62qTl6WBiO_CNXtZQ}{192.168.37.52}{192.168.37.52:9300}]
  25. RemoteTransportException[[node-1][192.168.37.52:9300][internal:discovery/zen/unicast]]; nested: IllegalStateException[received ping request while not started];
  26. Caused by: java.lang.IllegalStateException: received ping request while not started
  27. at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.handlePingRequest(UnicastZenPing.java:497)
  28. at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.access$2400(UnicastZenPing.java:83)
  29. at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$UnicastPingRequestHandler.messageReceived(UnicastZenPing.java:522)
  30. at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$UnicastPingRequestHandler.messageReceived(UnicastZenPing.java:518)
  31. at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
  32. at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)
  33. at org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:227)
  34. at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:116)
  35. at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
  36. at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  37. at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
  38. at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
  39. at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
  40. at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
  41. at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
  42. at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
  43. at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  44. at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
  45. at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:75)
  46. at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  47. at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
  48. at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
  49. at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
  50. at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
  51. at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
  52. at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
  53. at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
  54. at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
  55. at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
  56. at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
  57. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  58. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  59. at java.lang.Thread.run(Thread.java:745)
  60. [2017-04-04 14:42:00,066][WARN ][discovery.zen.ping.unicast] [node-2] failed to send ping to [{#zen_unicast_1#}{192.168.37.52}{192.168.37.52:9300}]
  61. RemoteTransportException[[node-1][192.168.37.52:9300][internal:discovery/zen/unicast]]; nested: IllegalStateException[received ping request while not started];
  62. Caused by: java.lang.IllegalStateException: received ping request while not started
  63. at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.handlePingRequest(UnicastZenPing.java:497)
  64. at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.access$2400(UnicastZenPing.java:83)
  65. at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$UnicastPingRequestHandler.messageReceived(UnicastZenPing.java:522)
  66. at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$UnicastPingRequestHandler.messageReceived(UnicastZenPing.java:518)
  67. at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
  68. at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)
  69. at org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:227)
  70. at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:116)
  71. at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
  72. at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  73. at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
  74. at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
  75. at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
  76. at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
  77. at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
  78. at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
  79. at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  80. at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
  81. at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:75)
  82. at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  83. at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
  84. at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
  85. at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
  86. at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
  87. at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
  88. at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
  89. at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
  90. at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
  91. at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
  92. at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
  93. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  94. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  95. at java.lang.Thread.run(Thread.java:745)
  96. [2017-04-04 14:42:00,878][DEBUG][action.admin.cluster.health] [node-2] no known master node, scheduling a retry
  97. [2017-04-04 14:42:03,113][INFO ][cluster.routing.allocation] [node-2] Cluster health status changed from [GREEN] to [RED] (reason: [nodes joined]).
  98. [2017-04-04 14:42:03,114][INFO ][cluster.service ] [node-2] new_master {node-2}{VqPRyepFShaGl3rn3DRZ9A}{192.168.37.55}{192.168.37.55:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
  99. [2017-04-04 14:42:03,158][DEBUG][action.admin.cluster.health] [node-2] no known master node, scheduling a retry
  100. [2017-04-04 14:42:03,159][DEBUG][action.admin.cluster.health] [node-2] no known master node, scheduling a retry
  101. [2017-04-04 14:42:03,161][INFO ][cluster.routing ] [node-2] delaying allocation for [190] unassigned shards, next check in [1m]
  102. [2017-04-04 14:42:16,805][INFO ][cluster.service ] [node-2] added {{node-1}{3y8qctOdQEiESue_fB69-Q}{192.168.37.52}{192.168.37.52:9300},}, reason: zen-disco-join(join from node[{node-1}{3y8qctOdQEiESue_fB69-Q}{192.168.37.52}{192.168.37.52:9300}])
  103. [2017-04-04 14:42:19,885][INFO ][cluster.routing.allocation] [node-2] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[graylog_0][3], [graylog_0][3]] ...]).
  104. [2017-04-04 14:43:26,635][INFO ][cluster.routing.allocation] [node-2] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[session_login.cgi][1]] ...]).
  105. [2017-04-04 17:31:54,279][WARN ][transport ] [node-2] Received response for a request that has timed out, sent [36211ms] ago, timed out [6211ms] ago, action [internal:discovery/zen/fd/ping], node [{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{MlFlkzUDRQuKR6cuWXi5oQ}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false}], id [35492]
  106. [2017-04-04 17:32:57,850][WARN ][transport ] [node-2] Received response for a request that has timed out, sent [40816ms] ago, timed out [10816ms] ago, action [internal:discovery/zen/fd/ping], node [{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{MlFlkzUDRQuKR6cuWXi5oQ}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false}], id [35574]
  107. [2017-04-04 17:33:35,408][WARN ][transport ] [node-2] Received response for a request that has timed out, sent [32917ms] ago, timed out [2917ms] ago, action [internal:discovery/zen/fd/ping], node [{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{MlFlkzUDRQuKR6cuWXi5oQ}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false}], id [35638]
  108. [2017-04-04 17:36:30,961][WARN ][transport ] [node-2] Received response for a request that has timed out, sent [52633ms] ago, timed out [22633ms] ago, action [internal:discovery/zen/fd/ping], node [{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{MlFlkzUDRQuKR6cuWXi5oQ}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false}], id [35841]
  109. [2017-04-04 17:37:26,480][WARN ][transport ] [node-2] Received response for a request that has timed out, sent [30101ms] ago, timed out [100ms] ago, action [internal:discovery/zen/fd/ping], node [{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{MlFlkzUDRQuKR6cuWXi5oQ}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false}], id [35943]
  110. [2017-04-04 17:38:26,387][INFO ][cluster.service ] [node-2] removed {{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{MlFlkzUDRQuKR6cuWXi5oQ}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false},}, reason: zen-disco-node-failed({GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{MlFlkzUDRQuKR6cuWXi5oQ}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false}), reason(failed to ping, tried [3] times, each with maximum [30s] timeout)
  111. [2017-04-04 17:38:42,415][WARN ][transport ] [node-2] Received response for a request that has timed out, sent [42102ms] ago, timed out [12102ms] ago, action [internal:discovery/zen/fd/ping], node [{GRAY-73692d2f-f8ff-4c4a-be14-db8f93eeec8c}{s-IUwEO8TD24buAIGbC0Iw}{192.168.37.52}{192.168.37.52:9350}{client=true, data=false, master=false}], id [36022]
  112. [2017-04-04 17:38:44,545][WARN ][transport ] [node-2] Received response for a request that has timed out, sent [78166ms] ago, timed out [48166ms] ago, action [internal:discovery/zen/fd/ping], node [{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{MlFlkzUDRQuKR6cuWXi5oQ}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false}], id [35981]
  113. [2017-04-04 17:40:22,772][WARN ][transport ] [node-2] Received response for a request that has timed out, sent [51180ms] ago, timed out [21180ms] ago, action [internal:discovery/zen/fd/ping], node [{GRAY-73692d2f-f8ff-4c4a-be14-db8f93eeec8c}{s-IUwEO8TD24buAIGbC0Iw}{192.168.37.52}{192.168.37.52:9350}{client=true, data=false, master=false}], id [36136]
  114. [2017-04-04 17:42:20,787][WARN ][transport ] [node-2] Received response for a request that has timed out, sent [33780ms] ago, timed out [3780ms] ago, action [internal:discovery/zen/fd/ping], node [{GRAY-73692d2f-f8ff-4c4a-be14-db8f93eeec8c}{s-IUwEO8TD24buAIGbC0Iw}{192.168.37.52}{192.168.37.52:9350}{client=true, data=false, master=false}], id [36296]
  115. [2017-04-04 17:43:05,635][WARN ][transport ] [node-2] Received response for a request that has timed out, sent [32852ms] ago, timed out [2852ms] ago, action [internal:discovery/zen/fd/ping], node [{GRAY-73692d2f-f8ff-4c4a-be14-db8f93eeec8c}{s-IUwEO8TD24buAIGbC0Iw}{192.168.37.52}{192.168.37.52:9350}{client=true, data=false, master=false}], id [36352]
  116. [2017-04-04 17:44:19,087][INFO ][cluster.service ] [node-2] added {{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{MlFlkzUDRQuKR6cuWXi5oQ}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false},}, reason: zen-disco-join(join from node[{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{MlFlkzUDRQuKR6cuWXi5oQ}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false}])
  117. [2017-04-04 17:44:49,091][WARN ][discovery.zen.publish ] [node-2] timed out waiting for all nodes to process published state [367] (timeout [30s], pending nodes: [{GRAY-73692d2f-f8ff-4c4a-be14-db8f93eeec8c}{s-IUwEO8TD24buAIGbC0Iw}{192.168.37.52}{192.168.37.52:9350}{client=true, data=false, master=false}])
  118. [2017-04-04 17:44:49,093][WARN ][cluster.service ] [node-2] cluster state update task [zen-disco-join(join from node[{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{MlFlkzUDRQuKR6cuWXi5oQ}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false}])] took 30s above the warn threshold of 30s
  119. [2017-04-04 17:45:05,858][WARN ][transport ] [node-2] Received response for a request that has timed out, sent [37025ms] ago, timed out [7025ms] ago, action [internal:discovery/zen/fd/ping], node [{GRAY-73692d2f-f8ff-4c4a-be14-db8f93eeec8c}{s-IUwEO8TD24buAIGbC0Iw}{192.168.37.52}{192.168.37.52:9350}{client=true, data=false, master=false}], id [36501]
  120. [2017-04-04 17:46:02,841][WARN ][transport ] [node-2] Received response for a request that has timed out, sent [55981ms] ago, timed out [25981ms] ago, action [internal:discovery/zen/fd/ping], node [{GRAY-73692d2f-f8ff-4c4a-be14-db8f93eeec8c}{s-IUwEO8TD24buAIGbC0Iw}{192.168.37.52}{192.168.37.52:9350}{client=true, data=false, master=false}], id [36553]
  121. [2017-04-04 17:48:05,524][INFO ][cluster.service ] [node-2] removed {{GRAY-73692d2f-f8ff-4c4a-be14-db8f93eeec8c}{s-IUwEO8TD24buAIGbC0Iw}{192.168.37.52}{192.168.37.52:9350}{client=true, data=false, master=false},}, reason: zen-disco-node-failed({GRAY-73692d2f-f8ff-4c4a-be14-db8f93eeec8c}{s-IUwEO8TD24buAIGbC0Iw}{192.168.37.52}{192.168.37.52:9350}{client=true, data=false, master=false}), reason(failed to ping, tried [3] times, each with maximum [30s] timeout)
  122. [2017-04-04 17:49:32,381][INFO ][cluster.routing.allocation] [node-2] Cluster health status changed from [GREEN] to [RED] (reason: [{node-1}{3y8qctOdQEiESue_fB69-Q}{192.168.37.52}{192.168.37.52:9300} left]).
  123. [2017-04-04 17:49:32,381][INFO ][cluster.service ] [node-2] removed {{node-1}{3y8qctOdQEiESue_fB69-Q}{192.168.37.52}{192.168.37.52:9300},}, reason: zen-disco-node-left({node-1}{3y8qctOdQEiESue_fB69-Q}{192.168.37.52}{192.168.37.52:9300}), reason(left)
  124. [2017-04-04 17:49:37,857][INFO ][cluster.routing ] [node-2] delaying allocation for [190] unassigned shards, next check in [1m]
  125. [2017-04-04 17:50:49,549][INFO ][node ] [node-2] stopping ...
  126. [2017-04-04 17:50:49,943][INFO ][node ] [node-2] stopped
  127. [2017-04-04 17:50:49,943][INFO ][node ] [node-2] closing ...
  128. [2017-04-04 17:50:49,957][INFO ][node ] [node-2] closed
  129. [2017-04-04 17:54:01,797][INFO ][node ] [node-2] version[2.4.4], pid[1405], build[fcbb46d/2017-01-03T11:33:16Z]
  130. [2017-04-04 17:54:01,803][INFO ][node ] [node-2] initializing ...
  131. [2017-04-04 17:54:02,615][INFO ][plugins ] [node-2] modules [reindex, lang-expression, lang-groovy], plugins [], sites []
  132. [2017-04-04 17:54:02,666][INFO ][env ] [node-2] using [1] data paths, mounts [[/ (/dev/mapper/GraySrv2--vg-root)]], net usable_space [2.4tb], net total_space [2.6tb], spins? [possibly], types [ext4]
  133. [2017-04-04 17:54:02,666][INFO ][env ] [node-2] heap size [989.8mb], compressed ordinary object pointers [true]
  134. [2017-04-04 17:54:05,206][INFO ][node ] [node-2] initialized
  135. [2017-04-04 17:54:05,206][INFO ][node ] [node-2] starting ...
  136. [2017-04-04 17:54:05,343][INFO ][transport ] [node-2] publish_address {192.168.37.55:9300}, bound_addresses {127.0.1.1:9300}, {192.168.37.55:9300}
  137. [2017-04-04 17:54:05,349][INFO ][discovery ] [node-2] GRAY-graylog/_FSOp31nSYyHuMv7CVRjKw
  138. [2017-04-04 17:54:09,856][INFO ][cluster.service ] [node-2] detected_master {node-1}{xpw72ZZLR52uJCVXHwqI4A}{192.168.37.52}{192.168.37.52:9300}, added {{node-1}{xpw72ZZLR52uJCVXHwqI4A}{192.168.37.52}{192.168.37.52:9300},}, reason: zen-disco-receive(from master [{node-1}{xpw72ZZLR52uJCVXHwqI4A}{192.168.37.52}{192.168.37.52:9300}])
  139. [2017-04-04 17:54:12,078][INFO ][http ] [node-2] publish_address {192.168.37.55:9200}, bound_addresses {127.0.1.1:9200}, {192.168.37.55:9200}
  140. [2017-04-04 17:54:12,079][INFO ][node ] [node-2] started
  141. [2017-04-04 17:54:41,782][INFO ][cluster.service ] [node-2] added {{GRAY-73692d2f-f8ff-4c4a-be14-db8f93eeec8c}{O9lc8rQvRgWFu-CkCoSTew}{192.168.37.52}{192.168.37.52:9350}{client=true, data=false, master=false},{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{Kux56kn6SD6QDZ8eYjYkJg}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false},}, reason: zen-disco-receive(from master [{node-1}{xpw72ZZLR52uJCVXHwqI4A}{192.168.37.52}{192.168.37.52:9300}])
  142. [2017-04-04 18:06:49,229][INFO ][cluster.service ] [node-2] removed {{GRAY-73692d2f-f8ff-4c4a-be14-db8f93eeec8c}{O9lc8rQvRgWFu-CkCoSTew}{192.168.37.52}{192.168.37.52:9350}{client=true, data=false, master=false},}, reason: zen-disco-receive(from master [{node-1}{xpw72ZZLR52uJCVXHwqI4A}{192.168.37.52}{192.168.37.52:9300}])
  143. [2017-04-04 18:09:37,371][INFO ][cluster.service ] [node-2] added {{GRAY-73692d2f-f8ff-4c4a-be14-db8f93eeec8c}{O9lc8rQvRgWFu-CkCoSTew}{192.168.37.52}{192.168.37.52:9350}{client=true, data=false, master=false},}, reason: zen-disco-receive(from master [{node-1}{xpw72ZZLR52uJCVXHwqI4A}{192.168.37.52}{192.168.37.52:9300}])
  144. [2017-04-04 18:11:08,194][INFO ][cluster.service ] [node-2] removed {{GRAY-73692d2f-f8ff-4c4a-be14-db8f93eeec8c}{O9lc8rQvRgWFu-CkCoSTew}{192.168.37.52}{192.168.37.52:9350}{client=true, data=false, master=false},}, reason: zen-disco-receive(from master [{node-1}{xpw72ZZLR52uJCVXHwqI4A}{192.168.37.52}{192.168.37.52:9300}])
  145. [2017-04-04 18:11:32,078][INFO ][cluster.service ] [node-2] removed {{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{Kux56kn6SD6QDZ8eYjYkJg}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false},}, reason: zen-disco-receive(from master [{node-1}{xpw72ZZLR52uJCVXHwqI4A}{192.168.37.52}{192.168.37.52:9300}])
  146. [2017-04-04 18:11:38,543][INFO ][cluster.service ] [node-2] added {{GRAY-73692d2f-f8ff-4c4a-be14-db8f93eeec8c}{4_4RjayeSvyQ0Io8fh-4lA}{192.168.37.52}{192.168.37.52:9350}{client=true, data=false, master=false},}, reason: zen-disco-receive(from master [{node-1}{xpw72ZZLR52uJCVXHwqI4A}{192.168.37.52}{192.168.37.52:9300}])
  147. [2017-04-04 18:11:49,178][INFO ][cluster.service ] [node-2] added {{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{1fs2Ww6DQWKILFMdZmjcfA}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false},}, reason: zen-disco-receive(from master [{node-1}{xpw72ZZLR52uJCVXHwqI4A}{192.168.37.52}{192.168.37.52:9300}])
  148. [2017-04-04 18:21:31,438][INFO ][cluster.service ] [node-2] removed {{GRAY-73692d2f-f8ff-4c4a-be14-db8f93eeec8c}{4_4RjayeSvyQ0Io8fh-4lA}{192.168.37.52}{192.168.37.52:9350}{client=true, data=false, master=false},}, reason: zen-disco-receive(from master [{node-1}{xpw72ZZLR52uJCVXHwqI4A}{192.168.37.52}{192.168.37.52:9300}])
  149. [2017-04-04 18:21:53,333][INFO ][cluster.service ] [node-2] added {{GRAY-73692d2f-f8ff-4c4a-be14-db8f93eeec8c}{ia00q_BAQtC08vICSg6_Gw}{192.168.37.52}{192.168.37.52:9350}{client=true, data=false, master=false},}, reason: zen-disco-receive(from master [{node-1}{xpw72ZZLR52uJCVXHwqI4A}{192.168.37.52}{192.168.37.52:9300}])
  150. [2017-04-04 18:24:02,177][INFO ][cluster.service ] [node-2] removed {{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{1fs2Ww6DQWKILFMdZmjcfA}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false},}, reason: zen-disco-receive(from master [{node-1}{xpw72ZZLR52uJCVXHwqI4A}{192.168.37.52}{192.168.37.52:9300}])
  151. [2017-04-04 18:24:34,976][INFO ][cluster.service ] [node-2] added {{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{eu9-gknHSfOE1LYBbWcFAw}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false},}, reason: zen-disco-receive(from master [{node-1}{xpw72ZZLR52uJCVXHwqI4A}{192.168.37.52}{192.168.37.52:9300}])
  152. [2017-04-04 18:36:02,503][INFO ][discovery.zen ] [node-2] master_left [{node-1}{xpw72ZZLR52uJCVXHwqI4A}{192.168.37.52}{192.168.37.52:9300}], reason [shut_down]
  153. [2017-04-04 18:36:02,504][WARN ][discovery.zen ] [node-2] master left (reason = shut_down), current nodes: {{GRAY-73692d2f-f8ff-4c4a-be14-db8f93eeec8c}{ia00q_BAQtC08vICSg6_Gw}{192.168.37.52}{192.168.37.52:9350}{client=true, data=false, master=false},{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{eu9-gknHSfOE1LYBbWcFAw}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false},{node-2}{_FSOp31nSYyHuMv7CVRjKw}{192.168.37.55}{192.168.37.55:9300},}
  154. [2017-04-04 18:36:02,505][INFO ][cluster.service ] [node-2] removed {{node-1}{xpw72ZZLR52uJCVXHwqI4A}{192.168.37.52}{192.168.37.52:9300},}, reason: zen-disco-master_failed ({node-1}{xpw72ZZLR52uJCVXHwqI4A}{192.168.37.52}{192.168.37.52:9300})
  155. [2017-04-04 18:36:02,510][WARN ][discovery.zen.ping.unicast] [node-2] failed to send ping to [{node-1}{xpw72ZZLR52uJCVXHwqI4A}{192.168.37.52}{192.168.37.52:9300}]
  156. RemoteTransportException[[node-1][192.168.37.52:9300][internal:discovery/zen/unicast]]; nested: IllegalStateException[received ping request while not started];
  157. Caused by: java.lang.IllegalStateException: received ping request while not started
  158. at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.handlePingRequest(UnicastZenPing.java:497)
  159. at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.access$2400(UnicastZenPing.java:83)
  160. at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$UnicastPingRequestHandler.messageReceived(UnicastZenPing.java:522)
  161. at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$UnicastPingRequestHandler.messageReceived(UnicastZenPing.java:518)
  162. at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
  163. at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)
  164. at org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:227)
  165. at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:116)
  166. at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
  167. at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  168. at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
  169. at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
  170. at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
  171. at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
  172. at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
  173. at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
  174. at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  175. at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
  176. at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:75)
  177. at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  178. at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
  179. at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
  180. at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
  181. at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
  182. at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
  183. at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
  184. at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
  185. at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
  186. at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
  187. at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
  188. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  189. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  190. at java.lang.Thread.run(Thread.java:745)
  191. [2017-04-04 18:36:02,510][WARN ][discovery.zen.ping.unicast] [node-2] failed to send ping to [{#zen_unicast_1#}{192.168.37.52}{192.168.37.52:9300}]
  192. RemoteTransportException[[node-1][192.168.37.52:9300][internal:discovery/zen/unicast]]; nested: IllegalStateException[received ping request while not started];
  193. Caused by: java.lang.IllegalStateException: received ping request while not started
  194. at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.handlePingRequest(UnicastZenPing.java:497)
  195. at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.access$2400(UnicastZenPing.java:83)
  196. at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$UnicastPingRequestHandler.messageReceived(UnicastZenPing.java:522)
  197. at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$UnicastPingRequestHandler.messageReceived(UnicastZenPing.java:518)
  198. at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
  199. at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)
  200. at org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:227)
  201. at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:116)
  202. at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
  203. at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  204. at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
  205. at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
  206. at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
  207. at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
  208. at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
  209. at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
  210. at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  211. at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
  212. at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:75)
  213. at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  214. at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
  215. at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
  216. at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
  217. at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
  218. at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
  219. at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
  220. at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
  221. at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
  222. at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
  223. at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
  224. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  225. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  226. at java.lang.Thread.run(Thread.java:745)
  227. [2017-04-04 18:36:03,719][DEBUG][action.admin.cluster.health] [node-2] no known master node, scheduling a retry
  228. [2017-04-04 18:36:04,369][DEBUG][action.admin.cluster.health] [node-2] no known master node, scheduling a retry
  229. [2017-04-04 18:36:07,052][INFO ][cluster.routing.allocation] [node-2] Cluster health status changed from [GREEN] to [RED] (reason: [nodes joined]).
  230. [2017-04-04 18:36:07,054][INFO ][cluster.service ] [node-2] new_master {node-2}{_FSOp31nSYyHuMv7CVRjKw}{192.168.37.55}{192.168.37.55:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
  231. [2017-04-04 18:36:07,199][INFO ][node ] [node-2] stopping ...
  232. [2017-04-04 18:36:07,276][WARN ][rest.suppressed ] path: /_cat/health, params: {}
  233. NodeClosedException[node closed {node-2}{_FSOp31nSYyHuMv7CVRjKw}{192.168.37.55}{192.168.37.55:9300}]
  234. at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$5.onClusterServiceClose(TransportMasterNodeAction.java:228)
  235. at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onClose(ClusterStateObserver.java:222)
  236. at org.elasticsearch.cluster.service.InternalClusterService.doStop(InternalClusterService.java:216)
  237. at org.elasticsearch.common.component.AbstractLifecycleComponent.stop(AbstractLifecycleComponent.java:88)
  238. at org.elasticsearch.node.Node.stop(Node.java:326)
  239. at org.elasticsearch.node.Node.close(Node.java:351)
  240. at org.elasticsearch.bootstrap.Bootstrap$4.run(Bootstrap.java:173)
  241. [2017-04-04 18:36:07,276][WARN ][rest.suppressed ] path: /_cat/health, params: {}
  242. NodeClosedException[node closed {node-2}{_FSOp31nSYyHuMv7CVRjKw}{192.168.37.55}{192.168.37.55:9300}]
  243. at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$5.onClusterServiceClose(TransportMasterNodeAction.java:228)
  244. at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onClose(ClusterStateObserver.java:222)
  245. at org.elasticsearch.cluster.service.InternalClusterService.doStop(InternalClusterService.java:216)
  246. at org.elasticsearch.common.component.AbstractLifecycleComponent.stop(AbstractLifecycleComponent.java:88)
  247. at org.elasticsearch.node.Node.stop(Node.java:326)
  248. at org.elasticsearch.node.Node.close(Node.java:351)
  249. at org.elasticsearch.bootstrap.Bootstrap$4.run(Bootstrap.java:173)
  250. [2017-04-04 18:36:07,293][WARN ][netty.channel.DefaultChannelPipeline] An exception was thrown by an exception handler.
  251. java.util.concurrent.RejectedExecutionException: Worker has already been shutdown
  252. at org.jboss.netty.channel.socket.nio.AbstractNioSelector.registerTask(AbstractNioSelector.java:120)
  253. at org.jboss.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:72)
  254. at org.jboss.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)
  255. at org.jboss.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:56)
  256. at org.jboss.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)
  257. at org.jboss.netty.channel.socket.nio.AbstractNioChannelSink.execute(AbstractNioChannelSink.java:34)
  258. at org.jboss.netty.channel.DefaultChannelPipeline.execute(DefaultChannelPipeline.java:636)
  259. at org.jboss.netty.channel.Channels.fireExceptionCaughtLater(Channels.java:496)
  260. at org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:46)
  261. at org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:658)
  262. at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:781)
  263. at org.jboss.netty.channel.Channels.write(Channels.java:725)
  264. at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:71)
  265. at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
  266. at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
  267. at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784)
  268. at org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.handleDownstream(HttpPipeliningHandler.java:87)
  269. at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
  270. at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
  271. at org.elasticsearch.http.netty.NettyHttpChannel.sendResponse(NettyHttpChannel.java:147)
  272. at org.elasticsearch.http.HttpServer$ResourceHandlingHttpChannel.sendResponse(HttpServer.java:352)
  273. at org.elasticsearch.rest.action.support.RestActionListener.onFailure(RestActionListener.java:60)
  274. at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:95)
  275. at org.elasticsearch.action.support.ThreadedActionListener$2.doRun(ThreadedActionListener.java:104)
  276. at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
  277. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  278. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  279. at java.lang.Thread.run(Thread.java:745)
  280. [2017-04-04 18:36:07,293][WARN ][netty.channel.DefaultChannelPipeline] An exception was thrown by an exception handler.
  281. java.util.concurrent.RejectedExecutionException: Worker has already been shutdown
  282. at org.jboss.netty.channel.socket.nio.AbstractNioSelector.registerTask(AbstractNioSelector.java:120)
  283. at org.jboss.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:72)
  284. at org.jboss.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)
  285. at org.jboss.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:56)
  286. at org.jboss.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)
  287. at org.jboss.netty.channel.socket.nio.AbstractNioChannelSink.execute(AbstractNioChannelSink.java:34)
  288. at org.jboss.netty.channel.DefaultChannelPipeline.execute(DefaultChannelPipeline.java:636)
  289. at org.jboss.netty.channel.Channels.fireExceptionCaughtLater(Channels.java:496)
  290. at org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:46)
  291. at org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:658)
  292. at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:781)
  293. at org.jboss.netty.channel.Channels.write(Channels.java:725)
  294. at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:71)
  295. at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
  296. at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
  297. at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784)
  298. at org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.handleDownstream(HttpPipeliningHandler.java:87)
  299. at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
  300. at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
  301. at org.elasticsearch.http.netty.NettyHttpChannel.sendResponse(NettyHttpChannel.java:147)
  302. at org.elasticsearch.http.HttpServer$ResourceHandlingHttpChannel.sendResponse(HttpServer.java:352)
  303. at org.elasticsearch.rest.action.support.RestActionListener.onFailure(RestActionListener.java:60)
  304. at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:95)
  305. at org.elasticsearch.action.support.ThreadedActionListener$2.doRun(ThreadedActionListener.java:104)
  306. at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
  307. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  308. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  309. at java.lang.Thread.run(Thread.java:745)
  310. [2017-04-04 18:36:07,826][WARN ][discovery.zen.ping.unicast] [node-2] received ping response ping_response{node [{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{eu9-gknHSfOE1LYBbWcFAw}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false}], id[6], master [null], hasJoinedOnce [true], cluster_name[GRAY-graylog]} with no matching handler id [2]
  311. [2017-04-04 18:36:07,918][DEBUG][action.admin.cluster.health] [node-2] no known master node, scheduling a retry
  312. [2017-04-04 18:36:07,938][DEBUG][action.admin.cluster.health] [node-2] no known master node, scheduling a retry
  313. [2017-04-04 18:36:07,959][DEBUG][action.admin.cluster.health] [node-2] no known master node, scheduling a retry
  314. [2017-04-04 18:36:07,976][DEBUG][action.admin.cluster.health] [node-2] no known master node, scheduling a retry
  315. [2017-04-04 18:36:08,008][DEBUG][action.admin.cluster.health] [node-2] no known master node, scheduling a retry
  316. [2017-04-04 18:36:08,028][DEBUG][action.admin.cluster.health] [node-2] no known master node, scheduling a retry
  317. [2017-04-04 18:36:08,036][DEBUG][action.admin.cluster.health] [node-2] no known master node, scheduling a retry
  318. [2017-04-04 18:36:08,049][DEBUG][action.admin.cluster.health] [node-2] no known master node, scheduling a retry
  319. [2017-04-04 18:36:08,493][DEBUG][action.admin.cluster.health] [node-2] no known master node, scheduling a retry
  320. [2017-04-04 18:36:12,095][DEBUG][action.admin.cluster.health] [node-2] no known master node, scheduling a retry
  321. [2017-04-04 18:36:12,140][DEBUG][action.admin.cluster.health] [node-2] no known master node, scheduling a retry
  322. [2017-04-04 18:36:12,146][DEBUG][action.admin.cluster.health] [node-2] no known master node, scheduling a retry
  323. [2017-04-04 18:36:12,493][DEBUG][action.admin.cluster.health] [node-2] no known master node, scheduling a retry
  324. [2017-04-04 18:36:16,486][DEBUG][action.admin.cluster.health] [node-2] no known master node, scheduling a retry
  325. [2017-04-04 18:36:16,496][DEBUG][action.admin.cluster.health] [node-2] no known master node, scheduling a retry
  326. [2017-04-04 18:36:16,524][DEBUG][action.admin.cluster.health] [node-2] no known master node, scheduling a retry
  327. [2017-04-04 18:36:16,529][DEBUG][action.admin.cluster.health] [node-2] no known master node, scheduling a retry
  328. [2017-04-04 18:36:17,283][INFO ][cluster.routing ] [node-2] delaying allocation for [190] unassigned shards, next check in [1m]
  329. [2017-04-04 18:36:17,657][INFO ][node ] [node-2] stopped
  330. [2017-04-04 18:36:17,657][INFO ][node ] [node-2] closing ...
  331. [2017-04-04 18:36:17,673][INFO ][node ] [node-2] closed
  332. [2017-04-04 18:39:06,425][INFO ][node ] [node-2] version[2.4.4], pid[1365], build[fcbb46d/2017-01-03T11:33:16Z]
  333. [2017-04-04 18:39:06,442][INFO ][node ] [node-2] initializing ...
  334. [2017-04-04 18:39:07,324][INFO ][plugins ] [node-2] modules [reindex, lang-expression, lang-groovy], plugins [], sites []
  335. [2017-04-04 18:39:07,369][INFO ][env ] [node-2] using [1] data paths, mounts [[/ (/dev/mapper/GraySrv2--vg-root)]], net usable_space [2.4tb], net total_space [2.6tb], spins? [possibly], types [ext4]
  336. [2017-04-04 18:39:07,369][INFO ][env ] [node-2] heap size [989.8mb], compressed ordinary object pointers [true]
  337. [2017-04-04 18:39:10,170][INFO ][node ] [node-2] initialized
  338. [2017-04-04 18:39:10,170][INFO ][node ] [node-2] starting ...
  339. [2017-04-04 18:39:10,337][INFO ][transport ] [node-2] publish_address {192.168.37.55:9300}, bound_addresses {127.0.1.1:9300}, {192.168.37.55:9300}
  340. [2017-04-04 18:39:10,343][INFO ][discovery ] [node-2] GRAY-graylog/4SqFx2UaSPq08OCoV6exQQ
  341. [2017-04-04 18:39:16,063][INFO ][cluster.service ] [node-2] detected_master {node-1}{1APpnCwuRNaFyAd7u6V2Fg}{192.168.37.52}{192.168.37.52:9300}, added {{node-1}{1APpnCwuRNaFyAd7u6V2Fg}{192.168.37.52}{192.168.37.52:9300},}, reason: zen-disco-receive(from master [{node-1}{1APpnCwuRNaFyAd7u6V2Fg}{192.168.37.52}{192.168.37.52:9300}])
  342. [2017-04-04 18:39:16,184][INFO ][http ] [node-2] publish_address {192.168.37.55:9200}, bound_addresses {127.0.1.1:9200}, {192.168.37.55:9200}
  343. [2017-04-04 18:39:16,184][INFO ][node ] [node-2] started
  344. [2017-04-04 18:39:38,023][INFO ][cluster.service ] [node-2] added {{GRAY-73692d2f-f8ff-4c4a-be14-db8f93eeec8c}{0KWnX6rSRsa8jGj6VsaR_g}{192.168.37.52}{192.168.37.52:9350}{client=true, data=false, master=false},{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{4q64_6xmTxKQPyBR055blQ}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false},}, reason: zen-disco-receive(from master [{node-1}{1APpnCwuRNaFyAd7u6V2Fg}{192.168.37.52}{192.168.37.52:9300}])
  345. [2017-04-04 18:58:00,604][INFO ][cluster.service ] [node-2] removed {{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{4q64_6xmTxKQPyBR055blQ}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false},}, reason: zen-disco-receive(from master [{node-1}{1APpnCwuRNaFyAd7u6V2Fg}{192.168.37.52}{192.168.37.52:9300}])
  346. [2017-04-04 19:04:28,582][INFO ][cluster.service ] [node-2] removed {{GRAY-73692d2f-f8ff-4c4a-be14-db8f93eeec8c}{0KWnX6rSRsa8jGj6VsaR_g}{192.168.37.52}{192.168.37.52:9350}{client=true, data=false, master=false},}, reason: zen-disco-receive(from master [{node-1}{1APpnCwuRNaFyAd7u6V2Fg}{192.168.37.52}{192.168.37.52:9300}])
  347. [2017-04-04 19:04:45,640][INFO ][cluster.service ] [node-2] added {{GRAY-73692d2f-f8ff-4c4a-be14-db8f93eeec8c}{S-aP1kdwSL6Os86sSneocg}{192.168.37.52}{192.168.37.52:9350}{client=true, data=false, master=false},}, reason: zen-disco-receive(from master [{node-1}{1APpnCwuRNaFyAd7u6V2Fg}{192.168.37.52}{192.168.37.52:9300}])
  348. [2017-04-04 19:16:54,824][INFO ][cluster.service ] [node-2] removed {{GRAY-73692d2f-f8ff-4c4a-be14-db8f93eeec8c}{S-aP1kdwSL6Os86sSneocg}{192.168.37.52}{192.168.37.52:9350}{client=true, data=false, master=false},}, reason: zen-disco-receive(from master [{node-1}{1APpnCwuRNaFyAd7u6V2Fg}{192.168.37.52}{192.168.37.52:9300}])
  349. [2017-04-04 19:17:11,876][INFO ][cluster.service ] [node-2] added {{GRAY-73692d2f-f8ff-4c4a-be14-db8f93eeec8c}{h_ut2z4RRVqqagVEkaNmng}{192.168.37.52}{192.168.37.52:9350}{client=true, data=false, master=false},}, reason: zen-disco-receive(from master [{node-1}{1APpnCwuRNaFyAd7u6V2Fg}{192.168.37.52}{192.168.37.52:9300}])
  350. [2017-04-04 19:17:21,271][INFO ][cluster.service ] [node-2] added {{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{hg-GPtPdTziQIkZOZ3fgEg}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false},}, reason: zen-disco-receive(from master [{node-1}{1APpnCwuRNaFyAd7u6V2Fg}{192.168.37.52}{192.168.37.52:9300}])
  351. [2017-04-04 19:29:46,306][INFO ][cluster.service ] [node-2] removed {{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{hg-GPtPdTziQIkZOZ3fgEg}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false},}, reason: zen-disco-receive(from master [{node-1}{1APpnCwuRNaFyAd7u6V2Fg}{192.168.37.52}{192.168.37.52:9300}])
  352. [2017-04-04 19:30:21,231][INFO ][cluster.service ] [node-2] removed {{GRAY-73692d2f-f8ff-4c4a-be14-db8f93eeec8c}{h_ut2z4RRVqqagVEkaNmng}{192.168.37.52}{192.168.37.52:9350}{client=true, data=false, master=false},}, reason: zen-disco-receive(from master [{node-1}{1APpnCwuRNaFyAd7u6V2Fg}{192.168.37.52}{192.168.37.52:9300}])
  353. [2017-04-04 19:31:50,919][INFO ][cluster.service ] [node-2] added {{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{yMzeark-RsGElqanaQvrMw}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false},}, reason: zen-disco-receive(from master [{node-1}{1APpnCwuRNaFyAd7u6V2Fg}{192.168.37.52}{192.168.37.52:9300}])
  354. [2017-04-04 19:40:03,043][INFO ][node ] [node-2] stopping ...
  355. [2017-04-04 19:40:03,222][INFO ][node ] [node-2] stopped
  356. [2017-04-04 19:40:03,222][INFO ][node ] [node-2] closing ...
  357. [2017-04-04 19:40:03,234][INFO ][node ] [node-2] closed
  358. [2017-04-04 19:40:04,089][INFO ][node ] [node-2] version[2.4.4], pid[15990], build[fcbb46d/2017-01-03T11:33:16Z]
  359. [2017-04-04 19:40:04,090][INFO ][node ] [node-2] initializing ...
  360. [2017-04-04 19:40:04,785][INFO ][plugins ] [node-2] modules [reindex, lang-expression, lang-groovy], plugins [], sites []
  361. [2017-04-04 19:40:04,809][INFO ][env ] [node-2] using [1] data paths, mounts [[/ (/dev/mapper/GraySrv2--vg-root)]], net usable_space [2.4tb], net total_space [2.6tb], spins? [possibly], types [ext4]
  362. [2017-04-04 19:40:04,809][INFO ][env ] [node-2] heap size [989.8mb], compressed ordinary object pointers [true]
  363. [2017-04-04 19:40:06,768][INFO ][node ] [node-2] initialized
  364. [2017-04-04 19:40:06,769][INFO ][node ] [node-2] starting ...
  365. [2017-04-04 19:40:06,882][INFO ][transport ] [node-2] publish_address {192.168.37.55:9300}, bound_addresses {127.0.1.1:9300}, {192.168.37.55:9300}
  366. [2017-04-04 19:40:06,887][INFO ][discovery ] [node-2] GRAY-graylog/6iC4dCfRRau-PJ5y7yNkcg
  367. [2017-04-04 19:40:13,497][INFO ][cluster.service ] [node-2] detected_master {node-1}{1APpnCwuRNaFyAd7u6V2Fg}{192.168.37.52}{192.168.37.52:9300}, added {{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{yMzeark-RsGElqanaQvrMw}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false},{node-1}{1APpnCwuRNaFyAd7u6V2Fg}{192.168.37.52}{192.168.37.52:9300},}, reason: zen-disco-receive(from master [{node-1}{1APpnCwuRNaFyAd7u6V2Fg}{192.168.37.52}{192.168.37.52:9300}])
  368. [2017-04-04 19:40:15,619][INFO ][http ] [node-2] publish_address {192.168.37.55:9200}, bound_addresses {127.0.1.1:9200}, {192.168.37.55:9200}
  369. [2017-04-04 19:40:15,620][INFO ][node ] [node-2] started
  370. [2017-04-04 19:40:58,833][INFO ][discovery.zen ] [node-2] master_left [{node-1}{1APpnCwuRNaFyAd7u6V2Fg}{192.168.37.52}{192.168.37.52:9300}], reason [shut_down]
  371. [2017-04-04 19:40:58,835][WARN ][discovery.zen ] [node-2] master left (reason = shut_down), current nodes: {{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{yMzeark-RsGElqanaQvrMw}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false},{node-2}{6iC4dCfRRau-PJ5y7yNkcg}{192.168.37.55}{192.168.37.55:9300},}
  372. [2017-04-04 19:40:58,835][INFO ][cluster.service ] [node-2] removed {{node-1}{1APpnCwuRNaFyAd7u6V2Fg}{192.168.37.52}{192.168.37.52:9300},}, reason: zen-disco-master_failed ({node-1}{1APpnCwuRNaFyAd7u6V2Fg}{192.168.37.52}{192.168.37.52:9300})
  373. [2017-04-04 19:40:58,839][WARN ][discovery.zen.ping.unicast] [node-2] failed to send ping to [{#zen_unicast_1#}{192.168.37.52}{192.168.37.52:9300}]
  374. RemoteTransportException[[node-1][192.168.37.52:9300][internal:discovery/zen/unicast]]; nested: IllegalStateException[received ping request while not started];
  375. Caused by: java.lang.IllegalStateException: received ping request while not started
  376. at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.handlePingRequest(UnicastZenPing.java:497)
  377. at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.access$2400(UnicastZenPing.java:83)
  378. at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$UnicastPingRequestHandler.messageReceived(UnicastZenPing.java:522)
  379. at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$UnicastPingRequestHandler.messageReceived(UnicastZenPing.java:518)
  380. at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
  381. at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)
  382. at org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:227)
  383. at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:116)
  384. at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
  385. at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  386. at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
  387. at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
  388. at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
  389. at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
  390. at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
  391. at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
  392. at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  393. at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
  394. at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:75)
  395. at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  396. at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
  397. at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
  398. at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
  399. at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
  400. at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
  401. at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
  402. at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
  403. at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
  404. at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
  405. at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
  406. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  407. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  408. at java.lang.Thread.run(Thread.java:745)
  409. [2017-04-04 19:40:58,839][WARN ][discovery.zen.ping.unicast] [node-2] failed to send ping to [{node-1}{1APpnCwuRNaFyAd7u6V2Fg}{192.168.37.52}{192.168.37.52:9300}]
  410. RemoteTransportException[[node-1][192.168.37.52:9300][internal:discovery/zen/unicast]]; nested: IllegalStateException[received ping request while not started];
  411. Caused by: java.lang.IllegalStateException: received ping request while not started
  412. at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.handlePingRequest(UnicastZenPing.java:497)
  413. at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.access$2400(UnicastZenPing.java:83)
  414. at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$UnicastPingRequestHandler.messageReceived(UnicastZenPing.java:522)
  415. at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$UnicastPingRequestHandler.messageReceived(UnicastZenPing.java:518)
  416. at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
  417. at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)
  418. at org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:227)
  419. at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:116)
  420. at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
  421. at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  422. at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
  423. at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
  424. at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
  425. at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
  426. at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
  427. at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
  428. at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  429. at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
  430. at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:75)
  431. at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  432. at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
  433. at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
  434. at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
  435. at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
  436. at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
  437. at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
  438. at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
  439. at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
  440. at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
  441. at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
  442. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  443. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  444. at java.lang.Thread.run(Thread.java:745)
  445. [2017-04-04 19:40:59,929][DEBUG][action.admin.cluster.health] [node-2] no known master node, scheduling a retry
  446. [2017-04-04 19:40:59,930][DEBUG][action.admin.cluster.health] [node-2] no known master node, scheduling a retry
  447. [2017-04-04 19:41:03,457][INFO ][cluster.routing.allocation] [node-2] Cluster health status changed from [YELLOW] to [RED] (reason: [nodes joined]).
  448. [2017-04-04 19:41:03,458][INFO ][cluster.service ] [node-2] new_master {node-2}{6iC4dCfRRau-PJ5y7yNkcg}{192.168.37.55}{192.168.37.55:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
  449. [2017-04-04 19:41:03,931][INFO ][node ] [node-2] stopping ...
  450. [2017-04-04 19:41:03,936][DEBUG][action.admin.cluster.health] [node-2] no known master node, scheduling a retry
  451. [2017-04-04 19:41:03,936][DEBUG][action.admin.cluster.health] [node-2] no known master node, scheduling a retry
  452. [2017-04-04 19:41:03,951][WARN ][rest.suppressed ] path: /_cat/health, params: {}
  453. NodeClosedException[node closed {node-2}{6iC4dCfRRau-PJ5y7yNkcg}{192.168.37.55}{192.168.37.55:9300}]
  454. at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$5.onClusterServiceClose(TransportMasterNodeAction.java:228)
  455. at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onClose(ClusterStateObserver.java:222)
  456. at org.elasticsearch.cluster.service.InternalClusterService.doStop(InternalClusterService.java:216)
  457. at org.elasticsearch.common.component.AbstractLifecycleComponent.stop(AbstractLifecycleComponent.java:88)
  458. at org.elasticsearch.node.Node.stop(Node.java:326)
  459. at org.elasticsearch.node.Node.close(Node.java:351)
  460. at org.elasticsearch.bootstrap.Bootstrap$4.run(Bootstrap.java:173)
  461. [2017-04-04 19:41:03,952][WARN ][rest.suppressed ] path: /_cat/health, params: {}
  462. NodeClosedException[node closed {node-2}{6iC4dCfRRau-PJ5y7yNkcg}{192.168.37.55}{192.168.37.55:9300}]
  463. at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$5.onClusterServiceClose(TransportMasterNodeAction.java:228)
  464. at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onClose(ClusterStateObserver.java:222)
  465. at org.elasticsearch.cluster.service.InternalClusterService.doStop(InternalClusterService.java:216)
  466. at org.elasticsearch.common.component.AbstractLifecycleComponent.stop(AbstractLifecycleComponent.java:88)
  467. at org.elasticsearch.node.Node.stop(Node.java:326)
  468. at org.elasticsearch.node.Node.close(Node.java:351)
  469. at org.elasticsearch.bootstrap.Bootstrap$4.run(Bootstrap.java:173)
  470. [2017-04-04 19:41:03,955][WARN ][netty.channel.DefaultChannelPipeline] An exception was thrown by an exception handler.
  471. java.util.concurrent.RejectedExecutionException: Worker has already been shutdown
  472. at org.jboss.netty.channel.socket.nio.AbstractNioSelector.registerTask(AbstractNioSelector.java:120)
  473. at org.jboss.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:72)
  474. at org.jboss.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)
  475. at org.jboss.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:56)
  476. at org.jboss.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)
  477. at org.jboss.netty.channel.socket.nio.AbstractNioChannelSink.execute(AbstractNioChannelSink.java:34)
  478. at org.jboss.netty.channel.DefaultChannelPipeline.execute(DefaultChannelPipeline.java:636)
  479. at org.jboss.netty.channel.Channels.fireExceptionCaughtLater(Channels.java:496)
  480. at org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:46)
  481. at org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:658)
  482. at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:781)
  483. at org.jboss.netty.channel.Channels.write(Channels.java:725)
  484. at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:71)
  485. at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
  486. at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
  487. at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784)
  488. at org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.handleDownstream(HttpPipeliningHandler.java:87)
  489. at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
  490. at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
  491. at org.elasticsearch.http.netty.NettyHttpChannel.sendResponse(NettyHttpChannel.java:147)
  492. at org.elasticsearch.http.HttpServer$ResourceHandlingHttpChannel.sendResponse(HttpServer.java:352)
  493. at org.elasticsearch.rest.action.support.RestActionListener.onFailure(RestActionListener.java:60)
  494. at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:95)
  495. at org.elasticsearch.action.support.ThreadedActionListener$2.doRun(ThreadedActionListener.java:104)
  496. at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
  497. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  498. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  499. at java.lang.Thread.run(Thread.java:745)
  500. [2017-04-04 19:41:03,959][WARN ][netty.channel.DefaultChannelPipeline] An exception was thrown by an exception handler.
  501. java.util.concurrent.RejectedExecutionException: Worker has already been shutdown
  502. at org.jboss.netty.channel.socket.nio.AbstractNioSelector.registerTask(AbstractNioSelector.java:120)
  503. at org.jboss.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:72)
  504. at org.jboss.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)
  505. at org.jboss.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:56)
  506. at org.jboss.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)
  507. at org.jboss.netty.channel.socket.nio.AbstractNioChannelSink.execute(AbstractNioChannelSink.java:34)
  508. at org.jboss.netty.channel.DefaultChannelPipeline.execute(DefaultChannelPipeline.java:636)
  509. at org.jboss.netty.channel.Channels.fireExceptionCaughtLater(Channels.java:496)
  510. at org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:46)
  511. at org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:658)
  512. at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:781)
  513. at org.jboss.netty.channel.Channels.write(Channels.java:725)
  514. at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:71)
  515. at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
  516. at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
  517. at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784)
  518. at org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.handleDownstream(HttpPipeliningHandler.java:87)
  519. at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
  520. at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
  521. at org.elasticsearch.http.netty.NettyHttpChannel.sendResponse(NettyHttpChannel.java:147)
  522. at org.elasticsearch.http.HttpServer$ResourceHandlingHttpChannel.sendResponse(HttpServer.java:352)
  523. at org.elasticsearch.rest.action.support.RestActionListener.onFailure(RestActionListener.java:60)
  524. at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:95)
  525. at org.elasticsearch.action.support.ThreadedActionListener$2.doRun(ThreadedActionListener.java:104)
  526. at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
  527. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  528. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  529. at java.lang.Thread.run(Thread.java:745)
  530. [2017-04-04 19:41:13,953][INFO ][cluster.routing ] [node-2] delaying allocation for [188] unassigned shards, next check in [10.1s]
  531. [2017-04-04 19:41:30,353][WARN ][discovery.zen.ping.unicast] [node-2] [2] failed send ping to {#zen_unicast_1#}{192.168.37.52}{192.168.37.52:9300}
  532. java.lang.IllegalStateException: can't add nodes to a stopped transport
  533. at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:925)
  534. at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:906)
  535. at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:267)
  536. at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:395)
  537. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  538. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  539. at java.lang.Thread.run(Thread.java:745)
  540. [2017-04-04 19:41:30,440][INFO ][node ] [node-2] stopped
  541. [2017-04-04 19:41:30,440][INFO ][node ] [node-2] closing ...
  542. [2017-04-04 19:41:30,455][INFO ][node ] [node-2] closed
  543. [2017-04-04 19:44:05,422][INFO ][node ] [node-2] version[2.4.4], pid[1420], build[fcbb46d/2017-01-03T11:33:16Z]
  544. [2017-04-04 19:44:05,434][INFO ][node ] [node-2] initializing ...
  545. [2017-04-04 19:44:06,475][INFO ][plugins ] [node-2] modules [reindex, lang-expression, lang-groovy], plugins [], sites []
  546. [2017-04-04 19:44:06,534][INFO ][env ] [node-2] using [1] data paths, mounts [[/ (/dev/mapper/GraySrv2--vg-root)]], net usable_space [2.4tb], net total_space [2.6tb], spins? [possibly], types [ext4]
  547. [2017-04-04 19:44:06,534][INFO ][env ] [node-2] heap size [989.8mb], compressed ordinary object pointers [true]
  548. [2017-04-04 19:44:09,110][INFO ][node ] [node-2] initialized
  549. [2017-04-04 19:44:09,110][INFO ][node ] [node-2] starting ...
  550. [2017-04-04 19:44:09,254][INFO ][transport ] [node-2] publish_address {192.168.37.55:9300}, bound_addresses {127.0.1.1:9300}, {192.168.37.55:9300}
  551. [2017-04-04 19:44:09,259][INFO ][discovery ] [node-2] GRAY-graylog/xQKVr-jxQZGWodwP0hf8iw
  552. [2017-04-04 19:44:14,151][INFO ][cluster.service ] [node-2] detected_master {node-1}{DpHE5UbcSbqgpMjICgiLxQ}{192.168.37.52}{192.168.37.52:9300}, added {{node-1}{DpHE5UbcSbqgpMjICgiLxQ}{192.168.37.52}{192.168.37.52:9300},}, reason: zen-disco-receive(from master [{node-1}{DpHE5UbcSbqgpMjICgiLxQ}{192.168.37.52}{192.168.37.52:9300}])
  553. [2017-04-04 19:44:16,391][INFO ][http ] [node-2] publish_address {192.168.37.55:9200}, bound_addresses {127.0.1.1:9200}, {192.168.37.55:9200}
  554. [2017-04-04 19:44:16,391][INFO ][node ] [node-2] started
  555. [2017-04-04 19:44:28,326][INFO ][cluster.service ] [node-2] added {{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{u-bKA93wQs2DvOI_K30yeQ}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false},}, reason: zen-disco-receive(from master [{node-1}{DpHE5UbcSbqgpMjICgiLxQ}{192.168.37.52}{192.168.37.52:9300}])
  556. [2017-04-04 20:01:37,526][INFO ][discovery.zen ] [node-2] master_left [{node-1}{DpHE5UbcSbqgpMjICgiLxQ}{192.168.37.52}{192.168.37.52:9300}], reason [shut_down]
  557. [2017-04-04 20:01:37,528][WARN ][discovery.zen ] [node-2] master left (reason = shut_down), current nodes: {{node-2}{xQKVr-jxQZGWodwP0hf8iw}{192.168.37.55}{192.168.37.55:9300},{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{u-bKA93wQs2DvOI_K30yeQ}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false},}
  558. [2017-04-04 20:01:37,530][INFO ][cluster.service ] [node-2] removed {{node-1}{DpHE5UbcSbqgpMjICgiLxQ}{192.168.37.52}{192.168.37.52:9300},}, reason: zen-disco-master_failed ({node-1}{DpHE5UbcSbqgpMjICgiLxQ}{192.168.37.52}{192.168.37.52:9300})
  559. [2017-04-04 20:01:37,535][WARN ][discovery.zen.ping.unicast] [node-2] failed to send ping to [{node-1}{DpHE5UbcSbqgpMjICgiLxQ}{192.168.37.52}{192.168.37.52:9300}]
  560. RemoteTransportException[[node-1][192.168.37.52:9300][internal:discovery/zen/unicast]]; nested: IllegalStateException[received ping request while not started];
  561. Caused by: java.lang.IllegalStateException: received ping request while not started
  562. at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.handlePingRequest(UnicastZenPing.java:497)
  563. at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.access$2400(UnicastZenPing.java:83)
  564. at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$UnicastPingRequestHandler.messageReceived(UnicastZenPing.java:522)
  565. at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$UnicastPingRequestHandler.messageReceived(UnicastZenPing.java:518)
  566. at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
  567. at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)
  568. at org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:227)
  569. at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:116)
  570. at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
  571. at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  572. at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
  573. at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
  574. at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
  575. at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
  576. at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
  577. at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
  578. at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  579. at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
  580. at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:75)
  581. at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  582. at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
  583. at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
  584. at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
  585. at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
  586. at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
  587. at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
  588. at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
  589. at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
  590. at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
  591. at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
  592. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  593. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  594. at java.lang.Thread.run(Thread.java:745)
  595. [2017-04-04 20:01:37,535][WARN ][discovery.zen.ping.unicast] [node-2] failed to send ping to [{#zen_unicast_1#}{192.168.37.52}{192.168.37.52:9300}]
  596. RemoteTransportException[[node-1][192.168.37.52:9300][internal:discovery/zen/unicast]]; nested: IllegalStateException[received ping request while not started];
  597. Caused by: java.lang.IllegalStateException: received ping request while not started
  598. at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.handlePingRequest(UnicastZenPing.java:497)
  599. at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.access$2400(UnicastZenPing.java:83)
  600. at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$UnicastPingRequestHandler.messageReceived(UnicastZenPing.java:522)
  601. at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$UnicastPingRequestHandler.messageReceived(UnicastZenPing.java:518)
  602. at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
  603. at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)
  604. at org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:227)
  605. at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:116)
  606. at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
  607. at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  608. at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
  609. at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
  610. at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
  611. at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
  612. at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
  613. at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
  614. at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  615. at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
  616. at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:75)
  617. at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
  618. at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
  619. at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
  620. at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
  621. at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
  622. at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
  623. at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
  624. at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
  625. at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
  626. at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
  627. at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
  628. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  629. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  630. at java.lang.Thread.run(Thread.java:745)
  631. [2017-04-04 20:01:40,585][INFO ][cluster.routing.allocation] [node-2] Cluster health status changed from [GREEN] to [RED] (reason: [nodes joined]).
  632. [2017-04-04 20:01:40,585][INFO ][cluster.service ] [node-2] new_master {node-2}{xQKVr-jxQZGWodwP0hf8iw}{192.168.37.55}{192.168.37.55:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
  633. [2017-04-04 20:02:10,612][WARN ][discovery.zen.publish ] [node-2] timed out waiting for all nodes to process published state [248] (timeout [30s], pending nodes: [{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{u-bKA93wQs2DvOI_K30yeQ}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false}])
  634. [2017-04-04 20:02:10,615][INFO ][cluster.routing ] [node-2] delaying allocation for [190] unassigned shards, next check in [1m]
  635. [2017-04-04 20:02:19,542][WARN ][cluster.service ] [node-2] cluster state update task [zen-disco-join(elected_as_master, [0] joins received)] took 39s above the warn threshold of 30s
  636. [2017-04-04 20:02:19,542][INFO ][cluster.service ] [node-2] added {{node-1}{OEQt2xlyS3mOH4OCZSmNfw}{192.168.37.52}{192.168.37.52:9300},}, reason: zen-disco-join(pending joins after accumulation stop [election closed])
  637. [2017-04-04 20:02:49,545][WARN ][discovery.zen.publish ] [node-2] timed out waiting for all nodes to process published state [249] (timeout [30s], pending nodes: [{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{u-bKA93wQs2DvOI_K30yeQ}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false}])
  638. [2017-04-04 20:02:49,548][WARN ][cluster.service ] [node-2] cluster state update task [zen-disco-join(pending joins after accumulation stop [election closed])] took 30s above the warn threshold of 30s
  639. [2017-04-04 20:03:17,859][WARN ][transport ] [node-2] Received response for a request that has timed out, sent [85901ms] ago, timed out [55900ms] ago, action [internal:discovery/zen/fd/ping], node [{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{u-bKA93wQs2DvOI_K30yeQ}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false}], id [1935]
  640. [2017-04-04 20:03:19,549][WARN ][discovery.zen.publish ] [node-2] timed out waiting for all nodes to process published state [250] (timeout [30s], pending nodes: [{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{u-bKA93wQs2DvOI_K30yeQ}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false}])
  641. [2017-04-04 20:03:19,550][INFO ][cluster.routing ] [node-2] delaying allocation for [190] unassigned shards, next check in [1m]
  642. [2017-04-04 20:03:19,553][WARN ][cluster.service ] [node-2] cluster state update task [zen-disco-join(join from node[{node-1}{OEQt2xlyS3mOH4OCZSmNfw}{192.168.37.52}{192.168.37.52:9300}])] took 30s above the warn threshold of 30s
  643. [2017-04-04 20:03:49,727][WARN ][discovery.zen.publish ] [node-2] timed out waiting for all nodes to process published state [251] (timeout [30s], pending nodes: [{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{u-bKA93wQs2DvOI_K30yeQ}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false}])
  644. [2017-04-04 20:03:49,791][WARN ][cluster.service ] [node-2] cluster state update task [cluster_reroute(async_shard_fetch)] took 30.1s above the warn threshold of 30s
  645. [2017-04-04 20:03:49,808][INFO ][cluster.service ] [node-2] removed {{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{u-bKA93wQs2DvOI_K30yeQ}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false},}, reason: zen-disco-node-failed({GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{u-bKA93wQs2DvOI_K30yeQ}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false}), reason(failed to ping, tried [3] times, each with maximum [30s] timeout)
  646. [2017-04-04 20:03:49,977][INFO ][cluster.routing.allocation] [node-2] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[graylog_0][1], [graylog_0][1]] ...]).
  647. [2017-04-04 20:04:55,769][INFO ][cluster.routing.allocation] [node-2] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[session_login.cgi][3]] ...]).
  648. [2017-04-04 20:05:02,550][INFO ][node ] [node-2] stopping ...
  649. [2017-04-04 20:05:02,709][INFO ][node ] [node-2] stopped
  650. [2017-04-04 20:05:02,709][INFO ][node ] [node-2] closing ...
  651. [2017-04-04 20:05:02,717][INFO ][node ] [node-2] closed
  652. [2017-04-04 20:05:03,561][INFO ][node ] [node-2] version[2.4.4], pid[17453], build[fcbb46d/2017-01-03T11:33:16Z]
  653. [2017-04-04 20:05:03,562][INFO ][node ] [node-2] initializing ...
  654. [2017-04-04 20:05:04,202][INFO ][plugins ] [node-2] modules [reindex, lang-expression, lang-groovy], plugins [], sites []
  655. [2017-04-04 20:05:04,224][INFO ][env ] [node-2] using [1] data paths, mounts [[/ (/dev/mapper/GraySrv2--vg-root)]], net usable_space [2.4tb], net total_space [2.6tb], spins? [possibly], types [ext4]
  656. [2017-04-04 20:05:04,225][INFO ][env ] [node-2] heap size [989.8mb], compressed ordinary object pointers [true]
  657. [2017-04-04 20:05:06,215][INFO ][node ] [node-2] initialized
  658. [2017-04-04 20:05:06,215][INFO ][node ] [node-2] starting ...
  659. [2017-04-04 20:05:06,334][INFO ][transport ] [node-2] publish_address {192.168.37.55:9300}, bound_addresses {127.0.1.1:9300}, {192.168.37.55:9300}
  660. [2017-04-04 20:05:06,340][INFO ][discovery ] [node-2] GRAY-graylog/gXWWdKRJT8O8iGZh-5Xcfg
  661. [2017-04-04 20:05:15,564][INFO ][cluster.service ] [node-2] detected_master {node-1}{OEQt2xlyS3mOH4OCZSmNfw}{192.168.37.52}{192.168.37.52:9300}, added {{node-1}{OEQt2xlyS3mOH4OCZSmNfw}{192.168.37.52}{192.168.37.52:9300},}, reason: zen-disco-receive(from master [{node-1}{OEQt2xlyS3mOH4OCZSmNfw}{192.168.37.52}{192.168.37.52:9300}])
  662. [2017-04-04 20:05:17,593][INFO ][http ] [node-2] publish_address {192.168.37.55:9200}, bound_addresses {127.0.1.1:9200}, {192.168.37.55:9200}
  663. [2017-04-04 20:05:17,593][INFO ][node ] [node-2] started
  664. [2017-04-04 20:07:20,482][INFO ][cluster.service ] [node-2] added {{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{u-bKA93wQs2DvOI_K30yeQ}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false},}, reason: zen-disco-receive(from master [{node-1}{OEQt2xlyS3mOH4OCZSmNfw}{192.168.37.52}{192.168.37.52:9300}])
  665. [2017-04-04 20:30:26,582][INFO ][cluster.service ] [node-2] removed {{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{u-bKA93wQs2DvOI_K30yeQ}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false},}, reason: zen-disco-receive(from master [{node-1}{OEQt2xlyS3mOH4OCZSmNfw}{192.168.37.52}{192.168.37.52:9300}])
  666. [2017-04-04 20:35:45,631][INFO ][cluster.service ] [node-2] added {{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{u-bKA93wQs2DvOI_K30yeQ}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false},}, reason: zen-disco-receive(from master [{node-1}{OEQt2xlyS3mOH4OCZSmNfw}{192.168.37.52}{192.168.37.52:9300}])
  667. [2017-04-04 20:48:15,069][INFO ][cluster.service ] [node-2] removed {{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{u-bKA93wQs2DvOI_K30yeQ}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false},}, reason: zen-disco-receive(from master [{node-1}{OEQt2xlyS3mOH4OCZSmNfw}{192.168.37.52}{192.168.37.52:9300}])
  668. [2017-04-04 21:04:51,979][INFO ][cluster.service ] [node-2] added {{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{u-bKA93wQs2DvOI_K30yeQ}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false},}, reason: zen-disco-receive(from master [{node-1}{OEQt2xlyS3mOH4OCZSmNfw}{192.168.37.52}{192.168.37.52:9300}])
  669. [2017-04-04 21:14:48,719][INFO ][cluster.service ] [node-2] removed {{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{u-bKA93wQs2DvOI_K30yeQ}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false},}, reason: zen-disco-receive(from master [{node-1}{OEQt2xlyS3mOH4OCZSmNfw}{192.168.37.52}{192.168.37.52:9300}])
  670. [2017-04-04 21:38:52,212][INFO ][cluster.service ] [node-2] added {{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{u-bKA93wQs2DvOI_K30yeQ}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false},}, reason: zen-disco-receive(from master [{node-1}{OEQt2xlyS3mOH4OCZSmNfw}{192.168.37.52}{192.168.37.52:9300}])
  671. [2017-04-04 21:46:45,013][INFO ][cluster.service ] [node-2] removed {{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{u-bKA93wQs2DvOI_K30yeQ}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false},}, reason: zen-disco-receive(from master [{node-1}{OEQt2xlyS3mOH4OCZSmNfw}{192.168.37.52}{192.168.37.52:9300}])
  672. [2017-04-04 21:58:32,446][INFO ][cluster.service ] [node-2] added {{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{u-bKA93wQs2DvOI_K30yeQ}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false},}, reason: zen-disco-receive(from master [{node-1}{OEQt2xlyS3mOH4OCZSmNfw}{192.168.37.52}{192.168.37.52:9300}])
  673. [2017-04-04 22:04:29,687][INFO ][cluster.service ] [node-2] removed {{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{u-bKA93wQs2DvOI_K30yeQ}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false},}, reason: zen-disco-receive(from master [{node-1}{OEQt2xlyS3mOH4OCZSmNfw}{192.168.37.52}{192.168.37.52:9300}])
  674. [2017-04-04 22:10:56,586][INFO ][cluster.service ] [node-2] added {{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{u-bKA93wQs2DvOI_K30yeQ}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false},}, reason: zen-disco-receive(from master [{node-1}{OEQt2xlyS3mOH4OCZSmNfw}{192.168.37.52}{192.168.37.52:9300}])
  675. [2017-04-04 22:18:01,662][INFO ][cluster.service ] [node-2] removed {{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{u-bKA93wQs2DvOI_K30yeQ}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false},}, reason: zen-disco-receive(from master [{node-1}{OEQt2xlyS3mOH4OCZSmNfw}{192.168.37.52}{192.168.37.52:9300}])
  676. [2017-04-04 22:21:02,300][INFO ][cluster.service ] [node-2] added {{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{u-bKA93wQs2DvOI_K30yeQ}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false},}, reason: zen-disco-receive(from master [{node-1}{OEQt2xlyS3mOH4OCZSmNfw}{192.168.37.52}{192.168.37.52:9300}])
  677. [2017-04-04 22:27:35,320][INFO ][cluster.service ] [node-2] removed {{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{u-bKA93wQs2DvOI_K30yeQ}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false},}, reason: zen-disco-receive(from master [{node-1}{OEQt2xlyS3mOH4OCZSmNfw}{192.168.37.52}{192.168.37.52:9300}])
  678. [2017-04-04 22:34:32,170][INFO ][cluster.service ] [node-2] added {{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{u-bKA93wQs2DvOI_K30yeQ}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false},}, reason: zen-disco-receive(from master [{node-1}{OEQt2xlyS3mOH4OCZSmNfw}{192.168.37.52}{192.168.37.52:9300}])
  679. [2017-04-04 22:44:53,806][INFO ][cluster.service ] [node-2] removed {{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{u-bKA93wQs2DvOI_K30yeQ}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false},}, reason: zen-disco-receive(from master [{node-1}{OEQt2xlyS3mOH4OCZSmNfw}{192.168.37.52}{192.168.37.52:9300}])
  680. [2017-04-04 22:46:15,563][INFO ][cluster.service ] [node-2] added {{GRAY-c397e76f-56b3-4ec8-9f99-7e26f77f9f2c}{u-bKA93wQs2DvOI_K30yeQ}{192.168.37.55}{192.168.37.55:9350}{client=true, data=false, master=false},}, reason: zen-disco-receive(from master [{node-1}{OEQt2xlyS3mOH4OCZSmNfw}{192.168.37.52}{192.168.37.52:9300}])
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!