API tools faq
paste
Advertisement
Guest User

node3--standby-master

a guest
Jul 10th, 2017
95
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 280.98 KB | None | 0 0
raw download clone embed print report
  1. [2017-07-09T00:04:43,112][INFO ][o.e.m.j.JvmGcMonitorService] [node3] [gc][42674] overhead, spent [317ms] collecting in the last [1s]
  2. [2017-07-09T00:22:13,718][INFO ][o.e.m.j.JvmGcMonitorService] [node3] [gc][43723] overhead, spent [284ms] collecting in the last [1s]
  3. [2017-07-09T00:24:44,965][INFO ][o.e.m.j.JvmGcMonitorService] [node3] [gc][43874] overhead, spent [381ms] collecting in the last [1s]
  4. [2017-07-09T00:25:50,052][INFO ][o.e.m.j.JvmGcMonitorService] [node3] [gc][43939] overhead, spent [342ms] collecting in the last [1s]
  5. [2017-07-09T00:30:24,231][INFO ][o.e.m.j.JvmGcMonitorService] [node3] [gc][44213] overhead, spent [264ms] collecting in the last [1s]
  6. [2017-07-09T00:40:44,522][INFO ][o.e.m.j.JvmGcMonitorService] [node3] [gc][44832] overhead, spent [294ms] collecting in the last [1s]
  7. [2017-07-09T00:54:30,926][INFO ][o.e.m.j.JvmGcMonitorService] [node3] [gc][45657] overhead, spent [256ms] collecting in the last [1s]
  8. [2017-07-09T00:55:03,934][INFO ][o.e.m.j.JvmGcMonitorService] [node3] [gc][45690] overhead, spent [274ms] collecting in the last [1s]
  9. [2017-07-09T01:09:31,023][INFO ][o.e.m.j.JvmGcMonitorService] [node3] [gc][46556] overhead, spent [287ms] collecting in the last [1s]
  10. [2017-07-09T01:16:46,657][INFO ][o.e.m.j.JvmGcMonitorService] [node3] [gc][46991] overhead, spent [313ms] collecting in the last [1s]
  11. [2017-07-09T01:23:01,225][INFO ][o.e.m.j.JvmGcMonitorService] [node3] [gc][47365] overhead, spent [270ms] collecting in the last [1s]
  12. [2017-07-09T01:23:56,494][INFO ][o.e.m.j.JvmGcMonitorService] [node3] [gc][47420] overhead, spent [327ms] collecting in the last [1.1s]
  13. [2017-07-09T01:27:28,117][INFO ][o.e.d.z.ZenDiscovery ] [node3] master_left [ {node1}{---}{---}{10.10.10.1}{10.10.10.1:9300}{rack=rackA6-1}], reason [transport disconnected]
  14. [2017-07-09T01:27:28,120][WARN ][o.e.d.z.ZenDiscovery ] [node3] master left (reason = transport disconnected), current nodes: nodes:
  15. {node1}{---}{---}{10.10.10.1}{10.10.10.1:9300}{rack=rackA6-1}, master
  16. {node3}{---}{---}{10.10.10.3}{10.10.10.3:9300}{rack=rackA6-1}, local
  17. {index_node2}{---}{---}{10.10.10.101}{10.10.10.101:9300}{rack=rackA6-1}
  18. {node4}{---}{---}{10.10.10.4}{10.10.10.4:9300}{rack=rackA6-1}
  19. {index_node1}{---}{---}{10.10.10.100}{10.10.10.100:9300}{rack=rackA6-1}
  20. {node6}{---}{---}{10.10.10.6}{10.10.10.6:9300}{rack=rackA6-1}
  21. {node5}{---}{---}{10.10.10.5}{10.10.10.5:9300}{rack=rackA6-1}
  22. {node2}{---}{---}{10.10.10.2}{10.10.10.2:9300}{rack=rackA6-1}
  23.  
  24. [2017-07-09T01:27:28,168][WARN ][o.e.c.NodeConnectionsService] [node3] failed to connect to node {node1}{---}{---}{10.10.10.1}{10.10.10.1:9300}{rack=rackA6-1} (tried [1] times)
  25. org.elasticsearch.transport.ConnectTransportException: [node1][10.10.10.1:9300] connect_timeout[30s]
  26. at org.elasticsearch.transport.netty4.Netty4Transport.connectToChannels(Netty4Transport.java:361) ~[?:?]
  27. at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:548) ~[elasticsearch-5.5.0.jar:5.5.0]
  28. at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:472) ~[elasticsearch-5.5.0.jar:5.5.0]
  29. at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:332) ~[elasticsearch-5.5.0.jar:5.5.0]
  30. at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:319) ~[elasticsearch-5.5.0.jar:5.5.0]
  31. at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:154) [elasticsearch-5.5.0.jar:5.5.0]
  32. at org.elasticsearch.cluster.NodeConnectionsService$1.doRun(NodeConnectionsService.java:107) [elasticsearch-5.5.0.jar:5.5.0]
  33. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-5.5.0.jar:5.5.0]
  34. at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.5.0.jar:5.5.0]
  35. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
  36. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
  37. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
  38. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: 10.10.10.1/10.10.10.1:9300
  39. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  40. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
  41. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352) ~[?:?]
  42. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  43. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?]
  44. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?]
  45. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?]
  46. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?]
  47. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
  48. ... 1 more
  49. Caused by: java.net.ConnectException: Connection refused
  50. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  51. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
  52. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352) ~[?:?]
  53. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  54. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?]
  55. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?]
  56. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?]
  57. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?]
  58. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
  59. ... 1 more
  60. [2017-07-09T01:27:31,384][INFO ][o.e.c.s.ClusterService ] [node3] new_master {node3}{---}{---}{10.10.10.2}{10.10.10.2:9300}{rack=rackA6-1}]
  61. [2017-07-09T01:27:31,460][DEBUG][o.e.a.a.c.h.TransportClusterHealthAction] [node3] no known master node, scheduling a retry
  62. [2017-07-09T01:27:31,460][DEBUG][o.e.a.a.c.h.TransportClusterHealthAction] [node3] no known master node, scheduling a retry
  63. [2017-07-09T01:27:31,462][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  64. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  65. [2017-07-09T01:27:31,476][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  66. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  67. [2017-07-09T01:27:31,476][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  68. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  69. [2017-07-09T01:27:31,476][WARN ][o.e.c.a.s.ShardStateAction] [node3] [.monitoring-kibana-6-2017.07.09][0] received shard failed for shard id [[.monitoring-kibana-6-2017.07.09][0]], allocation id [99qnGHC_QwWQ9X7FneolBQ], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [.monitoring-kibana-6-2017.07.09][0], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=99qnGHC_QwWQ9X7FneolBQ]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  70. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  71. [2017-07-09T01:27:31,478][DEBUG][o.e.a.a.c.h.TransportClusterHealthAction] [node3] no known master node, scheduling a retry
  72. [2017-07-09T01:27:31,478][DEBUG][o.e.a.a.c.h.TransportClusterHealthAction] [node3] no known master node, scheduling a retry
  73. [2017-07-09T01:27:31,488][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  74. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  75. [2017-07-09T01:27:31,492][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  76. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  77. [2017-07-09T01:27:31,492][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  78. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  79. [2017-07-09T01:27:31,496][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  80. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  81. [2017-07-09T01:27:31,514][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  82. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  83. [2017-07-09T01:27:31,514][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  84. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  85. [2017-07-09T01:27:31,514][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  86. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  87. [2017-07-09T01:27:31,514][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  88. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  89. [2017-07-09T01:27:31,514][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  90. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  91. [2017-07-09T01:27:31,514][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  92. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  93. [2017-07-09T01:27:31,515][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  94. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  95. [2017-07-09T01:27:31,515][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  96. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  97. [2017-07-09T01:27:31,515][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  98. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  99. [2017-07-09T01:27:31,515][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  100. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  101. [2017-07-09T01:27:31,515][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  102. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  103. [2017-07-09T01:27:31,515][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  104. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  105. [2017-07-09T01:27:31,515][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  106. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  107. [2017-07-09T01:27:31,515][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  108. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  109. [2017-07-09T01:27:31,515][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  110. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  111. [2017-07-09T01:27:31,515][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  112. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  113. [2017-07-09T01:27:31,515][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  114. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  115. [2017-07-09T01:27:31,516][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  116. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  117. [2017-07-09T01:27:31,516][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  118. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  119. [2017-07-09T01:27:31,516][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  120. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  121. [2017-07-09T01:27:31,516][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  122. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  123. [2017-07-09T01:27:31,516][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  124. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  125. [2017-07-09T01:27:31,516][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  126. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  127. [2017-07-09T01:27:31,516][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  128. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  129. [2017-07-09T01:27:31,516][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  130. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  131. [2017-07-09T01:27:31,517][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  132. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  133. [2017-07-09T01:27:31,517][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  134. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  135. [2017-07-09T01:27:31,517][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  136. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  137. [2017-07-09T01:27:31,517][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  138. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  139. [2017-07-09T01:27:31,517][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  140. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  141. [2017-07-09T01:27:31,517][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  142. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  143. [2017-07-09T01:27:31,517][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  144. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  145. [2017-07-09T01:27:31,517][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  146. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  147. [2017-07-09T01:27:31,518][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  148. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  149. [2017-07-09T01:27:31,518][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  150. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  151. [2017-07-09T01:27:31,518][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  152. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  153. [2017-07-09T01:27:31,518][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  154. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  155. [2017-07-09T01:27:31,518][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  156. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  157. [2017-07-09T01:27:31,518][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  158. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  159. [2017-07-09T01:27:31,518][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  160. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  161. [2017-07-09T01:27:31,518][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  162. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  163. [2017-07-09T01:27:31,518][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  164. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  165. [2017-07-09T01:27:31,518][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  166. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  167. [2017-07-09T01:27:31,518][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  168. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  169. [2017-07-09T01:27:31,518][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  170. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  171. [2017-07-09T01:27:31,518][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  172. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  173. [2017-07-09T01:27:31,519][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  174. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  175. [2017-07-09T01:27:31,519][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  176. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  177. [2017-07-09T01:27:31,519][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  178. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  179. [2017-07-09T01:27:31,519][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  180. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  181. [2017-07-09T01:27:31,519][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  182. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  183. [2017-07-09T01:27:31,519][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  184. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  185. [2017-07-09T01:27:31,519][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  186. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  187. [2017-07-09T01:27:31,519][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  188. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  189. [2017-07-09T01:27:31,519][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  190. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  191. [2017-07-09T01:27:31,519][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  192. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  193. [2017-07-09T01:27:31,519][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  194. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  195. [2017-07-09T01:27:31,519][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  196. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  197. [2017-07-09T01:27:31,519][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  198. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  199. [2017-07-09T01:27:31,519][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  200. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  201. [2017-07-09T01:27:31,519][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  202. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  203. [2017-07-09T01:27:31,520][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  204. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  205. [2017-07-09T01:27:31,519][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  206. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  207. [2017-07-09T01:27:31,548][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  208. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  209. [2017-07-09T01:27:31,548][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  210. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  211. [2017-07-09T01:27:31,548][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  212. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  213. [2017-07-09T01:27:31,549][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  214. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  215. [2017-07-09T01:27:31,551][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  216. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  217. [2017-07-09T01:27:31,551][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][2] received shard failed for shard id [[logstash-2017.07.09][2]], allocation id [lpK-bHXmTnCQRpXNQYL7hg], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [logstash-2017.07.09][2], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=lpK-bHXmTnCQRpXNQYL7hg]], failure [NodeDisconnectedException[[node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected]]
  218. org.elasticsearch.transport.NodeDisconnectedException: [node1][10.10.10.1:9300][indices:data/write/bulk[s][r]] disconnected
  219. [2017-07-09T01:27:31,570][DEBUG][o.e.a.a.c.n.s.TransportNodesStatsAction] [node3] failed to execute on node [O7IIBUXxT3-IsVjcNjOelw]
  220. org.elasticsearch.transport.NodeNotConnectedException: [node1][10.10.10.1:9300] Node not connected
  221. at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:654) ~[elasticsearch-5.5.0.jar:5.5.0]
  222. at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:116) ~[elasticsearch-5.5.0.jar:5.5.0]
  223. at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:530) ~[elasticsearch-5.5.0.jar:5.5.0]
  224. at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:506) ~[elasticsearch-5.5.0.jar:5.5.0]
  225. at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.start(TransportNodesAction.java:197) ~[elasticsearch-5.5.0.jar:5.5.0]
  226. at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:89) ~[elasticsearch-5.5.0.jar:5.5.0]
  227. at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:52) ~[elasticsearch-5.5.0.jar:5.5.0]
  228. at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) ~[elasticsearch-5.5.0.jar:5.5.0]
  229. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) ~[elasticsearch-5.5.0.jar:5.5.0]
  230. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84) ~[elasticsearch-5.5.0.jar:5.5.0]
  231. at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) ~[elasticsearch-5.5.0.jar:5.5.0]
  232. at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) ~[elasticsearch-5.5.0.jar:5.5.0]
  233. at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408) ~[elasticsearch-5.5.0.jar:5.5.0]
  234. at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:730) ~[elasticsearch-5.5.0.jar:5.5.0]
  235. at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.nodesStats(AbstractClient.java:826) ~[elasticsearch-5.5.0.jar:5.5.0]
  236. at org.elasticsearch.cluster.InternalClusterInfoService.updateNodeStats(InternalClusterInfoService.java:256) ~[elasticsearch-5.5.0.jar:5.5.0]
  237. at org.elasticsearch.cluster.InternalClusterInfoService.refresh(InternalClusterInfoService.java:292) ~[elasticsearch-5.5.0.jar:5.5.0]
  238. at org.elasticsearch.cluster.InternalClusterInfoService.maybeRefresh(InternalClusterInfoService.java:277) ~[elasticsearch-5.5.0.jar:5.5.0]
  239. at org.elasticsearch.cluster.InternalClusterInfoService.lambda$onMaster$0(InternalClusterInfoService.java:137) ~[elasticsearch-5.5.0.jar:5.5.0]
  240. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-5.5.0.jar:5.5.0]
  241. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
  242. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
  243. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
  244. [2017-07-09T01:27:31,618][DEBUG][o.e.a.a.i.s.TransportIndicesStatsAction] [node3] failed to execute [indices:monitor/stats] on node [O7IIBUXxT3-IsVjcNjOelw]
  245. org.elasticsearch.transport.NodeNotConnectedException: [node1][10.10.10.1:9300] Node not connected
  246. at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:654) ~[elasticsearch-5.5.0.jar:5.5.0]
  247. at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:116) ~[elasticsearch-5.5.0.jar:5.5.0]
  248. at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:530) ~[elasticsearch-5.5.0.jar:5.5.0]
  249. at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:493) ~[elasticsearch-5.5.0.jar:5.5.0]
  250. at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.sendNodeRequest(TransportBroadcastByNodeAction.java:322) ~[elasticsearch-5.5.0.jar:5.5.0]
  251. at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.start(TransportBroadcastByNodeAction.java:311) ~[elasticsearch-5.5.0.jar:5.5.0]
  252. at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:234) ~[elasticsearch-5.5.0.jar:5.5.0]
  253. at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:79) ~[elasticsearch-5.5.0.jar:5.5.0]
  254. at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) ~[elasticsearch-5.5.0.jar:5.5.0]
  255. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) ~[elasticsearch-5.5.0.jar:5.5.0]
  256. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84) ~[elasticsearch-5.5.0.jar:5.5.0]
  257. at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) ~[elasticsearch-5.5.0.jar:5.5.0]
  258. at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) ~[elasticsearch-5.5.0.jar:5.5.0]
  259. at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408) ~[elasticsearch-5.5.0.jar:5.5.0]
  260. at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1256) ~[elasticsearch-5.5.0.jar:5.5.0]
  261. at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.stats(AbstractClient.java:1577) ~[elasticsearch-5.5.0.jar:5.5.0]
  262. at org.elasticsearch.cluster.InternalClusterInfoService.updateIndicesStats(InternalClusterInfoService.java:270) ~[elasticsearch-5.5.0.jar:5.5.0]
  263. at org.elasticsearch.cluster.InternalClusterInfoService.refresh(InternalClusterInfoService.java:321) ~[elasticsearch-5.5.0.jar:5.5.0]
  264. at org.elasticsearch.cluster.InternalClusterInfoService.maybeRefresh(InternalClusterInfoService.java:277) ~[elasticsearch-5.5.0.jar:5.5.0]
  265. at org.elasticsearch.cluster.InternalClusterInfoService.lambda$onMaster$0(InternalClusterInfoService.java:137) ~[elasticsearch-5.5.0.jar:5.5.0]
  266. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-5.5.0.jar:5.5.0]
  267. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
  268. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
  269. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
  270. [2017-07-09T01:27:32,038][INFO ][o.e.c.r.a.AllocationService] [node3] Cluster health status changed from [GREEN] to [YELLOW] (reason: [ {node1}{---}{---}{10.10.10.1}{10.10.10.1:9300}{rack=rackA6-1} transport disconnected]).
  271. [2017-07-09T01:27:32,039][INFO ][o.e.c.s.ClusterService ] [node3] removed { {node1}{---}{---}{10.10.10.1}{10.10.10.1:9300}{rack=rackA6-1} transport disconnected]
  272. [2017-07-09T01:27:33,464][WARN ][o.e.c.a.s.ShardStateAction] [node3] [.monitoring-kibana-6-2017.07.09][0] received shard failed for shard id [[.monitoring-kibana-6-2017.07.09][0]], allocation id [99qnGHC_QwWQ9X7FneolBQ], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [.monitoring-kibana-6-2017.07.09][0], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=99qnGHC_QwWQ9X7FneolBQ]], failure [NodeNotConnectedException[[node1][10.10.10.1:9300] Node not connected]]
  273. org.elasticsearch.transport.NodeNotConnectedException: [node1][10.10.10.1:9300] Node not connected
  274. at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:654) ~[elasticsearch-5.5.0.jar:5.5.0]
  275. at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:116) ~[elasticsearch-5.5.0.jar:5.5.0]
  276. at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:530) ~[elasticsearch-5.5.0.jar:5.5.0]
  277. at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:506) ~[elasticsearch-5.5.0.jar:5.5.0]
  278. at org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicasProxy.performOn(TransportReplicationAction.java:962) ~[elasticsearch-5.5.0.jar:5.5.0]
  279. at org.elasticsearch.action.support.replication.ReplicationOperation.performOnReplica(ReplicationOperation.java:186) ~[elasticsearch-5.5.0.jar:5.5.0]
  280. at org.elasticsearch.action.support.replication.ReplicationOperation.performOnReplicas(ReplicationOperation.java:170) ~[elasticsearch-5.5.0.jar:5.5.0]
  281. at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:130) ~[elasticsearch-5.5.0.jar:5.5.0]
  282. at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:322) ~[elasticsearch-5.5.0.jar:5.5.0]
  283. at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:264) ~[elasticsearch-5.5.0.jar:5.5.0]
  284. at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:888) ~[elasticsearch-5.5.0.jar:5.5.0]
  285. at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:885) ~[elasticsearch-5.5.0.jar:5.5.0]
  286. at org.elasticsearch.index.shard.IndexShardOperationsLock.acquire(IndexShardOperationsLock.java:147) ~[elasticsearch-5.5.0.jar:5.5.0]
  287. at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationLock(IndexShard.java:1657) ~[elasticsearch-5.5.0.jar:5.5.0]
  288. at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:897) ~[elasticsearch-5.5.0.jar:5.5.0]
  289. at org.elasticsearch.action.support.replication.TransportReplicationAction.access$400(TransportReplicationAction.java:93) ~[elasticsearch-5.5.0.jar:5.5.0]
  290. at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:281) ~[elasticsearch-5.5.0.jar:5.5.0]
  291. at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.5.0.jar:5.5.0]
  292. at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:260) ~[elasticsearch-5.5.0.jar:5.5.0]
  293. at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:252) ~[elasticsearch-5.5.0.jar:5.5.0]
  294. at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.5.0.jar:5.5.0]
  295. at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:644) ~[elasticsearch-5.5.0.jar:5.5.0]
  296. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) ~[elasticsearch-5.5.0.jar:5.5.0]
  297. at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.5.0.jar:5.5.0]
  298. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_131]
  299. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_131]
  300. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
  301. [2017-07-09T01:27:34,404][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  302. [2017-07-09T01:27:34,404][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  303. [2017-07-09T01:27:34,404][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  304. [2017-07-09T01:27:34,405][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  305. [2017-07-09T01:27:34,406][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  306. [2017-07-09T01:27:34,406][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  307. [2017-07-09T01:27:34,406][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  308. [2017-07-09T01:27:34,404][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  309. [2017-07-09T01:27:34,406][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  310. [2017-07-09T01:27:34,406][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  311. [2017-07-09T01:27:34,406][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  312. [2017-07-09T01:27:34,406][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  313. [2017-07-09T01:27:34,404][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  314. [2017-07-09T01:27:34,406][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  315. [2017-07-09T01:27:34,406][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  316. [2017-07-09T01:27:34,406][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  317. [2017-07-09T01:27:34,406][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  318. [2017-07-09T01:27:34,406][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  319. [2017-07-09T01:27:34,406][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  320. [2017-07-09T01:27:34,407][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  321. [2017-07-09T01:27:34,407][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  322. [2017-07-09T01:27:34,407][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  323. [2017-07-09T01:27:34,407][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  324. [2017-07-09T01:27:34,406][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  325. [2017-07-09T01:27:34,407][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  326. [2017-07-09T01:27:34,407][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  327. [2017-07-09T01:27:34,407][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  328. [2017-07-09T01:27:34,408][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  329. [2017-07-09T01:27:34,408][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  330. [2017-07-09T01:27:34,408][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  331. [2017-07-09T01:27:34,408][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  332. [2017-07-09T01:27:34,408][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  333. [2017-07-09T01:27:34,408][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  334. [2017-07-09T01:27:34,408][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  335. [2017-07-09T01:27:34,408][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  336. [2017-07-09T01:27:34,408][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  337. [2017-07-09T01:27:34,408][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  338. [2017-07-09T01:27:34,408][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  339. [2017-07-09T01:27:34,408][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  340. [2017-07-09T01:27:34,408][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  341. [2017-07-09T01:27:34,408][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  342. [2017-07-09T01:27:34,408][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  343. [2017-07-09T01:27:34,409][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  344. [2017-07-09T01:27:34,409][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  345. [2017-07-09T01:27:34,409][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  346. [2017-07-09T01:27:34,409][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  347. [2017-07-09T01:27:34,409][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  348. [2017-07-09T01:27:34,409][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  349. [2017-07-09T01:27:34,409][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  350. [2017-07-09T01:27:34,409][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  351. [2017-07-09T01:27:34,410][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  352. [2017-07-09T01:27:34,410][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  353. [2017-07-09T01:27:34,410][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  354. [2017-07-09T01:27:34,410][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  355. [2017-07-09T01:27:34,410][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  356. [2017-07-09T01:27:34,411][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  357. [2017-07-09T01:27:34,411][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  358. [2017-07-09T01:27:34,412][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  359. [2017-07-09T01:27:34,412][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  360. [2017-07-09T01:27:34,412][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  361. [2017-07-09T01:27:34,412][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  362. [2017-07-09T01:27:34,413][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  363. [2017-07-09T01:27:34,413][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  364. [2017-07-09T01:27:34,413][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  365. [2017-07-09T01:27:34,414][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  366. [2017-07-09T01:27:34,414][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  367. [2017-07-09T01:27:34,414][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  368. [2017-07-09T01:27:34,415][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  369. [2017-07-09T01:27:34,415][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  370. [2017-07-09T01:27:34,415][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  371. [2017-07-09T01:27:34,415][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  372. [2017-07-09T01:27:34,416][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  373. [2017-07-09T01:27:34,416][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  374. [2017-07-09T01:27:34,416][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  375. [2017-07-09T01:27:34,417][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  376. [2017-07-09T01:27:34,417][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  377. [2017-07-09T01:27:34,418][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  378. [2017-07-09T01:27:34,419][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  379. [2017-07-09T01:27:34,422][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  380. [2017-07-09T01:27:34,431][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  381. [2017-07-09T01:27:34,692][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  382. [2017-07-09T01:27:34,710][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  383. [2017-07-09T01:27:34,712][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  384. [2017-07-09T01:27:34,774][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  385. [2017-07-09T01:27:34,805][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [OQeGm3ObSM6zaEvjQhmMBQ], primary term [2], message [mark copy as stale]
  386. [2017-07-09T01:27:36,790][WARN ][o.e.c.a.s.ShardStateAction] [node3] [.monitoring-kibana-6-2017.07.09][0] received shard failed for shard id [[.monitoring-kibana-6-2017.07.09][0]], allocation id [99qnGHC_QwWQ9X7FneolBQ], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [.monitoring-kibana-6-2017.07.09][0], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=99qnGHC_QwWQ9X7FneolBQ]], failure [NodeNotConnectedException[[node1][10.10.10.1:9300] Node not connected]]
  387. org.elasticsearch.transport.NodeNotConnectedException: [node1][10.10.10.1:9300] Node not connected
  388. at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:654) ~[elasticsearch-5.5.0.jar:5.5.0]
  389. at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:116) ~[elasticsearch-5.5.0.jar:5.5.0]
  390. at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:530) ~[elasticsearch-5.5.0.jar:5.5.0]
  391. at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:506) ~[elasticsearch-5.5.0.jar:5.5.0]
  392. at org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicasProxy.performOn(TransportReplicationAction.java:962) ~[elasticsearch-5.5.0.jar:5.5.0]
  393. at org.elasticsearch.action.support.replication.ReplicationOperation.performOnReplica(ReplicationOperation.java:186) ~[elasticsearch-5.5.0.jar:5.5.0]
  394. at org.elasticsearch.action.support.replication.ReplicationOperation.performOnReplicas(ReplicationOperation.java:170) ~[elasticsearch-5.5.0.jar:5.5.0]
  395. at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:130) ~[elasticsearch-5.5.0.jar:5.5.0]
  396. at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:322) ~[elasticsearch-5.5.0.jar:5.5.0]
  397. at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:264) ~[elasticsearch-5.5.0.jar:5.5.0]
  398. at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:888) ~[elasticsearch-5.5.0.jar:5.5.0]
  399. at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:885) ~[elasticsearch-5.5.0.jar:5.5.0]
  400. at org.elasticsearch.index.shard.IndexShardOperationsLock.acquire(IndexShardOperationsLock.java:147) ~[elasticsearch-5.5.0.jar:5.5.0]
  401. at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationLock(IndexShard.java:1657) ~[elasticsearch-5.5.0.jar:5.5.0]
  402. at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:897) ~[elasticsearch-5.5.0.jar:5.5.0]
  403. at org.elasticsearch.action.support.replication.TransportReplicationAction.access$400(TransportReplicationAction.java:93) ~[elasticsearch-5.5.0.jar:5.5.0]
  404. at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:281) ~[elasticsearch-5.5.0.jar:5.5.0]
  405. at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.5.0.jar:5.5.0]
  406. at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:260) ~[elasticsearch-5.5.0.jar:5.5.0]
  407. at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:252) ~[elasticsearch-5.5.0.jar:5.5.0]
  408. at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.5.0.jar:5.5.0]
  409. at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:644) ~[elasticsearch-5.5.0.jar:5.5.0]
  410. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) ~[elasticsearch-5.5.0.jar:5.5.0]
  411. at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.5.0.jar:5.5.0]
  412. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_131]
  413. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_131]
  414. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
  415. [2017-07-09T01:27:39,907][DEBUG][o.e.a.a.c.s.TransportClusterStatsAction] [node3] failed to execute on node [O7IIBUXxT3-IsVjcNjOelw]
  416. org.elasticsearch.transport.NodeNotConnectedException: [node1][10.10.10.1:9300] Node not connected
  417. at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:654) ~[elasticsearch-5.5.0.jar:5.5.0]
  418. at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:116) ~[elasticsearch-5.5.0.jar:5.5.0]
  419. at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:530) ~[elasticsearch-5.5.0.jar:5.5.0]
  420. at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:506) ~[elasticsearch-5.5.0.jar:5.5.0]
  421. at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.start(TransportNodesAction.java:197) ~[elasticsearch-5.5.0.jar:5.5.0]
  422. at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:89) ~[elasticsearch-5.5.0.jar:5.5.0]
  423. at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:52) ~[elasticsearch-5.5.0.jar:5.5.0]
  424. at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) ~[elasticsearch-5.5.0.jar:5.5.0]
  425. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) ~[elasticsearch-5.5.0.jar:5.5.0]
  426. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84) ~[elasticsearch-5.5.0.jar:5.5.0]
  427. at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) ~[elasticsearch-5.5.0.jar:5.5.0]
  428. at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) ~[elasticsearch-5.5.0.jar:5.5.0]
  429. at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408) ~[elasticsearch-5.5.0.jar:5.5.0]
  430. at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:67) ~[elasticsearch-5.5.0.jar:5.5.0]
  431. at org.elasticsearch.xpack.security.InternalClient.doExecute(InternalClient.java:84) ~[?:?]
  432. at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408) ~[elasticsearch-5.5.0.jar:5.5.0]
  433. at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:730) ~[elasticsearch-5.5.0.jar:5.5.0]
  434. at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:80) ~[elasticsearch-5.5.0.jar:5.5.0]
  435. at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:54) ~[elasticsearch-5.5.0.jar:5.5.0]
  436. at org.elasticsearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:69) ~[elasticsearch-5.5.0.jar:5.5.0]
  437. at org.elasticsearch.xpack.monitoring.collector.cluster.ClusterStatsCollector.lambda$doCollect$0(ClusterStatsCollector.java:80) ~[x-pack-5.5.0.jar:5.5.0]
  438. at org.elasticsearch.xpack.monitoring.collector.cluster.ClusterStatsCollector.doCollect(ClusterStatsCollector.java:84) [x-pack-5.5.0.jar:5.5.0]
  439. at org.elasticsearch.xpack.monitoring.collector.Collector.collect(Collector.java:83) [x-pack-5.5.0.jar:5.5.0]
  440. at org.elasticsearch.xpack.monitoring.MonitoringService$MonitoringExecution$1.doRun(MonitoringService.java:195) [x-pack-5.5.0.jar:5.5.0]
  441. at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.5.0.jar:5.5.0]
  442. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_131]
  443. at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131]
  444. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-5.5.0.jar:5.5.0]
  445. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
  446. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
  447. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
  448. [2017-07-09T01:27:41,207][DEBUG][o.e.a.a.i.s.TransportIndicesStatsAction] [node3] failed to execute [indices:monitor/stats] on node [O7IIBUXxT3-IsVjcNjOelw]
  449. org.elasticsearch.transport.NodeNotConnectedException: [node1][10.10.10.1:9300] Node not connected
  450. at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:654) ~[elasticsearch-5.5.0.jar:5.5.0]
  451. at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:116) ~[elasticsearch-5.5.0.jar:5.5.0]
  452. at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:530) ~[elasticsearch-5.5.0.jar:5.5.0]
  453. at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:493) [elasticsearch-5.5.0.jar:5.5.0]
  454. at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.sendNodeRequest(TransportBroadcastByNodeAction.java:322) [elasticsearch-5.5.0.jar:5.5.0]
  455. at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.start(TransportBroadcastByNodeAction.java:311) [elasticsearch-5.5.0.jar:5.5.0]
  456. at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:234) [elasticsearch-5.5.0.jar:5.5.0]
  457. at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:79) [elasticsearch-5.5.0.jar:5.5.0]
  458. at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) [elasticsearch-5.5.0.jar:5.5.0]
  459. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) [elasticsearch-5.5.0.jar:5.5.0]
  460. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84) [elasticsearch-5.5.0.jar:5.5.0]
  461. at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) [elasticsearch-5.5.0.jar:5.5.0]
  462. at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) [elasticsearch-5.5.0.jar:5.5.0]
  463. at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408) [elasticsearch-5.5.0.jar:5.5.0]
  464. at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:67) [elasticsearch-5.5.0.jar:5.5.0]
  465. at org.elasticsearch.xpack.security.InternalClient.doExecute(InternalClient.java:84) [x-pack-5.5.0.jar:5.5.0]
  466. at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408) [elasticsearch-5.5.0.jar:5.5.0]
  467. at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1256) [elasticsearch-5.5.0.jar:5.5.0]
  468. at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:80) [elasticsearch-5.5.0.jar:5.5.0]
  469. at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:54) [elasticsearch-5.5.0.jar:5.5.0]
  470. at org.elasticsearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:69) [elasticsearch-5.5.0.jar:5.5.0]
  471. at org.elasticsearch.xpack.monitoring.collector.indices.IndicesStatsCollector.doCollect(IndicesStatsCollector.java:66) [x-pack-5.5.0.jar:5.5.0]
  472. at org.elasticsearch.xpack.monitoring.collector.Collector.collect(Collector.java:83) [x-pack-5.5.0.jar:5.5.0]
  473. at org.elasticsearch.xpack.monitoring.MonitoringService$MonitoringExecution$1.doRun(MonitoringService.java:195) [x-pack-5.5.0.jar:5.5.0]
  474. at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.5.0.jar:5.5.0]
  475. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_131]
  476. at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131]
  477. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-5.5.0.jar:5.5.0]
  478. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
  479. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
  480. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
  481. [2017-07-09T01:27:41,353][DEBUG][o.e.a.a.i.s.TransportIndicesStatsAction] [node3] failed to execute [indices:monitor/stats] on node [O7IIBUXxT3-IsVjcNjOelw]
  482. org.elasticsearch.transport.NodeNotConnectedException: [node1][10.10.10.1:9300] Node not connected
  483. at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:654) ~[elasticsearch-5.5.0.jar:5.5.0]
  484. at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:116) ~[elasticsearch-5.5.0.jar:5.5.0]
  485. at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:530) ~[elasticsearch-5.5.0.jar:5.5.0]
  486. at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:493) [elasticsearch-5.5.0.jar:5.5.0]
  487. at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.sendNodeRequest(TransportBroadcastByNodeAction.java:322) [elasticsearch-5.5.0.jar:5.5.0]
  488. at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.start(TransportBroadcastByNodeAction.java:311) [elasticsearch-5.5.0.jar:5.5.0]
  489. at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:234) [elasticsearch-5.5.0.jar:5.5.0]
  490. at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:79) [elasticsearch-5.5.0.jar:5.5.0]
  491. at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) [elasticsearch-5.5.0.jar:5.5.0]
  492. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) [elasticsearch-5.5.0.jar:5.5.0]
  493. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84) [elasticsearch-5.5.0.jar:5.5.0]
  494. at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) [elasticsearch-5.5.0.jar:5.5.0]
  495. at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) [elasticsearch-5.5.0.jar:5.5.0]
  496. at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408) [elasticsearch-5.5.0.jar:5.5.0]
  497. at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:67) [elasticsearch-5.5.0.jar:5.5.0]
  498. at org.elasticsearch.xpack.security.InternalClient.doExecute(InternalClient.java:84) [x-pack-5.5.0.jar:5.5.0]
  499. at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408) [elasticsearch-5.5.0.jar:5.5.0]
  500. at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1256) [elasticsearch-5.5.0.jar:5.5.0]
  501. at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:80) [elasticsearch-5.5.0.jar:5.5.0]
  502. at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:54) [elasticsearch-5.5.0.jar:5.5.0]
  503. at org.elasticsearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:69) [elasticsearch-5.5.0.jar:5.5.0]
  504. at org.elasticsearch.xpack.monitoring.collector.indices.IndexStatsCollector.doCollect(IndexStatsCollector.java:77) [x-pack-5.5.0.jar:5.5.0]
  505. at org.elasticsearch.xpack.monitoring.collector.Collector.collect(Collector.java:83) [x-pack-5.5.0.jar:5.5.0]
  506. at org.elasticsearch.xpack.monitoring.MonitoringService$MonitoringExecution$1.doRun(MonitoringService.java:195) [x-pack-5.5.0.jar:5.5.0]
  507. at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.5.0.jar:5.5.0]
  508. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_131]
  509. at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131]
  510. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-5.5.0.jar:5.5.0]
  511. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
  512. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
  513. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
  514. [2017-07-09T01:27:41,633][DEBUG][o.e.a.a.i.r.TransportRecoveryAction] [node3] failed to execute [indices:monitor/recovery] on node [O7IIBUXxT3-IsVjcNjOelw]
  515. org.elasticsearch.transport.NodeNotConnectedException: [node1][10.10.10.1:9300] Node not connected
  516. at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:654) ~[elasticsearch-5.5.0.jar:5.5.0]
  517. at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:116) ~[elasticsearch-5.5.0.jar:5.5.0]
  518. at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:530) ~[elasticsearch-5.5.0.jar:5.5.0]
  519. at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:493) [elasticsearch-5.5.0.jar:5.5.0]
  520. at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.sendNodeRequest(TransportBroadcastByNodeAction.java:322) [elasticsearch-5.5.0.jar:5.5.0]
  521. at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.start(TransportBroadcastByNodeAction.java:311) [elasticsearch-5.5.0.jar:5.5.0]
  522. at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:234) [elasticsearch-5.5.0.jar:5.5.0]
  523. at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:79) [elasticsearch-5.5.0.jar:5.5.0]
  524. at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) [elasticsearch-5.5.0.jar:5.5.0]
  525. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) [elasticsearch-5.5.0.jar:5.5.0]
  526. at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84) [elasticsearch-5.5.0.jar:5.5.0]
  527. at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) [elasticsearch-5.5.0.jar:5.5.0]
  528. at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) [elasticsearch-5.5.0.jar:5.5.0]
  529. at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408) [elasticsearch-5.5.0.jar:5.5.0]
  530. at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:67) [elasticsearch-5.5.0.jar:5.5.0]
  531. at org.elasticsearch.xpack.security.InternalClient.doExecute(InternalClient.java:84) [x-pack-5.5.0.jar:5.5.0]
  532. at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408) [elasticsearch-5.5.0.jar:5.5.0]
  533. at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1256) [elasticsearch-5.5.0.jar:5.5.0]
  534. at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:80) [elasticsearch-5.5.0.jar:5.5.0]
  535. at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:54) [elasticsearch-5.5.0.jar:5.5.0]
  536. at org.elasticsearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:69) [elasticsearch-5.5.0.jar:5.5.0]
  537. at org.elasticsearch.xpack.monitoring.collector.indices.IndexRecoveryCollector.doCollect(IndexRecoveryCollector.java:65) [x-pack-5.5.0.jar:5.5.0]
  538. at org.elasticsearch.xpack.monitoring.collector.Collector.collect(Collector.java:83) [x-pack-5.5.0.jar:5.5.0]
  539. at org.elasticsearch.xpack.monitoring.MonitoringService$MonitoringExecution$1.doRun(MonitoringService.java:195) [x-pack-5.5.0.jar:5.5.0]
  540. at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.5.0.jar:5.5.0]
  541. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_131]
  542. at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131]
  543. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-5.5.0.jar:5.5.0]
  544. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
  545. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
  546. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
  547. [2017-07-09T01:27:46,188][WARN ][o.e.c.a.s.ShardStateAction] [node3] [.monitoring-kibana-6-2017.07.09][0] received shard failed for shard id [[.monitoring-kibana-6-2017.07.09][0]], allocation id [99qnGHC_QwWQ9X7FneolBQ], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [.monitoring-kibana-6-2017.07.09][0], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=99qnGHC_QwWQ9X7FneolBQ]], failure [NodeNotConnectedException[[node1][10.10.10.1:9300] Node not connected]]
  548. org.elasticsearch.transport.NodeNotConnectedException: [node1][10.10.10.1:9300] Node not connected
  549. at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:654) ~[elasticsearch-5.5.0.jar:5.5.0]
  550. at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:116) ~[elasticsearch-5.5.0.jar:5.5.0]
  551. at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:530) ~[elasticsearch-5.5.0.jar:5.5.0]
  552. at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:506) ~[elasticsearch-5.5.0.jar:5.5.0]
  553. at org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicasProxy.performOn(TransportReplicationAction.java:962) ~[elasticsearch-5.5.0.jar:5.5.0]
  554. at org.elasticsearch.action.support.replication.ReplicationOperation.performOnReplica(ReplicationOperation.java:186) ~[elasticsearch-5.5.0.jar:5.5.0]
  555. at org.elasticsearch.action.support.replication.ReplicationOperation.performOnReplicas(ReplicationOperation.java:170) ~[elasticsearch-5.5.0.jar:5.5.0]
  556. at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:130) ~[elasticsearch-5.5.0.jar:5.5.0]
  557. at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:322) ~[elasticsearch-5.5.0.jar:5.5.0]
  558. at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:264) ~[elasticsearch-5.5.0.jar:5.5.0]
  559. at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:888) ~[elasticsearch-5.5.0.jar:5.5.0]
  560. at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:885) ~[elasticsearch-5.5.0.jar:5.5.0]
  561. at org.elasticsearch.index.shard.IndexShardOperationsLock.acquire(IndexShardOperationsLock.java:147) ~[elasticsearch-5.5.0.jar:5.5.0]
  562. at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationLock(IndexShard.java:1657) ~[elasticsearch-5.5.0.jar:5.5.0]
  563. at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:897) ~[elasticsearch-5.5.0.jar:5.5.0]
  564. at org.elasticsearch.action.support.replication.TransportReplicationAction.access$400(TransportReplicationAction.java:93) ~[elasticsearch-5.5.0.jar:5.5.0]
  565. at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:281) ~[elasticsearch-5.5.0.jar:5.5.0]
  566. at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.5.0.jar:5.5.0]
  567. at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:260) ~[elasticsearch-5.5.0.jar:5.5.0]
  568. at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:252) ~[elasticsearch-5.5.0.jar:5.5.0]
  569. at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.5.0.jar:5.5.0]
  570. at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:644) ~[elasticsearch-5.5.0.jar:5.5.0]
  571. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) ~[elasticsearch-5.5.0.jar:5.5.0]
  572. at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.5.0.jar:5.5.0]
  573. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_131]
  574. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_131]
  575. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
  576. [2017-07-09T01:27:46,418][WARN ][o.e.c.a.s.ShardStateAction] [node3] [.monitoring-kibana-6-2017.07.09][0] received shard failed for shard id [[.monitoring-kibana-6-2017.07.09][0]], allocation id [99qnGHC_QwWQ9X7FneolBQ], primary term [1], message [failed to perform indices:data/write/bulk[s] on replica [.monitoring-kibana-6-2017.07.09][0], node[O7IIBUXxT3-IsVjcNjOelw], [R], s[STARTED], a[id=99qnGHC_QwWQ9X7FneolBQ]], failure [NodeNotConnectedException[[node1][10.10.10.1:9300] Node not connected]]
  577. org.elasticsearch.transport.NodeNotConnectedException: [node1][10.10.10.1:9300] Node not connected
  578. at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:654) ~[elasticsearch-5.5.0.jar:5.5.0]
  579. at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:116) ~[elasticsearch-5.5.0.jar:5.5.0]
  580. at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:530) ~[elasticsearch-5.5.0.jar:5.5.0]
  581. at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:506) ~[elasticsearch-5.5.0.jar:5.5.0]
  582. at org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicasProxy.performOn(TransportReplicationAction.java:962) ~[elasticsearch-5.5.0.jar:5.5.0]
  583. at org.elasticsearch.action.support.replication.ReplicationOperation.performOnReplica(ReplicationOperation.java:186) ~[elasticsearch-5.5.0.jar:5.5.0]
  584. at org.elasticsearch.action.support.replication.ReplicationOperation.performOnReplicas(ReplicationOperation.java:170) ~[elasticsearch-5.5.0.jar:5.5.0]
  585. at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:130) ~[elasticsearch-5.5.0.jar:5.5.0]
  586. at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:322) ~[elasticsearch-5.5.0.jar:5.5.0]
  587. at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:264) ~[elasticsearch-5.5.0.jar:5.5.0]
  588. at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:888) ~[elasticsearch-5.5.0.jar:5.5.0]
  589. at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:885) ~[elasticsearch-5.5.0.jar:5.5.0]
  590. at org.elasticsearch.index.shard.IndexShardOperationsLock.acquire(IndexShardOperationsLock.java:147) ~[elasticsearch-5.5.0.jar:5.5.0]
  591. at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationLock(IndexShard.java:1657) ~[elasticsearch-5.5.0.jar:5.5.0]
  592. at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:897) ~[elasticsearch-5.5.0.jar:5.5.0]
  593. at org.elasticsearch.action.support.replication.TransportReplicationAction.access$400(TransportReplicationAction.java:93) ~[elasticsearch-5.5.0.jar:5.5.0]
  594. at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:281) ~[elasticsearch-5.5.0.jar:5.5.0]
  595. at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.5.0.jar:5.5.0]
  596. at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:260) ~[elasticsearch-5.5.0.jar:5.5.0]
  597. at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:252) ~[elasticsearch-5.5.0.jar:5.5.0]
  598. at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.5.0.jar:5.5.0]
  599. at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:644) ~[elasticsearch-5.5.0.jar:5.5.0]
  600. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) ~[elasticsearch-5.5.0.jar:5.5.0]
  601. at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.5.0.jar:5.5.0]
  602. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_131]
  603. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_131]
  604. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
  605. [2017-07-09T01:27:49,564][INFO ][o.e.c.r.DelayedAllocationService] [node3] scheduling reroute for delayed shards in [42s] (64 delayed shards)
  606. [2017-07-09T01:40:25,916][INFO ][o.e.m.j.JvmGcMonitorService] [node3] [gc][48408] overhead, spent [267ms] collecting in the last [1s]
  607. [2017-07-09T02:12:36,662][INFO ][o.e.m.j.JvmGcMonitorService] [node3] [gc][50336] overhead, spent [257ms] collecting in the last [1s]
  608. [2017-07-09T02:19:55,173][INFO ][o.e.m.j.JvmGcMonitorService] [node3] [gc][50774] overhead, spent [265ms] collecting in the last [1s]
  609. [2017-07-09T02:45:27,987][INFO ][o.e.m.j.JvmGcMonitorService] [node3] [gc][52305] overhead, spent [300ms] collecting in the last [1s]
  610. [2017-07-09T03:01:28,591][INFO ][o.e.m.j.JvmGcMonitorService] [node3] [gc][53264] overhead, spent [280ms] collecting in the last [1s]
  611. [2017-07-09T03:12:59,079][INFO ][o.e.m.j.JvmGcMonitorService] [node3] [gc][53953] overhead, spent [257ms] collecting in the last [1s]
  612. [2017-07-09T03:50:31,582][INFO ][o.e.m.j.JvmGcMonitorService] [node3] [gc][56202] overhead, spent [266ms] collecting in the last [1s]
  613. [2017-07-09T04:01:48,455][INFO ][o.e.m.j.JvmGcMonitorService] [node3] [gc][56878] overhead, spent [259ms] collecting in the last [1s]
  614. [2017-07-09T04:20:34,170][INFO ][o.e.m.j.JvmGcMonitorService] [node3] [gc][58002] overhead, spent [251ms] collecting in the last [1s]
  615. [2017-07-09T04:21:36,327][INFO ][o.e.m.j.JvmGcMonitorService] [node3] [gc][58064] overhead, spent [304ms] collecting in the last [1s]
  616. [2017-07-09T04:24:43,643][INFO ][o.e.m.j.JvmGcMonitorService] [node3] [gc][58251] overhead, spent [252ms] collecting in the last [1s]
  617. [2017-07-09T04:33:04,651][INFO ][o.e.m.j.JvmGcMonitorService] [node3] [gc][58751] overhead, spent [266ms] collecting in the last [1s]
  618. [2017-07-09T04:40:34,891][INFO ][o.e.m.j.JvmGcMonitorService] [node3] [gc][59200] overhead, spent [348ms] collecting in the last [1s]
  619. [2017-07-09T05:02:51,655][INFO ][o.e.m.j.JvmGcMonitorService] [node3] [gc][60532] overhead, spent [281ms] collecting in the last [1s]
  620. [2017-07-09T05:21:27,379][WARN ][i.n.c.AbstractChannel ] Force-closing a channel whose registration task was not accepted by an event loop: [id: 0xaf7c0ccc, L:/10.10.10.3:9300 - R:/10.10.10.1:44204]
  621. java.lang.OutOfMemoryError: unable to create new native thread
  622. at java.lang.Thread.start0(Native Method) ~[?:1.8.0_131]
  623. at java.lang.Thread.start(Thread.java:717) ~[?:1.8.0_131]
  624. at io.netty.util.concurrent.ThreadPerTaskExecutor.execute(ThreadPerTaskExecutor.java:33) ~[netty-common-4.1.11.Final.jar:4.1.11.Final]
  625. at io.netty.util.concurrent.SingleThreadEventExecutor.doStartThread(SingleThreadEventExecutor.java:847) ~[netty-common-4.1.11.Final.jar:4.1.11.Final]
  626. at io.netty.util.concurrent.SingleThreadEventExecutor.startThread(SingleThreadEventExecutor.java:840) ~[netty-common-4.1.11.Final.jar:4.1.11.Final]
  627. at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:745) ~[netty-common-4.1.11.Final.jar:4.1.11.Final]
  628. at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:475) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  629. at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:80) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  630. at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:74) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  631. at io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:86) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  632. at io.netty.bootstrap.ServerBootstrap$ServerBootstrapAcceptor.channelRead(ServerBootstrap.java:255) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  633. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  634. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  635. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  636. at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  637. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  638. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  639. at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  640. at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:93) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  641. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  642. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  643. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  644. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  645. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.11.Final.jar:4.1.11.Final]
  646. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
  647. [2017-07-09T05:21:27,381][WARN ][i.n.c.AbstractChannel ] Force-closing a channel whose registration task was not accepted by an event loop: [id: 0xedfaa904, L:/10.10.10.3:9300 - R:/10.10.10.1:44206]
  648. java.lang.OutOfMemoryError: unable to create new native thread
  649. at java.lang.Thread.start0(Native Method) ~[?:1.8.0_131]
  650. at java.lang.Thread.start(Thread.java:717) ~[?:1.8.0_131]
  651. at io.netty.util.concurrent.ThreadPerTaskExecutor.execute(ThreadPerTaskExecutor.java:33) ~[netty-common-4.1.11.Final.jar:4.1.11.Final]
  652. at io.netty.util.concurrent.SingleThreadEventExecutor.doStartThread(SingleThreadEventExecutor.java:847) ~[netty-common-4.1.11.Final.jar:4.1.11.Final]
  653. at io.netty.util.concurrent.SingleThreadEventExecutor.startThread(SingleThreadEventExecutor.java:840) ~[netty-common-4.1.11.Final.jar:4.1.11.Final]
  654. at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:745) ~[netty-common-4.1.11.Final.jar:4.1.11.Final]
  655. at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:475) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  656. at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:80) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  657. at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:74) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  658. at io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:86) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  659. at io.netty.bootstrap.ServerBootstrap$ServerBootstrapAcceptor.channelRead(ServerBootstrap.java:255) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  660. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  661. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  662. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  663. at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  664. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  665. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  666. at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  667. at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:93) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  668. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  669. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  670. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  671. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  672. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.11.Final.jar:4.1.11.Final]
  673. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
  674. [2017-07-09T05:21:27,381][WARN ][i.n.c.AbstractChannel ] Force-closing a channel whose registration task was not accepted by an event loop: [id: 0xc45892b0, L:/10.10.10.3:9300 - R:/10.10.10.1:44208]
  675. java.lang.OutOfMemoryError: unable to create new native thread
  676. at java.lang.Thread.start0(Native Method) ~[?:1.8.0_131]
  677. at java.lang.Thread.start(Thread.java:717) ~[?:1.8.0_131]
  678. at io.netty.util.concurrent.ThreadPerTaskExecutor.execute(ThreadPerTaskExecutor.java:33) ~[netty-common-4.1.11.Final.jar:4.1.11.Final]
  679. at io.netty.util.concurrent.SingleThreadEventExecutor.doStartThread(SingleThreadEventExecutor.java:847) ~[netty-common-4.1.11.Final.jar:4.1.11.Final]
  680. at io.netty.util.concurrent.SingleThreadEventExecutor.startThread(SingleThreadEventExecutor.java:840) ~[netty-common-4.1.11.Final.jar:4.1.11.Final]
  681. at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:745) ~[netty-common-4.1.11.Final.jar:4.1.11.Final]
  682. at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:475) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  683. at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:80) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  684. at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:74) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  685. at io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:86) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  686. at io.netty.bootstrap.ServerBootstrap$ServerBootstrapAcceptor.channelRead(ServerBootstrap.java:255) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  687. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  688. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  689. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  690. at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  691. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  692. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  693. at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  694. at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:93) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  695. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  696. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  697. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  698. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  699. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.11.Final.jar:4.1.11.Final]
  700. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
  701. [2017-07-09T05:21:27,382][WARN ][i.n.c.AbstractChannel ] Force-closing a channel whose registration task was not accepted by an event loop: [id: 0x775d83dc, L:/10.10.10.3:9300 - R:/10.10.10.1:44210]
  702. java.lang.OutOfMemoryError: unable to create new native thread
  703. at java.lang.Thread.start0(Native Method) ~[?:1.8.0_131]
  704. at java.lang.Thread.start(Thread.java:717) ~[?:1.8.0_131]
  705. at io.netty.util.concurrent.ThreadPerTaskExecutor.execute(ThreadPerTaskExecutor.java:33) ~[netty-common-4.1.11.Final.jar:4.1.11.Final]
  706. at io.netty.util.concurrent.SingleThreadEventExecutor.doStartThread(SingleThreadEventExecutor.java:847) ~[netty-common-4.1.11.Final.jar:4.1.11.Final]
  707. at io.netty.util.concurrent.SingleThreadEventExecutor.startThread(SingleThreadEventExecutor.java:840) ~[netty-common-4.1.11.Final.jar:4.1.11.Final]
  708. at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:745) ~[netty-common-4.1.11.Final.jar:4.1.11.Final]
  709. at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:475) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  710. at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:80) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  711. at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:74) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  712. at io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:86) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  713. at io.netty.bootstrap.ServerBootstrap$ServerBootstrapAcceptor.channelRead(ServerBootstrap.java:255) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  714. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  715. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  716. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  717. at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  718. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  719. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  720. at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  721. at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:93) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  722. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  723. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  724. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  725. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  726. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.11.Final.jar:4.1.11.Final]
  727. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
  728. [2017-07-09T05:21:27,383][WARN ][i.n.c.AbstractChannel ] Force-closing a channel whose registration task was not accepted by an event loop: [id: 0xa012ebbe, L:/10.10.10.3:9300 - R:/10.10.10.1:44212]
  729. java.lang.OutOfMemoryError: unable to create new native thread
  730. at java.lang.Thread.start0(Native Method) ~[?:1.8.0_131]
  731. at java.lang.Thread.start(Thread.java:717) ~[?:1.8.0_131]
  732. at io.netty.util.concurrent.ThreadPerTaskExecutor.execute(ThreadPerTaskExecutor.java:33) ~[netty-common-4.1.11.Final.jar:4.1.11.Final]
  733. at io.netty.util.concurrent.SingleThreadEventExecutor.doStartThread(SingleThreadEventExecutor.java:847) ~[netty-common-4.1.11.Final.jar:4.1.11.Final]
  734. at io.netty.util.concurrent.SingleThreadEventExecutor.startThread(SingleThreadEventExecutor.java:840) ~[netty-common-4.1.11.Final.jar:4.1.11.Final]
  735. at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:745) ~[netty-common-4.1.11.Final.jar:4.1.11.Final]
  736. at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:475) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  737. at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:80) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  738. at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:74) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  739. at io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:86) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  740. at io.netty.bootstrap.ServerBootstrap$ServerBootstrapAcceptor.channelRead(ServerBootstrap.java:255) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  741. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  742. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  743. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  744. at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  745. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  746. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  747. at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  748. at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:93) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  749. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  750. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  751. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  752. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  753. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.11.Final.jar:4.1.11.Final]
  754. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
  755. [2017-07-09T05:21:27,384][WARN ][i.n.c.AbstractChannel ] Force-closing a channel whose registration task was not accepted by an event loop: [id: 0xdbb867a2, L:/10.10.10.3:9300 - R:/10.10.10.1:44214]
  756. java.lang.OutOfMemoryError: unable to create new native thread
  757. at java.lang.Thread.start0(Native Method) ~[?:1.8.0_131]
  758. at java.lang.Thread.start(Thread.java:717) ~[?:1.8.0_131]
  759. at io.netty.util.concurrent.ThreadPerTaskExecutor.execute(ThreadPerTaskExecutor.java:33) ~[netty-common-4.1.11.Final.jar:4.1.11.Final]
  760. at io.netty.util.concurrent.SingleThreadEventExecutor.doStartThread(SingleThreadEventExecutor.java:847) ~[netty-common-4.1.11.Final.jar:4.1.11.Final]
  761. at io.netty.util.concurrent.SingleThreadEventExecutor.startThread(SingleThreadEventExecutor.java:840) ~[netty-common-4.1.11.Final.jar:4.1.11.Final]
  762. at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:745) ~[netty-common-4.1.11.Final.jar:4.1.11.Final]
  763. at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:475) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  764. at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:80) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  765. at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:74) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  766. at io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:86) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  767. at io.netty.bootstrap.ServerBootstrap$ServerBootstrapAcceptor.channelRead(ServerBootstrap.java:255) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  768. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  769. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  770. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  771. at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  772. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  773. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  774. at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  775. at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:93) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  776. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  777. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  778. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  779. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  780. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.11.Final.jar:4.1.11.Final]
  781. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
  782. [2017-07-09T05:21:27,384][WARN ][i.n.c.AbstractChannel ] Force-closing a channel whose registration task was not accepted by an event loop: [id: 0xc797f262, L:/10.10.10.3:9300 - R:/10.10.10.1:44218]
  783. java.lang.OutOfMemoryError: unable to create new native thread
  784. at java.lang.Thread.start0(Native Method) ~[?:1.8.0_131]
  785. at java.lang.Thread.start(Thread.java:717) ~[?:1.8.0_131]
  786. at io.netty.util.concurrent.ThreadPerTaskExecutor.execute(ThreadPerTaskExecutor.java:33) ~[netty-common-4.1.11.Final.jar:4.1.11.Final]
  787. at io.netty.util.concurrent.SingleThreadEventExecutor.doStartThread(SingleThreadEventExecutor.java:847) ~[netty-common-4.1.11.Final.jar:4.1.11.Final]
  788. at io.netty.util.concurrent.SingleThreadEventExecutor.startThread(SingleThreadEventExecutor.java:840) ~[netty-common-4.1.11.Final.jar:4.1.11.Final]
  789. at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:745) ~[netty-common-4.1.11.Final.jar:4.1.11.Final]
  790. at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:475) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  791. at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:80) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  792. at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:74) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  793. at io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:86) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  794. at io.netty.bootstrap.ServerBootstrap$ServerBootstrapAcceptor.channelRead(ServerBootstrap.java:255) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  795. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  796. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  797. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  798. at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  799. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  800. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  801. at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  802. at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:93) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  803. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  804. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  805. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  806. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  807. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.11.Final.jar:4.1.11.Final]
  808. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
  809. [2017-07-09T05:21:27,385][WARN ][i.n.c.AbstractChannel ] Force-closing a channel whose registration task was not accepted by an event loop: [id: 0x6aa66775, L:/10.10.10.3:9300 - R:/10.10.10.1:44216]
  810. java.lang.OutOfMemoryError: unable to create new native thread
  811. at java.lang.Thread.start0(Native Method) ~[?:1.8.0_131]
  812. at java.lang.Thread.start(Thread.java:717) ~[?:1.8.0_131]
  813. at io.netty.util.concurrent.ThreadPerTaskExecutor.execute(ThreadPerTaskExecutor.java:33) ~[netty-common-4.1.11.Final.jar:4.1.11.Final]
  814. at io.netty.util.concurrent.SingleThreadEventExecutor.doStartThread(SingleThreadEventExecutor.java:847) ~[netty-common-4.1.11.Final.jar:4.1.11.Final]
  815. at io.netty.util.concurrent.SingleThreadEventExecutor.startThread(SingleThreadEventExecutor.java:840) ~[netty-common-4.1.11.Final.jar:4.1.11.Final]
  816. at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:745) ~[netty-common-4.1.11.Final.jar:4.1.11.Final]
  817. at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:475) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  818. at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:80) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  819. at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:74) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  820. at io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:86) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  821. at io.netty.bootstrap.ServerBootstrap$ServerBootstrapAcceptor.channelRead(ServerBootstrap.java:255) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  822. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  823. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  824. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  825. at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  826. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  827. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  828. at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  829. at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:93) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  830. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  831. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  832. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  833. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
  834. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.11.Final.jar:4.1.11.Final]
  835. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
  836. [2017-07-09T05:21:27,763][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node3] fatal error in thread [elasticsearch[node3][refresh][T#3]], exiting
  837. java.lang.OutOfMemoryError: unable to create new native thread
  838. at java.lang.Thread.start0(Native Method) ~[?:1.8.0_131]
  839. at java.lang.Thread.start(Thread.java:717) ~[?:1.8.0_131]
  840. at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950) ~[?:1.8.0_131]
  841. at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1018) ~[?:1.8.0_131]
  842. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160) ~[?:1.8.0_131]
  843. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_131]
  844. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
  845. [2017-07-09T05:56:50,483][INFO ][o.e.n.Node ] [node3] initializing ...
  846. [2017-07-09T05:56:50,636][INFO ][o.e.e.NodeEnvironment ] [node3] using [1] data paths, mounts [[/opt/elasticsearch (/dev/md6)]], net usable_space [53.7tb], net total_space [65.2tb], spins? [possibly], types [ext4]
  847. [2017-07-09T05:56:50,636][INFO ][o.e.e.NodeEnvironment ] [node3] heap size [30.7gb], compressed ordinary object pointers [true]
  848. [2017-07-09T05:56:51,260][INFO ][o.e.n.Node ] [node3] node name [node3], node ID [HIwh8r97SReZqIVvlort-w]
  849. [2017-07-09T05:56:51,260][INFO ][o.e.n.Node ] [node3] version[5.5.0], pid[14419], build[260387d/2017-06-30T23:16:05.735Z], OS[Linux/4.9.0-0.bpo.3-amd64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_131/25.131-b11]
  850. [2017-07-09T05:56:51,261][INFO ][o.e.n.Node ] [node3] JVM arguments [-Xms31g, -Xmx31g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+DisableExplicitGC, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
  851. [2017-07-09T05:56:52,481][WARN ][o.e.x.XPackPlugin ] the [action.auto_create_index] setting is configured to be restrictive [.security,.monitoring*,.watches,.triggered_watches,.watcher-history*,logstash-*]. for the next 6 months daily history indices are allowed to be created, but please make sure that any future history indices after 6 months with the pattern [.watcher-history-YYYY.MM.dd] are allowed to be created
  852. [2017-07-09T05:56:52,500][INFO ][o.e.p.PluginsService ] [node3] loaded module [aggs-matrix-stats]
  853. [2017-07-09T05:56:52,500][INFO ][o.e.p.PluginsService ] [node3] loaded module [ingest-common]
  854. [2017-07-09T05:56:52,501][INFO ][o.e.p.PluginsService ] [node3] loaded module [lang-expression]
  855. [2017-07-09T05:56:52,501][INFO ][o.e.p.PluginsService ] [node3] loaded module [lang-groovy]
  856. [2017-07-09T05:56:52,501][INFO ][o.e.p.PluginsService ] [node3] loaded module [lang-mustache]
  857. [2017-07-09T05:56:52,501][INFO ][o.e.p.PluginsService ] [node3] loaded module [lang-painless]
  858. [2017-07-09T05:56:52,501][INFO ][o.e.p.PluginsService ] [node3] loaded module [parent-join]
  859. [2017-07-09T05:56:52,501][INFO ][o.e.p.PluginsService ] [node3] loaded module [percolator]
  860. [2017-07-09T05:56:52,501][INFO ][o.e.p.PluginsService ] [node3] loaded module [reindex]
  861. [2017-07-09T05:56:52,501][INFO ][o.e.p.PluginsService ] [node3] loaded module [transport-netty3]
  862. [2017-07-09T05:56:52,501][INFO ][o.e.p.PluginsService ] [node3] loaded module [transport-netty4]
  863. [2017-07-09T05:56:52,501][INFO ][o.e.p.PluginsService ] [node3] loaded plugin [x-pack]
  864. [2017-07-09T05:56:54,998][INFO ][o.e.d.DiscoveryModule ] [node3] using discovery type [zen]
  865. [2017-07-09T05:56:59,130][INFO ][o.e.n.Node ] [node3] initialized
  866. [2017-07-09T05:56:59,131][INFO ][o.e.n.Node ] [node3] starting ...
  867. [2017-07-09T05:56:59,465][INFO ][o.e.t.TransportService ] [node3] publish_address {10.10.10.3:9300}, bound_addresses {10.10.10.3:9300}
  868. [2017-07-09T05:56:59,475][INFO ][o.e.b.BootstrapChecks ] [node3] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
  869. [2017-07-09T05:57:02,563][INFO ][o.e.c.s.ClusterService ] [node3] new_master {node3}{---}{---}{10.10.10.1}{10.10.10.1:9300}{rack=rackA6-1}]
  870. [2017-07-09T05:57:02,635][INFO ][o.e.c.s.ClusterService ] [node3] added { {node5}{---}{---}{10.10.10.5}{10.10.10.5:9300}{rack=rackA6-1}]
  871. [2017-07-09T05:57:02,700][INFO ][o.e.h.n.Netty4HttpServerTransport] [node3] publish_address {10.10.10.3:9200}, bound_addresses {10.10.10.3:9200}
  872. [2017-07-09T05:57:02,700][INFO ][o.e.n.Node ] [node3] started
  873. [2017-07-09T05:57:06,175][INFO ][o.e.m.j.JvmGcMonitorService] [node3] [gc][7] overhead, spent [307ms] collecting in the last [1s]
  874. [2017-07-09T05:57:06,202][INFO ][o.e.c.s.ClusterSettings ] [node3] updating [cluster.routing.allocation.cluster_concurrent_rebalance] from [2] to [36]
  875. [2017-07-09T05:57:06,202][INFO ][o.e.c.s.ClusterSettings ] [node3] updating [cluster.routing.allocation.node_initial_primaries_recoveries] from [4] to [36]
  876. [2017-07-09T05:57:06,203][INFO ][o.e.c.s.ClusterSettings ] [node3] updating [cluster.routing.allocation.node_concurrent_incoming_recoveries] from [2] to [288]
  877. [2017-07-09T05:57:06,203][INFO ][o.e.c.s.ClusterSettings ] [node3] updating [cluster.routing.allocation.node_concurrent_outgoing_recoveries] from [2] to [288]
  878. [2017-07-09T05:57:06,203][INFO ][o.e.c.s.ClusterSettings ] [node3] updating [indices.recovery.max_bytes_per_sec] from [40mb] to [500mb]
  879. [2017-07-09T05:57:08,554][INFO ][o.e.l.LicenseService ] [node3] license [ea109dbc-eed3-4786-93a5-2954c7a57baf] mode [basic] - valid
  880. [2017-07-09T05:57:08,564][INFO ][o.e.g.GatewayService ] [node3] recovered [52] indices into cluster_state
  881. [2017-07-09T05:57:08,594][INFO ][o.e.c.s.ClusterService ] [node3] added { {node4}{---}{---}{10.10.10.2}{10.10.10.2:9300}{rack=rackA6-1}]
  882. [2017-07-09T05:57:13,286][INFO ][o.e.c.s.ClusterService ] [node3] added { {index_node2}{---}{---}{10.10.10.101}{10.10.10.101:9300}{rack=rackA6-1}]
  883. [2017-07-09T05:57:43,318][WARN ][o.e.d.z.PublishClusterStateAction] [node3] timed out waiting for all nodes to process published state [5] (timeout [30s], pending nodes: [ {node1}{---}{---}{10.10.10.6}{10.10.10.6:9300}{rack=rackA6-1}])
  884. [2017-07-09T05:58:42,536][WARN ][o.e.c.s.ClusterService ] [node3] cluster state update task [zen-disco-node-join[ {index_node2}{---}{---}{10.10.10.101}{10.10.10.101:9300}{rack=rackA6-1}]] took [1.4m] above the warn threshold of 30s
  885. [2017-07-09T05:58:42,544][INFO ][o.e.c.s.ClusterService ] [node3] added { {index_node1}{---}{---}{10.10.10.100}{10.10.10.100:9300}{rack=rackA6-1}]
  886. [2017-07-09T05:59:13,856][WARN ][o.e.d.z.PublishClusterStateAction] [node3] timed out waiting for all nodes to process published state [6] (timeout [30s], pending nodes: [ {node1}{---}{---}{10.10.10.5}{10.10.10.5:9300}{rack=rackA6-1}])
  887. [2017-07-09T05:59:43,146][INFO ][o.e.m.j.JvmGcMonitorService] [node3] [gc][153] overhead, spent [251ms] collecting in the last [1s]
  888. [2017-07-09T06:02:53,991][WARN ][o.e.c.s.ClusterService ] [node3] cluster state update task [zen-disco-node-join[ {index_node1}{---}{---}{10.10.10.100}{10.10.10.100:9300}{rack=rackA6-1}]] took [4.1m] above the warn threshold of 30s
  889. [2017-07-09T06:03:05,492][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  890. [2017-07-09T06:03:05,492][WARN ][o.e.c.a.s.ShardStateAction] [node3] [.monitoring-es-6-2017.07.09][0] received shard failed for shard id [[.monitoring-es-6-2017.07.09][0]], allocation id [bOPsxe-_TlagE3XGtp19Ng], primary term [2], message [mark copy as stale]
  891. [2017-07-09T06:03:05,492][WARN ][o.e.c.a.s.ShardStateAction] [node3] [.monitoring-es-6-2017.07.09][0] received shard failed for shard id [[.monitoring-es-6-2017.07.09][0]], allocation id [bOPsxe-_TlagE3XGtp19Ng], primary term [2], message [mark copy as stale]
  892. [2017-07-09T06:03:05,492][WARN ][o.e.c.a.s.ShardStateAction] [node3] [.monitoring-es-6-2017.07.09][0] received shard failed for shard id [[.monitoring-es-6-2017.07.09][0]], allocation id [bOPsxe-_TlagE3XGtp19Ng], primary term [2], message [mark copy as stale]
  893. [2017-07-09T06:03:05,492][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  894. [2017-07-09T06:03:05,493][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  895. [2017-07-09T06:03:05,525][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  896. [2017-07-09T06:03:05,525][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  897. [2017-07-09T06:03:05,525][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  898. [2017-07-09T06:03:05,525][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  899. [2017-07-09T06:03:05,525][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  900. [2017-07-09T06:03:05,525][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  901. [2017-07-09T06:03:05,595][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  902. [2017-07-09T06:03:05,715][WARN ][o.e.c.a.s.ShardStateAction] [node3] [.monitoring-es-6-2017.07.09][0] received shard failed for shard id [[.monitoring-es-6-2017.07.09][0]], allocation id [bOPsxe-_TlagE3XGtp19Ng], primary term [2], message [mark copy as stale]
  903. [2017-07-09T06:03:05,776][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  904. [2017-07-09T06:03:05,808][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  905. [2017-07-09T06:03:05,883][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  906. [2017-07-09T06:03:05,889][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  907. [2017-07-09T06:03:05,896][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  908. [2017-07-09T06:03:05,900][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  909. [2017-07-09T06:03:05,912][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  910. [2017-07-09T06:03:05,913][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  911. [2017-07-09T06:03:05,917][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  912. [2017-07-09T06:03:06,701][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  913. [2017-07-09T06:03:06,726][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  914. [2017-07-09T06:03:06,736][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  915. [2017-07-09T06:03:06,740][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  916. [2017-07-09T06:03:06,740][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  917. [2017-07-09T06:03:06,777][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  918. [2017-07-09T06:03:06,777][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  919. [2017-07-09T06:03:06,778][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  920. [2017-07-09T06:03:06,787][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  921. [2017-07-09T06:03:06,796][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  922. [2017-07-09T06:03:06,810][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  923. [2017-07-09T06:03:06,815][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  924. [2017-07-09T06:03:07,167][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  925. [2017-07-09T06:03:07,571][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  926. [2017-07-09T06:03:08,282][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  927. [2017-07-09T06:03:08,325][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  928. [2017-07-09T06:03:08,334][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  929. [2017-07-09T06:03:08,754][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  930. [2017-07-09T06:03:08,760][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  931. [2017-07-09T06:03:08,760][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  932. [2017-07-09T06:03:08,803][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  933. [2017-07-09T06:03:08,811][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  934. [2017-07-09T06:03:08,826][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  935. [2017-07-09T06:03:08,857][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  936. [2017-07-09T06:03:09,833][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  937. [2017-07-09T06:03:09,839][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  938. [2017-07-09T06:03:18,296][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  939. [2017-07-09T06:03:19,830][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  940. [2017-07-09T06:03:19,831][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  941. [2017-07-09T06:03:19,870][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  942. [2017-07-09T06:03:20,177][WARN ][o.e.c.a.s.ShardStateAction] [node3] [.monitoring-es-6-2017.07.09][0] received shard failed for shard id [[.monitoring-es-6-2017.07.09][0]], allocation id [bOPsxe-_TlagE3XGtp19Ng], primary term [2], message [mark copy as stale]
  943. [2017-07-09T06:03:20,280][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  944. [2017-07-09T06:03:21,440][WARN ][o.e.c.a.s.ShardStateAction] [node3] [.monitoring-es-6-2017.07.09][0] received shard failed for shard id [[.monitoring-es-6-2017.07.09][0]], allocation id [bOPsxe-_TlagE3XGtp19Ng], primary term [2], message [mark copy as stale]
  945. [2017-07-09T06:03:21,460][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  946. [2017-07-09T06:03:21,460][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  947. [2017-07-09T06:03:21,460][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  948. [2017-07-09T06:03:21,474][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  949. [2017-07-09T06:03:21,477][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  950. [2017-07-09T06:03:21,478][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  951. [2017-07-09T06:03:21,477][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  952. [2017-07-09T06:03:21,484][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  953. [2017-07-09T06:03:21,485][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  954. [2017-07-09T06:03:21,486][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  955. [2017-07-09T06:03:21,486][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  956. [2017-07-09T06:03:21,491][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  957. [2017-07-09T06:03:21,491][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  958. [2017-07-09T06:03:21,492][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  959. [2017-07-09T06:03:21,493][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  960. [2017-07-09T06:03:21,493][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  961. [2017-07-09T06:03:21,493][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  962. [2017-07-09T06:03:21,493][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  963. [2017-07-09T06:03:21,535][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  964. [2017-07-09T06:03:21,535][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  965. [2017-07-09T06:03:21,550][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  966. [2017-07-09T06:03:21,570][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  967. [2017-07-09T06:03:21,617][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  968. [2017-07-09T06:03:21,620][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  969. [2017-07-09T06:03:21,625][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  970. [2017-07-09T06:03:21,684][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  971. [2017-07-09T06:03:21,686][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  972. [2017-07-09T06:03:21,686][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  973. [2017-07-09T06:03:21,829][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  974. [2017-07-09T06:03:21,852][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  975. [2017-07-09T06:03:21,853][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  976. [2017-07-09T06:03:21,854][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  977. [2017-07-09T06:03:21,854][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  978. [2017-07-09T06:03:21,856][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  979. [2017-07-09T06:03:21,884][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  980. [2017-07-09T06:03:21,885][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  981. [2017-07-09T06:03:21,886][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  982. [2017-07-09T06:03:21,886][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  983. [2017-07-09T06:03:21,886][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  984. [2017-07-09T06:03:21,887][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  985. [2017-07-09T06:03:21,887][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  986. [2017-07-09T06:03:21,887][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  987. [2017-07-09T06:03:21,887][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  988. [2017-07-09T06:03:21,889][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  989. [2017-07-09T06:03:21,889][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  990. [2017-07-09T06:03:21,890][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  991. [2017-07-09T06:03:21,890][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  992. [2017-07-09T06:03:21,890][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  993. [2017-07-09T06:03:21,891][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  994. [2017-07-09T06:03:21,902][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  995. [2017-07-09T06:03:21,904][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  996. [2017-07-09T06:03:21,938][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  997. [2017-07-09T06:03:21,953][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  998. [2017-07-09T06:03:24,679][WARN ][o.e.d.z.PublishClusterStateAction] [node3] timed out waiting for all nodes to process published state [7] (timeout [30s], pending nodes: [ {node2}{---}{---}{10.10.10.2}{10.10.10.2:9300}{rack=rackA6-1}])
  999. [2017-07-09T06:03:25,732][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1000. [2017-07-09T06:03:25,732][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1001. [2017-07-09T06:03:25,735][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1002. [2017-07-09T06:03:25,736][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1003. [2017-07-09T06:03:25,854][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1004. [2017-07-09T06:03:25,983][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1005. [2017-07-09T06:03:25,983][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1006. [2017-07-09T06:03:25,990][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1007. [2017-07-09T06:03:25,994][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1008. [2017-07-09T06:03:25,996][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1009. [2017-07-09T06:03:26,274][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1010. [2017-07-09T06:03:26,281][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1011. [2017-07-09T06:03:26,458][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1012. [2017-07-09T06:03:26,464][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1013. [2017-07-09T06:03:29,144][WARN ][o.e.c.a.s.ShardStateAction] [node3] [.monitoring-es-6-2017.07.09][0] received shard failed for shard id [[.monitoring-es-6-2017.07.09][0]], allocation id [bOPsxe-_TlagE3XGtp19Ng], primary term [2], message [mark copy as stale]
  1014. [2017-07-09T06:03:40,358][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1015. [2017-07-09T06:03:40,362][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1016. [2017-07-09T06:03:40,365][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1017. [2017-07-09T06:03:40,381][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1018. [2017-07-09T06:03:48,140][WARN ][o.e.c.s.ClusterService ] [node3] cluster state update task [shard-started shard id [[logstash-2017.06.29][4]], allocation id [A8XBOq6HRqKxCFDWyVdwWA], primary term [0], message [after existing recovery][shard id [[logstash-2017.06.29][4]], allocation id [A8XBOq6HRqKxCFDWyVdwWA], primary term [0], message [after existing recovery]], shard-started shard id [[logstash-2017.06.23][3]], allocation id [bZ195U7jQLy5_tg8rJLd5w], primary term [0], message [after existing recovery][shard id [[logstash-2017.06.23][3]], allocation id [bZ195U7jQLy5_tg8rJLd5w], primary term [0], message [after existing recovery]], shard-started shard id [[logstash-2017.06.18][0]], allocation id [f2N-wAWFQEGGKBxiVy5OrQ], primary term [0], message [after existing recovery][shard id [[logstash-2017.06.18][0]], allocation id [f2N-wAWFQEGGKBxiVy5OrQ], primary term [0], message [after existing recovery]], shard-started shard id [[logstash-2017.07.07][0]], allocation id [BWCZ_xYjQtmY-nHGMOT-pw], primary term [0], message [after existing recovery][shard id [[logstash-2017.07.07][0]], allocation id [BWCZ_xYjQtmY-nHGMOT-pw], primary term [0], message [after existing recovery]], shard-started shard id [[logstash-2017.06.22][1]], allocation id [iZNLycM8SAS03Kfi5Z3NMQ], primary term [0], message [after existing recovery][shard id [[logstash-2017.06.22][1]], allocation id [iZNLycM8SAS03Kfi5Z3NMQ], primary term [0], message [after existing recovery]], shard-started shard id [[logstash-2017.06.16][3]], allocation id [2BSAQR8oRvi9AClW-FhOfA], primary term [0], message [after existing recovery][shard id [[logstash-2017.06.16][3]], allocation id [2BSAQR8oRvi9AClW-FhOfA], primary term [0], message [after existing recovery]], shard-started shard id [[logstash-2017.06.24][1]], allocation id [V5setYJMRHKFqI1yCc3CJA], primary term [0], message [after existing recovery][shard id [[logstash-2017.06.24][1]], allocation id [V5setYJMRHKFqI1yCc3CJA], primary term [0], message [after existing recovery]], shard-started shard id [[logstash-2017.07.02][4]], allocation id [fQuZjI9gStezlO2o5yGeWg], primary term [0], message [master {node3}{---}{---}{10.10.000.000
  1019. [2017-07-09T06:03:48,559][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1020. [2017-07-09T06:03:48,565][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1021. [2017-07-09T06:03:49,911][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1022. [2017-07-09T06:03:49,912][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1023. [2017-07-09T06:03:50,126][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1024. [2017-07-09T06:03:50,201][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1025. [2017-07-09T06:03:50,838][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1026. [2017-07-09T06:03:50,839][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1027. [2017-07-09T06:03:51,815][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1028. [2017-07-09T06:03:51,815][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1029. [2017-07-09T06:03:51,815][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1030. [2017-07-09T06:03:52,030][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1031. [2017-07-09T06:03:52,032][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1032. [2017-07-09T06:03:52,520][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1033. [2017-07-09T06:03:52,522][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1034. [2017-07-09T06:03:52,826][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1035. [2017-07-09T06:03:52,831][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1036. [2017-07-09T06:03:53,814][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1037. [2017-07-09T06:03:53,874][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1038. [2017-07-09T06:03:53,874][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1039. [2017-07-09T06:03:54,003][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1040. [2017-07-09T06:03:54,018][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1041. [2017-07-09T06:03:54,049][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1042. [2017-07-09T06:03:54,052][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1043. [2017-07-09T06:03:54,358][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1044. [2017-07-09T06:03:54,361][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1045. [2017-07-09T06:03:54,863][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1046. [2017-07-09T06:03:54,950][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1047. [2017-07-09T06:03:54,950][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1048. [2017-07-09T06:03:54,950][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1049. [2017-07-09T06:03:54,951][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1050. [2017-07-09T06:03:54,951][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1051. [2017-07-09T06:04:01,392][WARN ][o.e.x.w.WatcherLifeCycleService] [node3] failed to start watcher. please wait for the cluster to become ready or try to start Watcher manually
  1052. org.elasticsearch.ElasticsearchTimeoutException: java.util.concurrent.TimeoutException: Timeout waiting for task.
  1053. at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:71) ~[elasticsearch-5.5.0.jar:5.5.0]
  1054. at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:63) ~[elasticsearch-5.5.0.jar:5.5.0]
  1055. at org.elasticsearch.xpack.watcher.support.init.proxy.WatcherClientProxy.refresh(WatcherClientProxy.java:118) ~[?:?]
  1056. at org.elasticsearch.xpack.watcher.watch.WatchStore.loadWatches(WatchStore.java:319) ~[?:?]
  1057. at org.elasticsearch.xpack.watcher.watch.WatchStore.start(WatchStore.java:108) ~[?:?]
  1058. at org.elasticsearch.xpack.watcher.WatcherService.start(WatcherService.java:80) ~[?:?]
  1059. at org.elasticsearch.xpack.watcher.WatcherLifeCycleService.start(WatcherLifeCycleService.java:108) ~[?:?]
  1060. at org.elasticsearch.xpack.watcher.WatcherLifeCycleService.lambda$applyClusterState$1(WatcherLifeCycleService.java:144) ~[?:?]
  1061. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-5.5.0.jar:5.5.0]
  1062. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
  1063. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
  1064. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
  1065. Caused by: java.util.concurrent.TimeoutException: Timeout waiting for task.
  1066. at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:232) ~[elasticsearch-5.5.0.jar:5.5.0]
  1067. at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:67) ~[elasticsearch-5.5.0.jar:5.5.0]
  1068. at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:69) ~[elasticsearch-5.5.0.jar:5.5.0]
  1069. ... 11 more
  1070. [2017-07-09T06:04:15,823][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1071. [2017-07-09T06:04:15,963][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1072. [2017-07-09T06:04:15,981][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1073. [2017-07-09T06:04:16,061][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1074. [2017-07-09T06:04:16,063][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1075. [2017-07-09T06:04:16,066][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1076. [2017-07-09T06:04:16,067][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1077. [2017-07-09T06:04:16,098][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1078. [2017-07-09T06:04:16,098][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1079. [2017-07-09T06:04:16,167][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1080. [2017-07-09T06:04:16,171][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1081. [2017-07-09T06:04:16,277][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1082. [2017-07-09T06:04:16,277][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1083. [2017-07-09T06:04:16,428][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1084. [2017-07-09T06:04:16,624][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1085. [2017-07-09T06:04:16,710][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1086. [2017-07-09T06:04:16,712][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1087. [2017-07-09T06:04:16,978][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1088. [2017-07-09T06:04:16,980][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1089. [2017-07-09T06:04:17,348][INFO ][o.e.m.j.JvmGcMonitorService] [node3] [gc][411] overhead, spent [280ms] collecting in the last [1s]
  1090. [2017-07-09T06:04:17,824][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1091. [2017-07-09T06:04:17,827][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1092. [2017-07-09T06:04:17,934][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1093. [2017-07-09T06:04:17,939][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1094. [2017-07-09T06:04:17,943][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1095. [2017-07-09T06:04:17,945][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1096. [2017-07-09T06:04:17,968][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1097. [2017-07-09T06:04:17,971][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1098. [2017-07-09T06:04:18,160][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1099. [2017-07-09T06:04:18,164][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1100. [2017-07-09T06:04:18,380][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1101. [2017-07-09T06:04:18,384][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1102. [2017-07-09T06:04:18,393][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1103. [2017-07-09T06:04:18,393][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1104. [2017-07-09T06:04:29,604][WARN ][o.e.c.s.ClusterService ] [node3] cluster state update task [zen-disco-node-join[ {index_node2}{---}{---}{10.10.10.100}{10.10.10.100:9300}{rack=rackA6-1}]] took [41.4s] above the warn threshold of 30s
  1105. [2017-07-09T06:04:29,620][WARN ][o.e.c.a.s.ShardStateAction] [node3] [.watches][0] received shard failed for shard id [[.watches][0]], allocation id [doz3CeNyR-SguZmtbjmgtQ], primary term [8], message [mark copy as stale]
  1106. [2017-07-09T06:04:29,709][INFO ][o.e.c.r.a.AllocationService] [node3] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[logstash-2017.07.09][1], [rally-2017][2], [logstash-2017.07.09][4], [logstash-2017.07.09][3], [rally-2017][1], [logstash-2017.07.09][2], [.kibana][0], [logstash-2017.07.07][4], [logstash-2017.06.17][3], [logstash-2017.06.15][4]] ...]).
  1107. [2017-07-09T06:04:57,974][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1108. [2017-07-09T06:04:59,602][WARN ][o.e.x.w.WatcherLifeCycleService] [node3] failed to start watcher. please wait for the cluster to become ready or try to start Watcher manually
  1109. org.elasticsearch.ElasticsearchTimeoutException: java.util.concurrent.TimeoutException: Timeout waiting for task.
  1110. at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:71) ~[elasticsearch-5.5.0.jar:5.5.0]
  1111. at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:63) ~[elasticsearch-5.5.0.jar:5.5.0]
  1112. at org.elasticsearch.xpack.watcher.support.init.proxy.WatcherClientProxy.refresh(WatcherClientProxy.java:118) ~[?:?]
  1113. at org.elasticsearch.xpack.watcher.watch.WatchStore.loadWatches(WatchStore.java:319) ~[?:?]
  1114. at org.elasticsearch.xpack.watcher.watch.WatchStore.start(WatchStore.java:108) ~[?:?]
  1115. at org.elasticsearch.xpack.watcher.WatcherService.start(WatcherService.java:80) ~[?:?]
  1116. at org.elasticsearch.xpack.watcher.WatcherLifeCycleService.start(WatcherLifeCycleService.java:108) ~[?:?]
  1117. at org.elasticsearch.xpack.watcher.WatcherLifeCycleService.lambda$applyClusterState$1(WatcherLifeCycleService.java:144) ~[?:?]
  1118. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-5.5.0.jar:5.5.0]
  1119. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
  1120. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
  1121. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
  1122. Caused by: java.util.concurrent.TimeoutException: Timeout waiting for task.
  1123. at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:232) ~[elasticsearch-5.5.0.jar:5.5.0]
  1124. at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:67) ~[elasticsearch-5.5.0.jar:5.5.0]
  1125. at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:69) ~[elasticsearch-5.5.0.jar:5.5.0]
  1126. ... 11 more
  1127. [2017-07-09T06:04:59,712][WARN ][o.e.d.z.PublishClusterStateAction] [node3] timed out waiting for all nodes to process published state [9] (timeout [30s], pending nodes: [ {node5}{---}{---}{10.10.10.5}{10.10.10.5:9300}{rack=rackA6-1}])
  1128. [2017-07-09T06:05:02,809][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1129. [2017-07-09T06:05:03,802][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1130. [2017-07-09T06:05:03,851][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1131. [2017-07-09T06:05:04,012][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1132. [2017-07-09T06:05:09,437][WARN ][o.e.c.a.s.ShardStateAction] [node3] [.watches][0] received shard failed for shard id [[.watches][0]], allocation id [doz3CeNyR-SguZmtbjmgtQ], primary term [8], message [mark copy as stale]
  1133. [2017-07-09T06:05:10,681][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1134. [2017-07-09T06:05:10,698][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1135. [2017-07-09T06:05:10,709][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1136. [2017-07-09T06:05:10,774][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1137. [2017-07-09T06:05:10,832][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1138. [2017-07-09T06:05:10,838][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1139. [2017-07-09T06:05:10,840][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1140. [2017-07-09T06:05:10,840][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1141. [2017-07-09T06:05:10,841][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1142. [2017-07-09T06:05:10,877][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1143. [2017-07-09T06:05:10,884][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1144. [2017-07-09T06:05:10,889][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1145. [2017-07-09T06:05:10,896][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1146. [2017-07-09T06:05:11,200][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1147. [2017-07-09T06:05:11,200][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1148. [2017-07-09T06:05:11,200][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1149. [2017-07-09T06:05:11,200][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1150. [2017-07-09T06:05:11,200][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1151. [2017-07-09T06:05:11,200][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1152. [2017-07-09T06:05:11,200][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1153. [2017-07-09T06:05:11,201][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1154. [2017-07-09T06:05:11,200][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1155. [2017-07-09T06:05:11,201][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1156. [2017-07-09T06:05:11,200][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1157. [2017-07-09T06:05:11,293][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1158. [2017-07-09T06:05:11,555][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1159. [2017-07-09T06:05:11,627][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1160. [2017-07-09T06:05:11,812][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1161. [2017-07-09T06:05:12,426][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1162. [2017-07-09T06:05:12,428][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1163. [2017-07-09T06:05:12,429][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1164. [2017-07-09T06:05:12,753][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1165. [2017-07-09T06:05:12,764][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1166. [2017-07-09T06:05:12,775][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1167. [2017-07-09T06:05:12,805][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1168. [2017-07-09T06:05:12,806][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1169. [2017-07-09T06:05:12,886][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1170. [2017-07-09T06:05:12,949][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1171. [2017-07-09T06:05:12,959][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1172. [2017-07-09T06:05:12,980][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1173. [2017-07-09T06:05:13,460][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1174. [2017-07-09T06:05:13,463][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1175. [2017-07-09T06:05:13,466][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1176. [2017-07-09T06:05:13,905][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1177. [2017-07-09T06:05:13,905][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1178. [2017-07-09T06:05:13,970][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1179. [2017-07-09T06:05:13,972][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1180. [2017-07-09T06:05:14,000][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1181. [2017-07-09T06:05:14,035][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1182. [2017-07-09T06:05:14,039][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1183. [2017-07-09T06:05:14,075][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1184. [2017-07-09T06:05:14,084][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1185. [2017-07-09T06:05:15,007][WARN ][o.e.c.s.ClusterService ] [node3] cluster state update task [shard-started shard id [[.kibana][0]], allocation id [7mWqTni4Shm5i-iToRLoPw], primary term [0], message [after existing recovery][shard id [[.kibana][0]], allocation id [7mWqTni4Shm5i-iToRLoPw], primary term [0], message [after existing recovery]], shard-started shard id [[logstash-2017.07.09][1]], allocation id [42JzlxJrQm6kzZfS4oVToQ], primary term [0], message [after existing recovery][shard id [[logstash-2017.07.09][1]], allocation id [42JzlxJrQm6kzZfS4oVToQ], primary term [0], message [after existing recovery]], shard-started shard id [[logstash-2017.07.09][3]], allocation id [C5gTDAxZRne_j1nDBITuXQ], primary term [0], message [after existing recovery][shard id [[logstash-2017.07.09][3]], allocation id [C5gTDAxZRne_j1nDBITuXQ], primary term [0], message [after existing recovery]], shard-started shard id [[.kibana][0]], allocation id [7mWqTni4Shm5i-iToRLoPw], primary term [0], message [master {node3}{---}{---}{10.10.10.3}{10.10.10.3:9300}{rack=rackA6-1} marked shard as initializing, but shard state is [POST_RECOVERY], mark shard as started]], shard-started shard id [[logstash-2017.06.14][1]], allocation id [SnYnea-VRy2v1mW86oNA8Q], primary term [0], message [after existing recovery][shard id [[logstash-2017.06.14][1]], allocation id [SnYnea-VRy2v1mW86oNA8Q], primary term [0], message [after existing recovery]], shard-started shard id [[rally-2017][2]], allocation id [dyfLmQLCRmacb8r39suoqg], primary term [0], message [after existing recovery][shard id [[rally-2017][2]], allocation id [dyfLmQLCRmacb8r39suoqg], primary term [0], message [after existing recovery]]] took [45.4s] above the warn threshold of 30s
  1186. [2017-07-09T06:05:22,626][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1187. [2017-07-09T06:05:22,627][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1188. [2017-07-09T06:05:22,923][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1189. [2017-07-09T06:05:22,933][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [sUWLoYOBRNWmM4-KjBKgeg], primary term [3], message [mark copy as stale]
  1190. [2017-07-09T06:05:22,935][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1191. [2017-07-09T06:05:23,829][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1192. [2017-07-09T06:05:24,107][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1193. [2017-07-09T06:05:24,107][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1194. [2017-07-09T06:05:24,107][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1195. [2017-07-09T06:05:24,189][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [sUWLoYOBRNWmM4-KjBKgeg], primary term [3], message [mark copy as stale]
  1196. [2017-07-09T06:05:24,232][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1197. [2017-07-09T06:05:24,235][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1198. [2017-07-09T06:05:24,551][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1199. [2017-07-09T06:05:24,552][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1200. [2017-07-09T06:05:24,552][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1201. [2017-07-09T06:05:24,552][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [sUWLoYOBRNWmM4-KjBKgeg], primary term [3], message [mark copy as stale]
  1202. [2017-07-09T06:05:24,564][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1203. [2017-07-09T06:05:24,598][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [sUWLoYOBRNWmM4-KjBKgeg], primary term [3], message [mark copy as stale]
  1204. [2017-07-09T06:05:24,551][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1205. [2017-07-09T06:05:24,946][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1206. [2017-07-09T06:05:24,948][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1207. [2017-07-09T06:05:24,955][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1208. [2017-07-09T06:05:24,961][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1209. [2017-07-09T06:05:26,042][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1210. [2017-07-09T06:05:26,042][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1211. [2017-07-09T06:05:26,042][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1212. [2017-07-09T06:05:26,042][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1213. [2017-07-09T06:05:26,363][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1214. [2017-07-09T06:05:26,363][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1215. [2017-07-09T06:05:26,377][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1216. [2017-07-09T06:05:26,378][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1217. [2017-07-09T06:05:26,624][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [sUWLoYOBRNWmM4-KjBKgeg], primary term [3], message [mark copy as stale]
  1218. [2017-07-09T06:05:26,624][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [sUWLoYOBRNWmM4-KjBKgeg], primary term [3], message [mark copy as stale]
  1219. [2017-07-09T06:05:26,624][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1220. [2017-07-09T06:05:26,624][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [sUWLoYOBRNWmM4-KjBKgeg], primary term [3], message [mark copy as stale]
  1221. [2017-07-09T06:05:27,401][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1222. [2017-07-09T06:05:27,410][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1223. [2017-07-09T06:05:27,419][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1224. [2017-07-09T06:05:28,962][WARN ][o.e.c.a.s.ShardStateAction] [node3] [.monitoring-kibana-6-2017.07.09][0] received shard failed for shard id [[.monitoring-kibana-6-2017.07.09][0]], allocation id [i5ged9XERViJ0IqoMBWRYA], primary term [2], message [mark copy as stale]
  1225. [2017-07-09T06:05:29,772][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1226. [2017-07-09T06:05:29,773][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [sUWLoYOBRNWmM4-KjBKgeg], primary term [3], message [mark copy as stale]
  1227. [2017-07-09T06:05:29,774][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1228. [2017-07-09T06:05:29,774][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1229. [2017-07-09T06:05:29,813][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1230. [2017-07-09T06:05:29,864][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1231. [2017-07-09T06:05:29,910][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1232. [2017-07-09T06:05:29,957][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1233. [2017-07-09T06:05:30,067][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1234. [2017-07-09T06:05:30,086][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1235. [2017-07-09T06:05:30,088][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1236. [2017-07-09T06:05:30,104][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1237. [2017-07-09T06:05:30,104][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1238. [2017-07-09T06:05:30,105][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1239. [2017-07-09T06:05:30,110][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1240. [2017-07-09T06:05:30,118][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1241. [2017-07-09T06:05:30,199][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1242. [2017-07-09T06:05:30,201][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1243. [2017-07-09T06:05:30,202][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1244. [2017-07-09T06:05:30,243][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1245. [2017-07-09T06:05:30,540][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1246. [2017-07-09T06:05:30,540][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1247. [2017-07-09T06:05:30,540][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [sUWLoYOBRNWmM4-KjBKgeg], primary term [3], message [mark copy as stale]
  1248. [2017-07-09T06:05:30,540][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [sUWLoYOBRNWmM4-KjBKgeg], primary term [3], message [mark copy as stale]
  1249. [2017-07-09T06:05:30,554][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1250. [2017-07-09T06:05:30,570][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1251. [2017-07-09T06:05:30,576][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1252. [2017-07-09T06:05:30,578][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1253. [2017-07-09T06:05:30,579][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [sUWLoYOBRNWmM4-KjBKgeg], primary term [3], message [mark copy as stale]
  1254. [2017-07-09T06:05:30,581][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1255. [2017-07-09T06:05:30,581][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [sUWLoYOBRNWmM4-KjBKgeg], primary term [3], message [mark copy as stale]
  1256. [2017-07-09T06:05:31,717][WARN ][o.e.c.a.s.ShardStateAction] [node3] [.monitoring-kibana-6-2017.07.09][0] received shard failed for shard id [[.monitoring-kibana-6-2017.07.09][0]], allocation id [i5ged9XERViJ0IqoMBWRYA], primary term [2], message [mark copy as stale]
  1257. [2017-07-09T06:05:38,138][WARN ][o.e.c.a.s.ShardStateAction] [node3] [.monitoring-kibana-6-2017.07.09][0] received shard failed for shard id [[.monitoring-kibana-6-2017.07.09][0]], allocation id [i5ged9XERViJ0IqoMBWRYA], primary term [2], message [mark copy as stale]
  1258. [2017-07-09T06:05:39,105][WARN ][o.e.x.w.WatcherLifeCycleService] [node3] failed to start watcher. please wait for the cluster to become ready or try to start Watcher manually
  1259. org.elasticsearch.ElasticsearchTimeoutException: java.util.concurrent.TimeoutException: Timeout waiting for task.
  1260. at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:71) ~[elasticsearch-5.5.0.jar:5.5.0]
  1261. at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:63) ~[elasticsearch-5.5.0.jar:5.5.0]
  1262. at org.elasticsearch.xpack.watcher.support.init.proxy.WatcherClientProxy.refresh(WatcherClientProxy.java:118) ~[?:?]
  1263. at org.elasticsearch.xpack.watcher.watch.WatchStore.loadWatches(WatchStore.java:319) ~[?:?]
  1264. at org.elasticsearch.xpack.watcher.watch.WatchStore.start(WatchStore.java:108) ~[?:?]
  1265. at org.elasticsearch.xpack.watcher.WatcherService.start(WatcherService.java:80) ~[?:?]
  1266. at org.elasticsearch.xpack.watcher.WatcherLifeCycleService.start(WatcherLifeCycleService.java:108) ~[?:?]
  1267. at org.elasticsearch.xpack.watcher.WatcherLifeCycleService.lambda$applyClusterState$1(WatcherLifeCycleService.java:144) ~[?:?]
  1268. at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-5.5.0.jar:5.5.0]
  1269. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
  1270. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
  1271. at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
  1272. Caused by: java.util.concurrent.TimeoutException: Timeout waiting for task.
  1273. at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:232) ~[elasticsearch-5.5.0.jar:5.5.0]
  1274. at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:67) ~[elasticsearch-5.5.0.jar:5.5.0]
  1275. at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:69) ~[elasticsearch-5.5.0.jar:5.5.0]
  1276. ... 11 more
  1277. [2017-07-09T06:05:42,714][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1278. [2017-07-09T06:05:44,520][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [sUWLoYOBRNWmM4-KjBKgeg], primary term [3], message [mark copy as stale]
  1279. [2017-07-09T06:05:44,520][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [sUWLoYOBRNWmM4-KjBKgeg], primary term [3], message [mark copy as stale]
  1280. [2017-07-09T06:05:44,520][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1281. [2017-07-09T06:05:44,520][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1282. [2017-07-09T06:05:44,521][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1283. [2017-07-09T06:05:44,521][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1284. [2017-07-09T06:05:44,834][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1285. [2017-07-09T06:05:44,834][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1286. [2017-07-09T06:05:45,356][WARN ][o.e.d.z.PublishClusterStateAction] [node3] timed out waiting for all nodes to process published state [10] (timeout [30s], pending nodes: [ {node1}{---}{---}{10.10.10.5}{10.10.10.5:9300}{rack=rackA6-1}])
  1287. [2017-07-09T06:05:45,981][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1288. [2017-07-09T06:05:45,992][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1289. [2017-07-09T06:05:47,636][WARN ][o.e.c.a.s.ShardStateAction] [node3] [.monitoring-kibana-6-2017.07.09][0] received shard failed for shard id [[.monitoring-kibana-6-2017.07.09][0]], allocation id [i5ged9XERViJ0IqoMBWRYA], primary term [2], message [mark copy as stale]
  1290. [2017-07-09T06:05:47,923][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1291. [2017-07-09T06:05:47,924][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1292. [2017-07-09T06:05:47,925][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1293. [2017-07-09T06:05:47,924][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1294. [2017-07-09T06:05:47,924][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1295. [2017-07-09T06:05:48,041][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1296. [2017-07-09T06:05:48,043][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1297. [2017-07-09T06:05:48,045][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [sUWLoYOBRNWmM4-KjBKgeg], primary term [3], message [mark copy as stale]
  1298. [2017-07-09T06:05:48,160][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1299. [2017-07-09T06:05:48,161][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [sUWLoYOBRNWmM4-KjBKgeg], primary term [3], message [mark copy as stale]
  1300. [2017-07-09T06:05:48,169][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1301. [2017-07-09T06:05:48,170][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1302. [2017-07-09T06:05:48,173][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [sUWLoYOBRNWmM4-KjBKgeg], primary term [3], message [mark copy as stale]
  1303. [2017-07-09T06:05:48,174][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1304. [2017-07-09T06:05:48,176][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1305. [2017-07-09T06:05:48,176][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1306. [2017-07-09T06:05:48,201][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1307. [2017-07-09T06:05:48,201][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [sUWLoYOBRNWmM4-KjBKgeg], primary term [3], message [mark copy as stale]
  1308. [2017-07-09T06:05:48,201][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1309. [2017-07-09T06:05:48,237][WARN ][o.e.c.a.s.ShardStateAction] [node3] [.monitoring-kibana-6-2017.07.09][0] received shard failed for shard id [[.monitoring-kibana-6-2017.07.09][0]], allocation id [i5ged9XERViJ0IqoMBWRYA], primary term [2], message [mark copy as stale]
  1310. [2017-07-09T06:05:48,265][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1311. [2017-07-09T06:05:48,265][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1312. [2017-07-09T06:05:48,267][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1313. [2017-07-09T06:05:48,267][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1314. [2017-07-09T06:05:48,269][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1315. [2017-07-09T06:05:48,273][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1316. [2017-07-09T06:05:48,274][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1317. [2017-07-09T06:05:48,278][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [sUWLoYOBRNWmM4-KjBKgeg], primary term [3], message [mark copy as stale]
  1318. [2017-07-09T06:05:48,280][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1319. [2017-07-09T06:05:48,280][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1320. [2017-07-09T06:05:48,574][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1321. [2017-07-09T06:05:48,574][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1322. [2017-07-09T06:05:48,574][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [sUWLoYOBRNWmM4-KjBKgeg], primary term [3], message [mark copy as stale]
  1323. [2017-07-09T06:05:48,574][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1324. [2017-07-09T06:05:48,574][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1325. [2017-07-09T06:05:48,574][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1326. [2017-07-09T06:05:48,574][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1327. [2017-07-09T06:05:48,574][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [sUWLoYOBRNWmM4-KjBKgeg], primary term [3], message [mark copy as stale]
  1328. [2017-07-09T06:05:48,576][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1329. [2017-07-09T06:05:48,619][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1330. [2017-07-09T06:05:49,029][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1331. [2017-07-09T06:05:49,029][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1332. [2017-07-09T06:05:49,029][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [sUWLoYOBRNWmM4-KjBKgeg], primary term [3], message [mark copy as stale]
  1333. [2017-07-09T06:05:49,029][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1334. [2017-07-09T06:05:49,042][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1335. [2017-07-09T06:05:49,091][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1336. [2017-07-09T06:05:49,093][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1337. [2017-07-09T06:05:49,093][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1338. [2017-07-09T06:05:49,095][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1339. [2017-07-09T06:05:49,097][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [sUWLoYOBRNWmM4-KjBKgeg], primary term [3], message [mark copy as stale]
  1340. [2017-07-09T06:05:49,893][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1341. [2017-07-09T06:05:49,894][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1342. [2017-07-09T06:05:49,897][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1343. [2017-07-09T06:05:49,905][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [sUWLoYOBRNWmM4-KjBKgeg], primary term [3], message [mark copy as stale]
  1344. [2017-07-09T06:05:49,909][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1345. [2017-07-09T06:05:50,212][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1346. [2017-07-09T06:05:50,212][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1347. [2017-07-09T06:05:50,212][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1348. [2017-07-09T06:05:50,212][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1349. [2017-07-09T06:05:50,212][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1350. [2017-07-09T06:05:50,212][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1351. [2017-07-09T06:05:50,320][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1352. [2017-07-09T06:05:50,322][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1353. [2017-07-09T06:05:50,320][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1354. [2017-07-09T06:05:50,212][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [sUWLoYOBRNWmM4-KjBKgeg], primary term [3], message [mark copy as stale]
  1355. [2017-07-09T06:05:50,320][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [sUWLoYOBRNWmM4-KjBKgeg], primary term [3], message [mark copy as stale]
  1356. [2017-07-09T06:05:50,212][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [sUWLoYOBRNWmM4-KjBKgeg], primary term [3], message [mark copy as stale]
  1357. [2017-07-09T06:05:50,320][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1358. [2017-07-09T06:05:50,324][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1359. [2017-07-09T06:05:50,324][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [sUWLoYOBRNWmM4-KjBKgeg], primary term [3], message [mark copy as stale]
  1360. [2017-07-09T06:05:50,424][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1361. [2017-07-09T06:05:50,614][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1362. [2017-07-09T06:05:50,614][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1363. [2017-07-09T06:05:50,614][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [sUWLoYOBRNWmM4-KjBKgeg], primary term [3], message [mark copy as stale]
  1364. [2017-07-09T06:05:50,614][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1365. [2017-07-09T06:05:50,614][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [sUWLoYOBRNWmM4-KjBKgeg], primary term [3], message [mark copy as stale]
  1366. [2017-07-09T06:05:50,614][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1367. [2017-07-09T06:05:50,614][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1368. [2017-07-09T06:05:50,614][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1369. [2017-07-09T06:05:50,614][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1370. [2017-07-09T06:05:50,781][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1371. [2017-07-09T06:05:50,792][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1372. [2017-07-09T06:05:51,091][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1373. [2017-07-09T06:05:51,104][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1374. [2017-07-09T06:05:51,113][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1375. [2017-07-09T06:05:52,728][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1376. [2017-07-09T06:05:52,728][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1377. [2017-07-09T06:05:52,730][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [sUWLoYOBRNWmM4-KjBKgeg], primary term [3], message [mark copy as stale]
  1378. [2017-07-09T06:05:52,730][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1379. [2017-07-09T06:05:52,790][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1380. [2017-07-09T06:05:54,051][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1381. [2017-07-09T06:05:54,058][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1382. [2017-07-09T06:05:54,061][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1383. [2017-07-09T06:05:54,070][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1384. [2017-07-09T06:05:54,133][WARN ][o.e.c.a.s.ShardStateAction] [node3] [.monitoring-kibana-6-2017.07.09][0] received shard failed for shard id [[.monitoring-kibana-6-2017.07.09][0]], allocation id [i5ged9XERViJ0IqoMBWRYA], primary term [2], message [mark copy as stale]
  1385. [2017-07-09T06:05:54,378][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1386. [2017-07-09T06:05:54,378][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1387. [2017-07-09T06:05:54,378][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1388. [2017-07-09T06:05:54,378][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [sUWLoYOBRNWmM4-KjBKgeg], primary term [3], message [mark copy as stale]
  1389. [2017-07-09T06:05:54,378][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [sUWLoYOBRNWmM4-KjBKgeg], primary term [3], message [mark copy as stale]
  1390. [2017-07-09T06:05:54,570][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1391. [2017-07-09T06:05:55,071][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [sUWLoYOBRNWmM4-KjBKgeg], primary term [3], message [mark copy as stale]
  1392. [2017-07-09T06:05:55,071][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1393. [2017-07-09T06:05:55,071][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1394. [2017-07-09T06:05:55,088][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1395. [2017-07-09T06:05:55,243][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1396. [2017-07-09T06:05:55,783][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1397. [2017-07-09T06:05:55,784][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1398. [2017-07-09T06:05:55,872][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1399. [2017-07-09T06:05:55,873][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1400. [2017-07-09T06:05:55,873][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [sUWLoYOBRNWmM4-KjBKgeg], primary term [3], message [mark copy as stale]
  1401. [2017-07-09T06:05:56,289][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [sUWLoYOBRNWmM4-KjBKgeg], primary term [3], message [mark copy as stale]
  1402. [2017-07-09T06:05:56,289][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1403. [2017-07-09T06:05:56,289][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1404. [2017-07-09T06:05:56,353][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1405. [2017-07-09T06:05:56,355][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1406. [2017-07-09T06:05:56,402][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1407. [2017-07-09T06:05:56,667][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1408. [2017-07-09T06:05:56,669][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1409. [2017-07-09T06:05:56,672][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1410. [2017-07-09T06:05:56,674][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1411. [2017-07-09T06:05:56,691][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [sUWLoYOBRNWmM4-KjBKgeg], primary term [3], message [mark copy as stale]
  1412. [2017-07-09T06:05:57,155][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1413. [2017-07-09T06:05:57,155][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1414. [2017-07-09T06:05:57,155][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [sUWLoYOBRNWmM4-KjBKgeg], primary term [3], message [mark copy as stale]
  1415. [2017-07-09T06:05:57,155][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1416. [2017-07-09T06:05:57,178][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1417. [2017-07-09T06:05:57,699][WARN ][o.e.c.a.s.ShardStateAction] [node3] [.monitoring-kibana-6-2017.07.09][0] received shard failed for shard id [[.monitoring-kibana-6-2017.07.09][0]], allocation id [i5ged9XERViJ0IqoMBWRYA], primary term [2], message [mark copy as stale]
  1418. [2017-07-09T06:05:58,057][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [sUWLoYOBRNWmM4-KjBKgeg], primary term [3], message [mark copy as stale]
  1419. [2017-07-09T06:05:58,057][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1420. [2017-07-09T06:05:58,057][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1421. [2017-07-09T06:05:58,057][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1422. [2017-07-09T06:05:58,057][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1423. [2017-07-09T06:05:58,057][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [sUWLoYOBRNWmM4-KjBKgeg], primary term [3], message [mark copy as stale]
  1424. [2017-07-09T06:05:58,057][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1425. [2017-07-09T06:05:58,057][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [sUWLoYOBRNWmM4-KjBKgeg], primary term [3], message [mark copy as stale]
  1426. [2017-07-09T06:05:58,057][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1427. [2017-07-09T06:05:58,085][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1428. [2017-07-09T06:05:58,144][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1429. [2017-07-09T06:05:58,148][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1430. [2017-07-09T06:05:58,150][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1431. [2017-07-09T06:05:58,151][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1432. [2017-07-09T06:05:58,168][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1433. [2017-07-09T06:05:58,169][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1434. [2017-07-09T06:05:58,172][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1435. [2017-07-09T06:05:58,276][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1436. [2017-07-09T06:05:58,299][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1437. [2017-07-09T06:05:58,299][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [sUWLoYOBRNWmM4-KjBKgeg], primary term [3], message [mark copy as stale]
  1438. [2017-07-09T06:05:58,310][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [sUWLoYOBRNWmM4-KjBKgeg], primary term [3], message [mark copy as stale]
  1439. [2017-07-09T06:05:58,498][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][0] received shard failed for shard id [[logstash-2017.07.09][0]], allocation id [yv_WhIeVQmef_TmrrRlzfA], primary term [2], message [mark copy as stale]
  1440. [2017-07-09T06:05:58,504][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
  1441. [2017-07-09T06:05:58,508][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][1] received shard failed for shard id [[logstash-2017.07.09][1]], allocation id [k6z_l-YwTL-W6UIP5XHMIg], primary term [2], message [mark copy as stale]
  1442. [2017-07-09T06:05:58,508][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][3] received shard failed for shard id [[logstash-2017.07.09][3]], allocation id [sUWLoYOBRNWmM4-KjBKgeg], primary term [3], message [mark copy as stale]
  1443. [2017-07-09T06:05:58,542][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][5] received shard failed for shard id [[logstash-2017.07.09][5]], allocation id [1O_mDDuKSYmCUM6ChemqIA], primary term [2], message [mark copy as stale]
  1444. [2017-07-09T06:05:58,991][WARN ][o.e.c.a.s.ShardStateAction] [node3] [logstash-2017.07.09][4] received shard failed for shard id [[logstash-2017.07.09][4]], allocation id [nzxveS8uRsCoVbwmeOjLXA], primary term [2], message [mark copy as stale]
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!