API tools faq
paste
Advertisement
Guest User

elasticsearch logs

a guest
Jul 1st, 2019
454
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
JSON 238.32 KB | None | 0 0
raw download clone embed print report
  1. OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
  2. {"type": "server", "timestamp": "2019-07-01T18:18:38,187+0000", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "using [1] data paths, mounts [[/usr/share/elasticsearch/data (fs-20c9ba59.efs.us-east-2.amazonaws.com://elasticsearch)]], net usable_space [8191.9pb], net total_space [8191.9pb], types [nfs4]"  }
  3. {"type": "server", "timestamp": "2019-07-01T18:18:38,198+0000", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "heap size [1.9gb], compressed ordinary object pointers [true]"  }
  4. {"type": "server", "timestamp": "2019-07-01T18:18:38,259+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "node name [elasticsearch-master-2], node ID [scM0pZ9MTnGum8-xyt37mQ], cluster name [clevyr-elk-cluster]"  }
  5. {"type": "server", "timestamp": "2019-07-01T18:18:38,260+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "version[7.2.0], pid[1], build[default/docker/508c38a/2019-06-20T15:54:18.811730Z], OS[Linux/4.4.0-1079-aws/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/12.0.1/12.0.1+12]"  }
  6. {"type": "server", "timestamp": "2019-07-01T18:18:38,261+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "JVM home [/usr/share/elasticsearch/jdk]"  }
  7. {"type": "server", "timestamp": "2019-07-01T18:18:38,261+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-12235550234405142175, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -Xmx2g, -Xms2g, -Dio.netty.allocator.type=pooled, -XX:MaxDirectMemorySize=1073741824, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]"  }
  8. {"type": "server", "timestamp": "2019-07-01T18:18:41,267+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "loaded module [aggs-matrix-stats]"  }
  9. {"type": "server", "timestamp": "2019-07-01T18:18:41,267+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "loaded module [analysis-common]"  }
  10. {"type": "server", "timestamp": "2019-07-01T18:18:41,268+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "loaded module [data-frame]"  }
  11. {"type": "server", "timestamp": "2019-07-01T18:18:41,269+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "loaded module [ingest-common]"  }
  12. {"type": "server", "timestamp": "2019-07-01T18:18:41,269+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "loaded module [ingest-geoip]"  }
  13. {"type": "server", "timestamp": "2019-07-01T18:18:41,270+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "loaded module [ingest-user-agent]"  }
  14. {"type": "server", "timestamp": "2019-07-01T18:18:41,271+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "loaded module [lang-expression]"  }
  15. {"type": "server", "timestamp": "2019-07-01T18:18:41,272+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "loaded module [lang-mustache]"  }
  16. {"type": "server", "timestamp": "2019-07-01T18:18:41,276+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "loaded module [lang-painless]"  }
  17. {"type": "server", "timestamp": "2019-07-01T18:18:41,276+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "loaded module [mapper-extras]"  }
  18. {"type": "server", "timestamp": "2019-07-01T18:18:41,277+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "loaded module [parent-join]"  }
  19. {"type": "server", "timestamp": "2019-07-01T18:18:41,277+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "loaded module [percolator]"  }
  20. {"type": "server", "timestamp": "2019-07-01T18:18:41,279+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "loaded module [rank-eval]"  }
  21. {"type": "server", "timestamp": "2019-07-01T18:18:41,279+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "loaded module [reindex]"  }
  22. {"type": "server", "timestamp": "2019-07-01T18:18:41,281+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "loaded module [repository-url]"  }
  23. {"type": "server", "timestamp": "2019-07-01T18:18:41,286+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "loaded module [transport-netty4]"  }
  24. {"type": "server", "timestamp": "2019-07-01T18:18:41,286+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "loaded module [x-pack-ccr]"  }
  25. {"type": "server", "timestamp": "2019-07-01T18:18:41,288+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "loaded module [x-pack-core]"  }
  26. {"type": "server", "timestamp": "2019-07-01T18:18:41,289+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "loaded module [x-pack-deprecation]"  }
  27. {"type": "server", "timestamp": "2019-07-01T18:18:41,289+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "loaded module [x-pack-graph]"  }
  28. {"type": "server", "timestamp": "2019-07-01T18:18:41,289+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "loaded module [x-pack-ilm]"  }
  29. {"type": "server", "timestamp": "2019-07-01T18:18:41,290+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "loaded module [x-pack-logstash]"  }
  30. {"type": "server", "timestamp": "2019-07-01T18:18:41,291+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "loaded module [x-pack-ml]"  }
  31. {"type": "server", "timestamp": "2019-07-01T18:18:41,292+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "loaded module [x-pack-monitoring]"  }
  32. {"type": "server", "timestamp": "2019-07-01T18:18:41,293+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "loaded module [x-pack-rollup]"  }
  33. {"type": "server", "timestamp": "2019-07-01T18:18:41,293+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "loaded module [x-pack-security]"  }
  34. {"type": "server", "timestamp": "2019-07-01T18:18:41,294+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "loaded module [x-pack-sql]"  }
  35. {"type": "server", "timestamp": "2019-07-01T18:18:41,295+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "loaded module [x-pack-watcher]"  }
  36. {"type": "server", "timestamp": "2019-07-01T18:18:41,296+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "no plugins loaded"  }
  37. {"type": "server", "timestamp": "2019-07-01T18:18:47,183+0000", "level": "INFO", "component": "o.e.x.s.a.s.FileRolesStore", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]"  }
  38. {"type": "server", "timestamp": "2019-07-01T18:18:48,337+0000", "level": "INFO", "component": "o.e.x.m.p.l.CppLogMessageHandler", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "[controller/99] [Main.cc@110] controller (64 bit): Version 7.2.0 (Build 65aefcbfce449b) Copyright (c) 2019 Elasticsearch BV"  }
  39. {"type": "server", "timestamp": "2019-07-01T18:18:48,804+0000", "level": "DEBUG", "component": "o.e.a.ActionModule", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "Using REST wrapper from plugin org.elasticsearch.xpack.security.Security"  }
  40. {"type": "server", "timestamp": "2019-07-01T18:18:49,230+0000", "level": "INFO", "component": "o.e.d.DiscoveryModule", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "using discovery type [zen] and seed hosts providers [settings]"  }
  41. {"type": "server", "timestamp": "2019-07-01T18:18:50,730+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "initialized"  }
  42. {"type": "server", "timestamp": "2019-07-01T18:18:50,731+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "starting ..."  }
  43. {"type": "server", "timestamp": "2019-07-01T18:18:50,881+0000", "level": "INFO", "component": "o.e.t.TransportService", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "publish_address {172.31.29.199:9300}, bound_addresses {0.0.0.0:9300}"  }
  44. {"type": "server", "timestamp": "2019-07-01T18:18:50,891+0000", "level": "INFO", "component": "o.e.b.BootstrapChecks", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "bound or publishing to a non-loopback address, enforcing bootstrap checks"  }
  45. {"type": "server", "timestamp": "2019-07-01T18:18:54,111+0000", "level": "INFO", "component": "o.e.c.c.Coordinator", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "setting initial configuration to VotingConfiguration{{bootstrap-placeholder}-elasticsearch-master-1,lwLc0G2zQEW3-bZDvnsKmg,scM0pZ9MTnGum8-xyt37mQ}"  }
  46. {"type": "server", "timestamp": "2019-07-01T18:18:54,337+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "failed to join {elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=4, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}}]}" ,
  47. "stacktrace": ["org.elasticsearch.transport.NodeNotConnectedException: [elasticsearch-master-1][172.31.11.195:9300] Node not connected",
  48. "at org.elasticsearch.transport.ConnectionManager.getConnection(ConnectionManager.java:151) ~[elasticsearch-7.2.0.jar:7.2.0]",
  49. "at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:568) ~[elasticsearch-7.2.0.jar:7.2.0]",
  50. "at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:540) [elasticsearch-7.2.0.jar:7.2.0]",
  51. "at org.elasticsearch.cluster.coordination.JoinHelper.sendJoinRequest(JoinHelper.java:278) [elasticsearch-7.2.0.jar:7.2.0]",
  52. "at org.elasticsearch.cluster.coordination.JoinHelper.sendJoinRequest(JoinHelper.java:211) [elasticsearch-7.2.0.jar:7.2.0]",
  53. "at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$2(JoinHelper.java:135) [elasticsearch-7.2.0.jar:7.2.0]",
  54. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) [x-pack-security-7.2.0.jar:7.2.0]",
  55. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]",
  56. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:308) [x-pack-security-7.2.0.jar:7.2.0]",
  57. "at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [elasticsearch-7.2.0.jar:7.2.0]",
  58. "at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) [elasticsearch-7.2.0.jar:7.2.0]",
  59. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.2.0.jar:7.2.0]",
  60. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]",
  61. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
  62. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
  63. "at java.lang.Thread.run(Thread.java:835) [?:?]"] }
  64. {"type": "server", "timestamp": "2019-07-01T18:18:54,369+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "failed to join {elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=4, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}}]}" ,
  65. "stacktrace": ["org.elasticsearch.transport.NodeNotConnectedException: [elasticsearch-master-1][172.31.11.195:9300] Node not connected",
  66. "at org.elasticsearch.transport.ConnectionManager.getConnection(ConnectionManager.java:151) ~[elasticsearch-7.2.0.jar:7.2.0]",
  67. "at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:568) ~[elasticsearch-7.2.0.jar:7.2.0]",
  68. "at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:540) [elasticsearch-7.2.0.jar:7.2.0]",
  69. "at org.elasticsearch.cluster.coordination.JoinHelper.sendJoinRequest(JoinHelper.java:278) [elasticsearch-7.2.0.jar:7.2.0]",
  70. "at org.elasticsearch.cluster.coordination.JoinHelper.sendJoinRequest(JoinHelper.java:211) [elasticsearch-7.2.0.jar:7.2.0]",
  71. "at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$2(JoinHelper.java:135) [elasticsearch-7.2.0.jar:7.2.0]",
  72. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) [x-pack-security-7.2.0.jar:7.2.0]",
  73. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]",
  74. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:308) [x-pack-security-7.2.0.jar:7.2.0]",
  75. "at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [elasticsearch-7.2.0.jar:7.2.0]",
  76. "at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) [elasticsearch-7.2.0.jar:7.2.0]",
  77. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.2.0.jar:7.2.0]",
  78. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]",
  79. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
  80. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
  81. "at java.lang.Thread.run(Thread.java:835) [?:?]"] }
  82. {"type": "server", "timestamp": "2019-07-01T18:18:54,937+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "failed to join {elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=7, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}}]}" ,
  83. "stacktrace": ["org.elasticsearch.transport.NodeNotConnectedException: [elasticsearch-master-1][172.31.11.195:9300] Node not connected",
  84. "at org.elasticsearch.transport.ConnectionManager.getConnection(ConnectionManager.java:151) ~[elasticsearch-7.2.0.jar:7.2.0]",
  85. "at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:568) ~[elasticsearch-7.2.0.jar:7.2.0]",
  86. "at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:540) [elasticsearch-7.2.0.jar:7.2.0]",
  87. "at org.elasticsearch.cluster.coordination.JoinHelper.sendJoinRequest(JoinHelper.java:278) [elasticsearch-7.2.0.jar:7.2.0]",
  88. "at org.elasticsearch.cluster.coordination.JoinHelper.sendJoinRequest(JoinHelper.java:211) [elasticsearch-7.2.0.jar:7.2.0]",
  89. "at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$2(JoinHelper.java:135) [elasticsearch-7.2.0.jar:7.2.0]",
  90. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) [x-pack-security-7.2.0.jar:7.2.0]",
  91. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]",
  92. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:308) [x-pack-security-7.2.0.jar:7.2.0]",
  93. "at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [elasticsearch-7.2.0.jar:7.2.0]",
  94. "at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) [elasticsearch-7.2.0.jar:7.2.0]",
  95. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.2.0.jar:7.2.0]",
  96. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]",
  97. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
  98. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
  99. "at java.lang.Thread.run(Thread.java:835) [?:?]"] }
  100. {"type": "server", "timestamp": "2019-07-01T18:18:54,939+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "failed to join {elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=7, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}}]}" ,
  101. "stacktrace": ["org.elasticsearch.transport.NodeNotConnectedException: [elasticsearch-master-1][172.31.11.195:9300] Node not connected",
  102. "at org.elasticsearch.transport.ConnectionManager.getConnection(ConnectionManager.java:151) ~[elasticsearch-7.2.0.jar:7.2.0]",
  103. "at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:568) ~[elasticsearch-7.2.0.jar:7.2.0]",
  104. "at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:540) [elasticsearch-7.2.0.jar:7.2.0]",
  105. "at org.elasticsearch.cluster.coordination.JoinHelper.sendJoinRequest(JoinHelper.java:278) [elasticsearch-7.2.0.jar:7.2.0]",
  106. "at org.elasticsearch.cluster.coordination.JoinHelper.sendJoinRequest(JoinHelper.java:211) [elasticsearch-7.2.0.jar:7.2.0]",
  107. "at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$2(JoinHelper.java:135) [elasticsearch-7.2.0.jar:7.2.0]",
  108. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) [x-pack-security-7.2.0.jar:7.2.0]",
  109. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]",
  110. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:308) [x-pack-security-7.2.0.jar:7.2.0]",
  111. "at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [elasticsearch-7.2.0.jar:7.2.0]",
  112. "at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) [elasticsearch-7.2.0.jar:7.2.0]",
  113. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.2.0.jar:7.2.0]",
  114. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]",
  115. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
  116. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
  117. "at java.lang.Thread.run(Thread.java:835) [?:?]"] }
  118. {"type": "server", "timestamp": "2019-07-01T18:18:54,998+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "failed to join {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=5, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}}]}" ,
  119. "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-2][172.31.29.199:9300][internal:cluster/coordination/join]",
  120. "Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: received a newer join from {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}",
  121. "at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.handleJoinRequest(JoinHelper.java:451) [elasticsearch-7.2.0.jar:7.2.0]",
  122. "at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:512) [elasticsearch-7.2.0.jar:7.2.0]",
  123. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:478) [elasticsearch-7.2.0.jar:7.2.0]",
  124. "at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:124) [elasticsearch-7.2.0.jar:7.2.0]",
  125. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) [x-pack-security-7.2.0.jar:7.2.0]",
  126. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]",
  127. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:308) [x-pack-security-7.2.0.jar:7.2.0]",
  128. "at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [elasticsearch-7.2.0.jar:7.2.0]",
  129. "at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:703) [elasticsearch-7.2.0.jar:7.2.0]",
  130. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.2.0.jar:7.2.0]",
  131. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]",
  132. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
  133. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
  134. "at java.lang.Thread.run(Thread.java:835) [?:?]"] }
  135. {"type": "server", "timestamp": "2019-07-01T18:18:55,519+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "failed to join {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=8, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}}]}" ,
  136. "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-2][172.31.29.199:9300][internal:cluster/coordination/join]",
  137. "Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: received a newer join from {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}",
  138. "at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.handleJoinRequest(JoinHelper.java:451) [elasticsearch-7.2.0.jar:7.2.0]",
  139. "at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:512) [elasticsearch-7.2.0.jar:7.2.0]",
  140. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:478) [elasticsearch-7.2.0.jar:7.2.0]",
  141. "at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:124) [elasticsearch-7.2.0.jar:7.2.0]",
  142. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) [x-pack-security-7.2.0.jar:7.2.0]",
  143. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]",
  144. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:308) [x-pack-security-7.2.0.jar:7.2.0]",
  145. "at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [elasticsearch-7.2.0.jar:7.2.0]",
  146. "at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:703) [elasticsearch-7.2.0.jar:7.2.0]",
  147. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.2.0.jar:7.2.0]",
  148. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]",
  149. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
  150. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
  151. "at java.lang.Thread.run(Thread.java:835) [?:?]"] }
  152. {"type": "server", "timestamp": "2019-07-01T18:18:55,568+0000", "level": "INFO", "component": "o.e.c.s.MasterService", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "elected-as-master ([2] nodes joined)[{elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} elect leader, {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 9, version: 1, reason: master node changed {previous [], current [{elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}]}, added {{elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true},}"  }
  153. {"type": "server", "timestamp": "2019-07-01T18:18:55,752+0000", "level": "WARN", "component": "o.e.c.s.MasterService", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "failing [elected-as-master ([2] nodes joined)[{elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} elect leader, {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_]]: failed to commit cluster state version [1]" ,
  154. "stacktrace": ["org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: publication failed",
  155. "at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication$3.onFailure(Coordinator.java:1353) ~[elasticsearch-7.2.0.jar:7.2.0]",
  156. "at org.elasticsearch.common.util.concurrent.ListenableFuture$1.run(ListenableFuture.java:101) ~[elasticsearch-7.2.0.jar:7.2.0]",
  157. "at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:193) ~[elasticsearch-7.2.0.jar:7.2.0]",
  158. "at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-7.2.0.jar:7.2.0]",
  159. "at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:54) ~[elasticsearch-7.2.0.jar:7.2.0]",
  160. "at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication.onCompletion(Coordinator.java:1293) ~[elasticsearch-7.2.0.jar:7.2.0]",
  161. "at org.elasticsearch.cluster.coordination.Publication.onPossibleCompletion(Publication.java:124) ~[elasticsearch-7.2.0.jar:7.2.0]",
  162. "at org.elasticsearch.cluster.coordination.Publication.onPossibleCommitFailure(Publication.java:172) ~[elasticsearch-7.2.0.jar:7.2.0]",
  163. "at org.elasticsearch.cluster.coordination.Publication.access$600(Publication.java:41) ~[elasticsearch-7.2.0.jar:7.2.0]",
  164. "at org.elasticsearch.cluster.coordination.Publication$PublicationTarget$PublishResponseHandler.onFailure(Publication.java:348) ~[elasticsearch-7.2.0.jar:7.2.0]",
  165. "at org.elasticsearch.cluster.coordination.Coordinator$6.onFailure(Coordinator.java:1080) ~[elasticsearch-7.2.0.jar:7.2.0]",
  166. "at org.elasticsearch.cluster.coordination.PublicationTransportHandler.lambda$sendClusterStateToNode$6(PublicationTransportHandler.java:267) ~[elasticsearch-7.2.0.jar:7.2.0]",
  167. "at org.elasticsearch.cluster.coordination.PublicationTransportHandler$3.handleException(PublicationTransportHandler.java:285) ~[elasticsearch-7.2.0.jar:7.2.0]",
  168. "at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1111) ~[elasticsearch-7.2.0.jar:7.2.0]",
  169. "at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1111) ~[elasticsearch-7.2.0.jar:7.2.0]",
  170. "at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:246) ~[elasticsearch-7.2.0.jar:7.2.0]",
  171. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688) ~[elasticsearch-7.2.0.jar:7.2.0]",
  172. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
  173. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
  174. "at java.lang.Thread.run(Thread.java:835) [?:?]",
  175. "Caused by: org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: non-failed nodes do not form a quorum",
  176. "at org.elasticsearch.cluster.coordination.Publication.onPossibleCommitFailure(Publication.java:170) ~[elasticsearch-7.2.0.jar:7.2.0]",
  177. "... 12 more"] }
  178. {"type": "server", "timestamp": "2019-07-01T18:18:55,764+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "failed to join {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=9, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}}]}" ,
  179. "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-2][172.31.29.199:9300][internal:cluster/coordination/join]",
  180. "Caused by: org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: publication failed",
  181. "at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication$3.onFailure(Coordinator.java:1353) ~[elasticsearch-7.2.0.jar:7.2.0]",
  182. "at org.elasticsearch.common.util.concurrent.ListenableFuture$1.run(ListenableFuture.java:101) ~[elasticsearch-7.2.0.jar:7.2.0]",
  183. "at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:193) ~[elasticsearch-7.2.0.jar:7.2.0]",
  184. "at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-7.2.0.jar:7.2.0]",
  185. "at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:54) ~[elasticsearch-7.2.0.jar:7.2.0]",
  186. "at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication.onCompletion(Coordinator.java:1293) ~[elasticsearch-7.2.0.jar:7.2.0]",
  187. "at org.elasticsearch.cluster.coordination.Publication.onPossibleCompletion(Publication.java:124) ~[elasticsearch-7.2.0.jar:7.2.0]",
  188. "at org.elasticsearch.cluster.coordination.Publication.onPossibleCommitFailure(Publication.java:172) ~[elasticsearch-7.2.0.jar:7.2.0]",
  189. "at org.elasticsearch.cluster.coordination.Publication.access$600(Publication.java:41) ~[elasticsearch-7.2.0.jar:7.2.0]",
  190. "at org.elasticsearch.cluster.coordination.Publication$PublicationTarget$PublishResponseHandler.onFailure(Publication.java:348) ~[elasticsearch-7.2.0.jar:7.2.0]",
  191. "at org.elasticsearch.cluster.coordination.Coordinator$6.onFailure(Coordinator.java:1080) ~[elasticsearch-7.2.0.jar:7.2.0]",
  192. "at org.elasticsearch.cluster.coordination.PublicationTransportHandler.lambda$sendClusterStateToNode$6(PublicationTransportHandler.java:267) ~[elasticsearch-7.2.0.jar:7.2.0]",
  193. "at org.elasticsearch.cluster.coordination.PublicationTransportHandler$3.handleException(PublicationTransportHandler.java:285) ~[elasticsearch-7.2.0.jar:7.2.0]",
  194. "at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1111) ~[elasticsearch-7.2.0.jar:7.2.0]",
  195. "at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1111) ~[elasticsearch-7.2.0.jar:7.2.0]",
  196. "at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:246) ~[elasticsearch-7.2.0.jar:7.2.0]",
  197. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688) ~[elasticsearch-7.2.0.jar:7.2.0]",
  198. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
  199. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
  200. "at java.lang.Thread.run(Thread.java:835) [?:?]",
  201. "Caused by: org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: non-failed nodes do not form a quorum",
  202. "at org.elasticsearch.cluster.coordination.Publication.onPossibleCommitFailure(Publication.java:170) ~[elasticsearch-7.2.0.jar:7.2.0]",
  203. "at org.elasticsearch.cluster.coordination.Publication.access$600(Publication.java:41) ~[elasticsearch-7.2.0.jar:7.2.0]",
  204. "at org.elasticsearch.cluster.coordination.Publication$PublicationTarget$PublishResponseHandler.onFailure(Publication.java:348) ~[elasticsearch-7.2.0.jar:7.2.0]",
  205. "at org.elasticsearch.cluster.coordination.Coordinator$6.onFailure(Coordinator.java:1080) ~[elasticsearch-7.2.0.jar:7.2.0]",
  206. "at org.elasticsearch.cluster.coordination.PublicationTransportHandler.lambda$sendClusterStateToNode$6(PublicationTransportHandler.java:267) ~[elasticsearch-7.2.0.jar:7.2.0]",
  207. "at org.elasticsearch.cluster.coordination.PublicationTransportHandler$3.handleException(PublicationTransportHandler.java:285) ~[elasticsearch-7.2.0.jar:7.2.0]",
  208. "at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1111) ~[elasticsearch-7.2.0.jar:7.2.0]",
  209. "at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1111) ~[elasticsearch-7.2.0.jar:7.2.0]",
  210. "at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:246) ~[elasticsearch-7.2.0.jar:7.2.0]",
  211. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688) ~[elasticsearch-7.2.0.jar:7.2.0]",
  212. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]",
  213. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]",
  214. "at java.lang.Thread.run(Thread.java:835) ~[?:?]"] }
  215. {"type": "server", "timestamp": "2019-07-01T18:18:57,038+0000", "level": "INFO", "component": "o.e.c.s.MasterService", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "elected-as-master ([2] nodes joined)[{elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} elect leader, {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 12, version: 2, reason: master node changed {previous [], current [{elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}]}"  }
  216. {"type": "server", "timestamp": "2019-07-01T18:18:57,162+0000", "level": "WARN", "component": "o.e.c.s.MasterService", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "failing [elected-as-master ([2] nodes joined)[{elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} elect leader, {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_]]: failed to commit cluster state version [2]" ,
  217. "stacktrace": ["org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: publication failed",
  218. "at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication$3.onFailure(Coordinator.java:1353) ~[elasticsearch-7.2.0.jar:7.2.0]",
  219. "at org.elasticsearch.common.util.concurrent.ListenableFuture$1.run(ListenableFuture.java:101) ~[elasticsearch-7.2.0.jar:7.2.0]",
  220. "at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:193) ~[elasticsearch-7.2.0.jar:7.2.0]",
  221. "at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-7.2.0.jar:7.2.0]",
  222. "at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:54) ~[elasticsearch-7.2.0.jar:7.2.0]",
  223. "at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication.onCompletion(Coordinator.java:1293) ~[elasticsearch-7.2.0.jar:7.2.0]",
  224. "at org.elasticsearch.cluster.coordination.Publication.onPossibleCompletion(Publication.java:124) ~[elasticsearch-7.2.0.jar:7.2.0]",
  225. "at org.elasticsearch.cluster.coordination.Publication.onPossibleCommitFailure(Publication.java:172) ~[elasticsearch-7.2.0.jar:7.2.0]",
  226. "at org.elasticsearch.cluster.coordination.Publication.access$600(Publication.java:41) ~[elasticsearch-7.2.0.jar:7.2.0]",
  227. "at org.elasticsearch.cluster.coordination.Publication$PublicationTarget$PublishResponseHandler.onFailure(Publication.java:348) ~[elasticsearch-7.2.0.jar:7.2.0]",
  228. "at org.elasticsearch.cluster.coordination.Coordinator$6.onFailure(Coordinator.java:1080) ~[elasticsearch-7.2.0.jar:7.2.0]",
  229. "at org.elasticsearch.cluster.coordination.PublicationTransportHandler.lambda$sendClusterStateToNode$6(PublicationTransportHandler.java:267) ~[elasticsearch-7.2.0.jar:7.2.0]",
  230. "at org.elasticsearch.cluster.coordination.PublicationTransportHandler$3.handleException(PublicationTransportHandler.java:285) ~[elasticsearch-7.2.0.jar:7.2.0]",
  231. "at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1111) ~[elasticsearch-7.2.0.jar:7.2.0]",
  232. "at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1111) ~[elasticsearch-7.2.0.jar:7.2.0]",
  233. "at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:246) ~[elasticsearch-7.2.0.jar:7.2.0]",
  234. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688) ~[elasticsearch-7.2.0.jar:7.2.0]",
  235. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
  236. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
  237. "at java.lang.Thread.run(Thread.java:835) [?:?]",
  238. "Caused by: org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: non-failed nodes do not form a quorum",
  239. "at org.elasticsearch.cluster.coordination.Publication.onPossibleCommitFailure(Publication.java:170) ~[elasticsearch-7.2.0.jar:7.2.0]",
  240. "... 12 more"] }
  241. {"type": "server", "timestamp": "2019-07-01T18:18:57,163+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "failed to join {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=12, lastAcceptedTerm=9, lastAcceptedVersion=1, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}}]}" ,
  242. "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-2][172.31.29.199:9300][internal:cluster/coordination/join]",
  243. "Caused by: org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: publication failed",
  244. "at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication$3.onFailure(Coordinator.java:1353) ~[elasticsearch-7.2.0.jar:7.2.0]",
  245. "at org.elasticsearch.common.util.concurrent.ListenableFuture$1.run(ListenableFuture.java:101) ~[elasticsearch-7.2.0.jar:7.2.0]",
  246. "at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:193) ~[elasticsearch-7.2.0.jar:7.2.0]",
  247. "at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-7.2.0.jar:7.2.0]",
  248. "at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:54) ~[elasticsearch-7.2.0.jar:7.2.0]",
  249. "at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication.onCompletion(Coordinator.java:1293) ~[elasticsearch-7.2.0.jar:7.2.0]",
  250. "at org.elasticsearch.cluster.coordination.Publication.onPossibleCompletion(Publication.java:124) ~[elasticsearch-7.2.0.jar:7.2.0]",
  251. "at org.elasticsearch.cluster.coordination.Publication.onPossibleCommitFailure(Publication.java:172) ~[elasticsearch-7.2.0.jar:7.2.0]",
  252. "at org.elasticsearch.cluster.coordination.Publication.access$600(Publication.java:41) ~[elasticsearch-7.2.0.jar:7.2.0]",
  253. "at org.elasticsearch.cluster.coordination.Publication$PublicationTarget$PublishResponseHandler.onFailure(Publication.java:348) ~[elasticsearch-7.2.0.jar:7.2.0]",
  254. "at org.elasticsearch.cluster.coordination.Coordinator$6.onFailure(Coordinator.java:1080) ~[elasticsearch-7.2.0.jar:7.2.0]",
  255. "at org.elasticsearch.cluster.coordination.PublicationTransportHandler.lambda$sendClusterStateToNode$6(PublicationTransportHandler.java:267) ~[elasticsearch-7.2.0.jar:7.2.0]",
  256. "at org.elasticsearch.cluster.coordination.PublicationTransportHandler$3.handleException(PublicationTransportHandler.java:285) ~[elasticsearch-7.2.0.jar:7.2.0]",
  257. "at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1111) ~[elasticsearch-7.2.0.jar:7.2.0]",
  258. "at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1111) ~[elasticsearch-7.2.0.jar:7.2.0]",
  259. "at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:246) ~[elasticsearch-7.2.0.jar:7.2.0]",
  260. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688) ~[elasticsearch-7.2.0.jar:7.2.0]",
  261. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
  262. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
  263. "at java.lang.Thread.run(Thread.java:835) [?:?]",
  264. "Caused by: org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: non-failed nodes do not form a quorum",
  265. "at org.elasticsearch.cluster.coordination.Publication.onPossibleCommitFailure(Publication.java:170) ~[elasticsearch-7.2.0.jar:7.2.0]",
  266. "at org.elasticsearch.cluster.coordination.Publication.access$600(Publication.java:41) ~[elasticsearch-7.2.0.jar:7.2.0]",
  267. "at org.elasticsearch.cluster.coordination.Publication$PublicationTarget$PublishResponseHandler.onFailure(Publication.java:348) ~[elasticsearch-7.2.0.jar:7.2.0]",
  268. "at org.elasticsearch.cluster.coordination.Coordinator$6.onFailure(Coordinator.java:1080) ~[elasticsearch-7.2.0.jar:7.2.0]",
  269. "at org.elasticsearch.cluster.coordination.PublicationTransportHandler.lambda$sendClusterStateToNode$6(PublicationTransportHandler.java:267) ~[elasticsearch-7.2.0.jar:7.2.0]",
  270. "at org.elasticsearch.cluster.coordination.PublicationTransportHandler$3.handleException(PublicationTransportHandler.java:285) ~[elasticsearch-7.2.0.jar:7.2.0]",
  271. "at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1111) ~[elasticsearch-7.2.0.jar:7.2.0]",
  272. "at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1111) ~[elasticsearch-7.2.0.jar:7.2.0]",
  273. "at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:246) ~[elasticsearch-7.2.0.jar:7.2.0]",
  274. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688) ~[elasticsearch-7.2.0.jar:7.2.0]",
  275. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]",
  276. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]",
  277. "at java.lang.Thread.run(Thread.java:835) ~[?:?]"] }
  278. {"type": "server", "timestamp": "2019-07-01T18:18:57,247+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "failed to join {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=13, lastAcceptedTerm=12, lastAcceptedVersion=2, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}}]}" ,
  279. "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-0][172.17.0.2:9300][internal:cluster/coordination/join]",
  280. "Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming last accepted term 12 of join higher than current last accepted term 0",
  281. "at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:230) ~[elasticsearch-7.2.0.jar:7.2.0]",
  282. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:942) ~[elasticsearch-7.2.0.jar:7.2.0]",
  283. "at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]",
  284. "at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:511) ~[elasticsearch-7.2.0.jar:7.2.0]",
  285. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:478) ~[elasticsearch-7.2.0.jar:7.2.0]",
  286. "at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:124) ~[elasticsearch-7.2.0.jar:7.2.0]",
  287. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) ~[?:?]",
  288. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  289. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:308) ~[?:?]",
  290. "at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.2.0.jar:7.2.0]",
  291. "at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.2.0.jar:7.2.0]",
  292. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.2.0.jar:7.2.0]",
  293. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  294. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]",
  295. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]",
  296. "at java.lang.Thread.run(Thread.java:835) [?:?]"] }
  297. {"type": "server", "timestamp": "2019-07-01T18:18:59,089+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "failed to join {elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=16, lastAcceptedTerm=12, lastAcceptedVersion=2, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}}]}" ,
  298. "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-1][172.17.0.2:9300][internal:cluster/coordination/join]",
  299. "Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming last accepted term 12 of join higher than current last accepted term 0",
  300. "at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:230) ~[elasticsearch-7.2.0.jar:7.2.0]",
  301. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:942) ~[elasticsearch-7.2.0.jar:7.2.0]",
  302. "at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]",
  303. "at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:511) ~[elasticsearch-7.2.0.jar:7.2.0]",
  304. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:478) ~[elasticsearch-7.2.0.jar:7.2.0]",
  305. "at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:124) ~[elasticsearch-7.2.0.jar:7.2.0]",
  306. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) ~[?:?]",
  307. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  308. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:308) ~[?:?]",
  309. "at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.2.0.jar:7.2.0]",
  310. "at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.2.0.jar:7.2.0]",
  311. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.2.0.jar:7.2.0]",
  312. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  313. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]",
  314. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]",
  315. "at java.lang.Thread.run(Thread.java:835) [?:?]"] }
  316. {"type": "server", "timestamp": "2019-07-01T18:18:59,322+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "failed to join {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=15, lastAcceptedTerm=12, lastAcceptedVersion=2, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}}]}" ,
  317. "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-2][172.31.29.199:9300][internal:cluster/coordination/join]",
  318. "Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: received a newer join from {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}",
  319. "at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.handleJoinRequest(JoinHelper.java:451) [elasticsearch-7.2.0.jar:7.2.0]",
  320. "at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:512) [elasticsearch-7.2.0.jar:7.2.0]",
  321. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:478) [elasticsearch-7.2.0.jar:7.2.0]",
  322. "at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:124) [elasticsearch-7.2.0.jar:7.2.0]",
  323. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) [x-pack-security-7.2.0.jar:7.2.0]",
  324. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]",
  325. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:308) [x-pack-security-7.2.0.jar:7.2.0]",
  326. "at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [elasticsearch-7.2.0.jar:7.2.0]",
  327. "at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:703) [elasticsearch-7.2.0.jar:7.2.0]",
  328. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.2.0.jar:7.2.0]",
  329. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]",
  330. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
  331. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
  332. "at java.lang.Thread.run(Thread.java:835) [?:?]"] }
  333. {"type": "server", "timestamp": "2019-07-01T18:18:59,837+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "failed to join {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=18, lastAcceptedTerm=12, lastAcceptedVersion=2, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}}]}" ,
  334. "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-2][172.31.29.199:9300][internal:cluster/coordination/join]",
  335. "Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: received a newer join from {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}",
  336. "at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.handleJoinRequest(JoinHelper.java:451) [elasticsearch-7.2.0.jar:7.2.0]",
  337. "at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:512) [elasticsearch-7.2.0.jar:7.2.0]",
  338. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:478) [elasticsearch-7.2.0.jar:7.2.0]",
  339. "at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:124) [elasticsearch-7.2.0.jar:7.2.0]",
  340. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) [x-pack-security-7.2.0.jar:7.2.0]",
  341. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]",
  342. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:308) [x-pack-security-7.2.0.jar:7.2.0]",
  343. "at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [elasticsearch-7.2.0.jar:7.2.0]",
  344. "at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:703) [elasticsearch-7.2.0.jar:7.2.0]",
  345. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.2.0.jar:7.2.0]",
  346. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]",
  347. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
  348. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
  349. "at java.lang.Thread.run(Thread.java:835) [?:?]"] }
  350. {"type": "server", "timestamp": "2019-07-01T18:18:59,847+0000", "level": "INFO", "component": "o.e.c.s.MasterService", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "elected-as-master ([2] nodes joined)[{elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} elect leader, {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 19, version: 3, reason: master node changed {previous [], current [{elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}]}"  }
  351. {"type": "server", "timestamp": "2019-07-01T18:18:59,860+0000", "level": "WARN", "component": "o.e.c.s.MasterService", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "failing [elected-as-master ([2] nodes joined)[{elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} elect leader, {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_]]: failed to commit cluster state version [3]" ,
  352. "stacktrace": ["org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: publication failed",
  353. "at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication$3.onFailure(Coordinator.java:1353) ~[elasticsearch-7.2.0.jar:7.2.0]",
  354. "at org.elasticsearch.common.util.concurrent.ListenableFuture$1.run(ListenableFuture.java:101) ~[elasticsearch-7.2.0.jar:7.2.0]",
  355. "at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:193) ~[elasticsearch-7.2.0.jar:7.2.0]",
  356. "at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-7.2.0.jar:7.2.0]",
  357. "at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:54) ~[elasticsearch-7.2.0.jar:7.2.0]",
  358. "at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication.onCompletion(Coordinator.java:1293) ~[elasticsearch-7.2.0.jar:7.2.0]",
  359. "at org.elasticsearch.cluster.coordination.Publication.onPossibleCompletion(Publication.java:124) ~[elasticsearch-7.2.0.jar:7.2.0]",
  360. "at org.elasticsearch.cluster.coordination.Publication.cancel(Publication.java:88) ~[elasticsearch-7.2.0.jar:7.2.0]",
  361. "at org.elasticsearch.cluster.coordination.Coordinator.cancelActivePublication(Coordinator.java:1089) ~[elasticsearch-7.2.0.jar:7.2.0]",
  362. "at org.elasticsearch.cluster.coordination.Coordinator.becomeFollower(Coordinator.java:610) ~[elasticsearch-7.2.0.jar:7.2.0]",
  363. "at org.elasticsearch.cluster.coordination.Coordinator.onFollowerCheckRequest(Coordinator.java:237) ~[elasticsearch-7.2.0.jar:7.2.0]",
  364. "at org.elasticsearch.cluster.coordination.FollowersChecker$2.doRun(FollowersChecker.java:187) ~[elasticsearch-7.2.0.jar:7.2.0]",
  365. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.2.0.jar:7.2.0]",
  366. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  367. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
  368. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
  369. "at java.lang.Thread.run(Thread.java:835) [?:?]",
  370. "Caused by: org.elasticsearch.ElasticsearchException: publication cancelled before committing: become follower: onFollowerCheckRequest",
  371. "at org.elasticsearch.cluster.coordination.Publication.cancel(Publication.java:85) ~[elasticsearch-7.2.0.jar:7.2.0]",
  372. "... 9 more"] }
  373. {"type": "server", "timestamp": "2019-07-01T18:18:59,865+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "failed to join {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=19, lastAcceptedTerm=12, lastAcceptedVersion=2, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}}]}" ,
  374. "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-2][172.31.29.199:9300][internal:cluster/coordination/join]",
  375. "Caused by: org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: publication failed",
  376. "at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication$3.onFailure(Coordinator.java:1353) ~[elasticsearch-7.2.0.jar:7.2.0]",
  377. "at org.elasticsearch.common.util.concurrent.ListenableFuture$1.run(ListenableFuture.java:101) ~[elasticsearch-7.2.0.jar:7.2.0]",
  378. "at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:193) ~[elasticsearch-7.2.0.jar:7.2.0]",
  379. "at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-7.2.0.jar:7.2.0]",
  380. "at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:54) ~[elasticsearch-7.2.0.jar:7.2.0]",
  381. "at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication.onCompletion(Coordinator.java:1293) ~[elasticsearch-7.2.0.jar:7.2.0]",
  382. "at org.elasticsearch.cluster.coordination.Publication.onPossibleCompletion(Publication.java:124) ~[elasticsearch-7.2.0.jar:7.2.0]",
  383. "at org.elasticsearch.cluster.coordination.Publication.cancel(Publication.java:88) ~[elasticsearch-7.2.0.jar:7.2.0]",
  384. "at org.elasticsearch.cluster.coordination.Coordinator.cancelActivePublication(Coordinator.java:1089) ~[elasticsearch-7.2.0.jar:7.2.0]",
  385. "at org.elasticsearch.cluster.coordination.Coordinator.becomeFollower(Coordinator.java:610) ~[elasticsearch-7.2.0.jar:7.2.0]",
  386. "at org.elasticsearch.cluster.coordination.Coordinator.onFollowerCheckRequest(Coordinator.java:237) ~[elasticsearch-7.2.0.jar:7.2.0]",
  387. "at org.elasticsearch.cluster.coordination.FollowersChecker$2.doRun(FollowersChecker.java:187) ~[elasticsearch-7.2.0.jar:7.2.0]",
  388. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.2.0.jar:7.2.0]",
  389. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  390. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
  391. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
  392. "at java.lang.Thread.run(Thread.java:835) [?:?]",
  393. "Caused by: org.elasticsearch.ElasticsearchException: publication cancelled before committing: become follower: onFollowerCheckRequest",
  394. "at org.elasticsearch.cluster.coordination.Publication.cancel(Publication.java:85) ~[elasticsearch-7.2.0.jar:7.2.0]",
  395. "at org.elasticsearch.cluster.coordination.Coordinator.cancelActivePublication(Coordinator.java:1089) ~[elasticsearch-7.2.0.jar:7.2.0]",
  396. "at org.elasticsearch.cluster.coordination.Coordinator.becomeFollower(Coordinator.java:610) ~[elasticsearch-7.2.0.jar:7.2.0]",
  397. "at org.elasticsearch.cluster.coordination.Coordinator.onFollowerCheckRequest(Coordinator.java:237) ~[elasticsearch-7.2.0.jar:7.2.0]",
  398. "at org.elasticsearch.cluster.coordination.FollowersChecker$2.doRun(FollowersChecker.java:187) ~[elasticsearch-7.2.0.jar:7.2.0]",
  399. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.2.0.jar:7.2.0]",
  400. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  401. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]",
  402. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]",
  403. "at java.lang.Thread.run(Thread.java:835) ~[?:?]"] }
  404. {"type": "server", "timestamp": "2019-07-01T18:19:00,631+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "failed to join {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=21, lastAcceptedTerm=12, lastAcceptedVersion=2, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}}]}" ,
  405. "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-2][172.31.29.199:9300][internal:cluster/coordination/join]",
  406. "Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: received a newer join from {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}",
  407. "at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.handleJoinRequest(JoinHelper.java:451) [elasticsearch-7.2.0.jar:7.2.0]",
  408. "at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:512) [elasticsearch-7.2.0.jar:7.2.0]",
  409. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:478) [elasticsearch-7.2.0.jar:7.2.0]",
  410. "at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:124) [elasticsearch-7.2.0.jar:7.2.0]",
  411. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) [x-pack-security-7.2.0.jar:7.2.0]",
  412. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]",
  413. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:308) [x-pack-security-7.2.0.jar:7.2.0]",
  414. "at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [elasticsearch-7.2.0.jar:7.2.0]",
  415. "at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:703) [elasticsearch-7.2.0.jar:7.2.0]",
  416. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.2.0.jar:7.2.0]",
  417. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]",
  418. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
  419. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
  420. "at java.lang.Thread.run(Thread.java:835) [?:?]"] }
  421. {"type": "server", "timestamp": "2019-07-01T18:19:01,058+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "failed to join {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=22, lastAcceptedTerm=12, lastAcceptedVersion=2, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}}]}" ,
  422. "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-2][172.31.29.199:9300][internal:cluster/coordination/join]",
  423. "Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: became follower",
  424. "at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.lambda$close$3(JoinHelper.java:476) [elasticsearch-7.2.0.jar:7.2.0]",
  425. "at java.util.HashMap$Values.forEach(HashMap.java:976) [?:?]",
  426. "at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.close(JoinHelper.java:476) [elasticsearch-7.2.0.jar:7.2.0]",
  427. "at org.elasticsearch.cluster.coordination.Coordinator.becomeFollower(Coordinator.java:600) [elasticsearch-7.2.0.jar:7.2.0]",
  428. "at org.elasticsearch.cluster.coordination.Coordinator.onFollowerCheckRequest(Coordinator.java:237) [elasticsearch-7.2.0.jar:7.2.0]",
  429. "at org.elasticsearch.cluster.coordination.FollowersChecker$2.doRun(FollowersChecker.java:187) [elasticsearch-7.2.0.jar:7.2.0]",
  430. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.2.0.jar:7.2.0]",
  431. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]",
  432. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
  433. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
  434. "at java.lang.Thread.run(Thread.java:835) [?:?]"] }
  435. {"type": "server", "timestamp": "2019-07-01T18:19:02,065+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "failed to join {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=24, lastAcceptedTerm=12, lastAcceptedVersion=2, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}}]}" ,
  436. "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-2][172.17.0.2:9300][internal:cluster/coordination/join]",
  437. "Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: became follower",
  438. "at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.lambda$close$3(JoinHelper.java:476) ~[elasticsearch-7.2.0.jar:7.2.0]",
  439. "at java.util.HashMap$Values.forEach(HashMap.java:976) ~[?:?]",
  440. "at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.close(JoinHelper.java:476) ~[elasticsearch-7.2.0.jar:7.2.0]",
  441. "at org.elasticsearch.cluster.coordination.Coordinator.becomeFollower(Coordinator.java:600) ~[elasticsearch-7.2.0.jar:7.2.0]",
  442. "at org.elasticsearch.cluster.coordination.Coordinator.onFollowerCheckRequest(Coordinator.java:237) ~[elasticsearch-7.2.0.jar:7.2.0]",
  443. "at org.elasticsearch.cluster.coordination.FollowersChecker$2.doRun(FollowersChecker.java:187) ~[elasticsearch-7.2.0.jar:7.2.0]",
  444. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.2.0.jar:7.2.0]",
  445. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  446. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]",
  447. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]",
  448. "at java.lang.Thread.run(Thread.java:835) [?:?]"] }
  449. {"type": "server", "timestamp": "2019-07-01T18:19:05,526+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "failed to join {elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=28, lastAcceptedTerm=12, lastAcceptedVersion=2, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}}]}" ,
  450. "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-1][172.17.0.2:9300][internal:cluster/coordination/join]",
  451. "Caused by: org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: publication failed",
  452. "at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication$3.onFailure(Coordinator.java:1353) ~[elasticsearch-7.2.0.jar:7.2.0]",
  453. "at org.elasticsearch.common.util.concurrent.ListenableFuture$1.run(ListenableFuture.java:101) ~[elasticsearch-7.2.0.jar:7.2.0]",
  454. "at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:193) ~[elasticsearch-7.2.0.jar:7.2.0]",
  455. "at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-7.2.0.jar:7.2.0]",
  456. "at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:54) ~[elasticsearch-7.2.0.jar:7.2.0]",
  457. "at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication.onCompletion(Coordinator.java:1293) ~[elasticsearch-7.2.0.jar:7.2.0]",
  458. "at org.elasticsearch.cluster.coordination.Publication.onPossibleCompletion(Publication.java:124) ~[elasticsearch-7.2.0.jar:7.2.0]",
  459. "at org.elasticsearch.cluster.coordination.Publication.onPossibleCommitFailure(Publication.java:172) ~[elasticsearch-7.2.0.jar:7.2.0]",
  460. "at org.elasticsearch.cluster.coordination.Publication.access$600(Publication.java:41) ~[elasticsearch-7.2.0.jar:7.2.0]",
  461. "at org.elasticsearch.cluster.coordination.Publication$PublicationTarget$PublishResponseHandler.onFailure(Publication.java:348) ~[elasticsearch-7.2.0.jar:7.2.0]",
  462. "at org.elasticsearch.cluster.coordination.Coordinator$6.onFailure(Coordinator.java:1080) ~[elasticsearch-7.2.0.jar:7.2.0]",
  463. "at org.elasticsearch.cluster.coordination.PublicationTransportHandler.lambda$sendClusterStateToNode$6(PublicationTransportHandler.java:267) ~[elasticsearch-7.2.0.jar:7.2.0]",
  464. "at org.elasticsearch.cluster.coordination.PublicationTransportHandler$3.handleException(PublicationTransportHandler.java:285) ~[elasticsearch-7.2.0.jar:7.2.0]",
  465. "at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1111) ~[elasticsearch-7.2.0.jar:7.2.0]",
  466. "at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1111) ~[elasticsearch-7.2.0.jar:7.2.0]",
  467. "at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:246) ~[elasticsearch-7.2.0.jar:7.2.0]",
  468. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688) ~[elasticsearch-7.2.0.jar:7.2.0]",
  469. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]",
  470. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]",
  471. "at java.lang.Thread.run(Thread.java:835) [?:?]",
  472. "Caused by: org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: non-failed nodes do not form a quorum",
  473. "at org.elasticsearch.cluster.coordination.Publication.onPossibleCommitFailure(Publication.java:170) ~[elasticsearch-7.2.0.jar:7.2.0]",
  474. "at org.elasticsearch.cluster.coordination.Publication.access$600(Publication.java:41) ~[elasticsearch-7.2.0.jar:7.2.0]",
  475. "at org.elasticsearch.cluster.coordination.Publication$PublicationTarget$PublishResponseHandler.onFailure(Publication.java:348) ~[elasticsearch-7.2.0.jar:7.2.0]",
  476. "at org.elasticsearch.cluster.coordination.Coordinator$6.onFailure(Coordinator.java:1080) ~[elasticsearch-7.2.0.jar:7.2.0]",
  477. "at org.elasticsearch.cluster.coordination.PublicationTransportHandler.lambda$sendClusterStateToNode$6(PublicationTransportHandler.java:267) ~[elasticsearch-7.2.0.jar:7.2.0]",
  478. "at org.elasticsearch.cluster.coordination.PublicationTransportHandler$3.handleException(PublicationTransportHandler.java:285) ~[elasticsearch-7.2.0.jar:7.2.0]",
  479. "at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1111) ~[elasticsearch-7.2.0.jar:7.2.0]",
  480. "at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1111) ~[elasticsearch-7.2.0.jar:7.2.0]",
  481. "at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:246) ~[elasticsearch-7.2.0.jar:7.2.0]",
  482. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688) ~[elasticsearch-7.2.0.jar:7.2.0]",
  483. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]",
  484. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]",
  485. "at java.lang.Thread.run(Thread.java:835) ~[?:?]"] }
  486. {"type": "server", "timestamp": "2019-07-01T18:19:17,357+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "last failed join attempt was 11.8s ago, failed to join {elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=28, lastAcceptedTerm=12, lastAcceptedVersion=2, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}}]}" ,
  487. "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-1][172.17.0.2:9300][internal:cluster/coordination/join]",
  488. "Caused by: org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: publication failed",
  489. "at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication$3.onFailure(Coordinator.java:1353) ~[elasticsearch-7.2.0.jar:7.2.0]",
  490. "at org.elasticsearch.common.util.concurrent.ListenableFuture$1.run(ListenableFuture.java:101) ~[elasticsearch-7.2.0.jar:7.2.0]",
  491. "at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:193) ~[elasticsearch-7.2.0.jar:7.2.0]",
  492. "at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-7.2.0.jar:7.2.0]",
  493. "at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:54) ~[elasticsearch-7.2.0.jar:7.2.0]",
  494. "at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication.onCompletion(Coordinator.java:1293) ~[elasticsearch-7.2.0.jar:7.2.0]",
  495. "at org.elasticsearch.cluster.coordination.Publication.onPossibleCompletion(Publication.java:124) ~[elasticsearch-7.2.0.jar:7.2.0]",
  496. "at org.elasticsearch.cluster.coordination.Publication.onPossibleCommitFailure(Publication.java:172) ~[elasticsearch-7.2.0.jar:7.2.0]",
  497. "at org.elasticsearch.cluster.coordination.Publication.access$600(Publication.java:41) ~[elasticsearch-7.2.0.jar:7.2.0]",
  498. "at org.elasticsearch.cluster.coordination.Publication$PublicationTarget$PublishResponseHandler.onFailure(Publication.java:348) ~[elasticsearch-7.2.0.jar:7.2.0]",
  499. "at org.elasticsearch.cluster.coordination.Coordinator$6.onFailure(Coordinator.java:1080) ~[elasticsearch-7.2.0.jar:7.2.0]",
  500. "at org.elasticsearch.cluster.coordination.PublicationTransportHandler.lambda$sendClusterStateToNode$6(PublicationTransportHandler.java:267) ~[elasticsearch-7.2.0.jar:7.2.0]",
  501. "at org.elasticsearch.cluster.coordination.PublicationTransportHandler$3.handleException(PublicationTransportHandler.java:285) ~[elasticsearch-7.2.0.jar:7.2.0]",
  502. "at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1111) ~[elasticsearch-7.2.0.jar:7.2.0]",
  503. "at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1111) ~[elasticsearch-7.2.0.jar:7.2.0]",
  504. "at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:246) ~[elasticsearch-7.2.0.jar:7.2.0]",
  505. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688) ~[elasticsearch-7.2.0.jar:7.2.0]",
  506. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
  507. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
  508. "at java.lang.Thread.run(Thread.java:835) [?:?]",
  509. "Caused by: org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: non-failed nodes do not form a quorum",
  510. "at org.elasticsearch.cluster.coordination.Publication.onPossibleCommitFailure(Publication.java:170) ~[elasticsearch-7.2.0.jar:7.2.0]",
  511. "at org.elasticsearch.cluster.coordination.Publication.access$600(Publication.java:41) ~[elasticsearch-7.2.0.jar:7.2.0]",
  512. "at org.elasticsearch.cluster.coordination.Publication$PublicationTarget$PublishResponseHandler.onFailure(Publication.java:348) ~[elasticsearch-7.2.0.jar:7.2.0]",
  513. "at org.elasticsearch.cluster.coordination.Coordinator$6.onFailure(Coordinator.java:1080) ~[elasticsearch-7.2.0.jar:7.2.0]",
  514. "at org.elasticsearch.cluster.coordination.PublicationTransportHandler.lambda$sendClusterStateToNode$6(PublicationTransportHandler.java:267) ~[elasticsearch-7.2.0.jar:7.2.0]",
  515. "at org.elasticsearch.cluster.coordination.PublicationTransportHandler$3.handleException(PublicationTransportHandler.java:285) ~[elasticsearch-7.2.0.jar:7.2.0]",
  516. "at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1111) ~[elasticsearch-7.2.0.jar:7.2.0]",
  517. "at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1111) ~[elasticsearch-7.2.0.jar:7.2.0]",
  518. "at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:246) ~[elasticsearch-7.2.0.jar:7.2.0]",
  519. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688) ~[elasticsearch-7.2.0.jar:7.2.0]",
  520. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]",
  521. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]",
  522. "at java.lang.Thread.run(Thread.java:835) ~[?:?]"] }
  523. {"type": "server", "timestamp": "2019-07-01T18:19:17,360+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "master not discovered or elected yet, an election requires 2 nodes with ids [NtLoCKHvSVWEO5hUT6jT2Q, lwLc0G2zQEW3-bZDvnsKmg], have discovered [{elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}] which is a quorum; discovery will continue using [172.31.33.10:9300, 172.31.11.195:9300] from hosts providers and [{elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 28, last-accepted version 4 in term 28"  }
  524. {"type": "server", "timestamp": "2019-07-01T18:19:17,653+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "failed to join {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=34, lastAcceptedTerm=28, lastAcceptedVersion=4, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}}]}" ,
  525. "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-0][172.17.0.2:9300][internal:cluster/coordination/join]",
  526. "Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming last accepted term 28 of join higher than current last accepted term 0",
  527. "at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:230) ~[elasticsearch-7.2.0.jar:7.2.0]",
  528. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:942) ~[elasticsearch-7.2.0.jar:7.2.0]",
  529. "at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]",
  530. "at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:511) ~[elasticsearch-7.2.0.jar:7.2.0]",
  531. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:478) ~[elasticsearch-7.2.0.jar:7.2.0]",
  532. "at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:124) ~[elasticsearch-7.2.0.jar:7.2.0]",
  533. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) ~[?:?]",
  534. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  535. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:308) ~[?:?]",
  536. "at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.2.0.jar:7.2.0]",
  537. "at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.2.0.jar:7.2.0]",
  538. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.2.0.jar:7.2.0]",
  539. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  540. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]",
  541. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]",
  542. "at java.lang.Thread.run(Thread.java:835) [?:?]"] }
  543. {"type": "server", "timestamp": "2019-07-01T18:19:20,929+0000", "level": "WARN", "component": "o.e.n.Node", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "timed out while waiting for initial discovery state - timeout: 30s"  }
  544. {"type": "server", "timestamp": "2019-07-01T18:19:20,936+0000", "level": "INFO", "component": "o.e.h.AbstractHttpServerTransport", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "publish_address {172.31.29.199:9200}, bound_addresses {0.0.0.0:9200}"  }
  545. {"type": "server", "timestamp": "2019-07-01T18:19:20,937+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "started"  }
  546. {"type": "server", "timestamp": "2019-07-01T18:19:27,362+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "last failed join attempt was 9.7s ago, failed to join {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=34, lastAcceptedTerm=28, lastAcceptedVersion=4, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}}]}" ,
  547. "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-0][172.17.0.2:9300][internal:cluster/coordination/join]",
  548. "Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming last accepted term 28 of join higher than current last accepted term 0",
  549. "at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:230) ~[elasticsearch-7.2.0.jar:7.2.0]",
  550. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:942) ~[elasticsearch-7.2.0.jar:7.2.0]",
  551. "at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]",
  552. "at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:511) ~[elasticsearch-7.2.0.jar:7.2.0]",
  553. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:478) ~[elasticsearch-7.2.0.jar:7.2.0]",
  554. "at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:124) ~[elasticsearch-7.2.0.jar:7.2.0]",
  555. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) ~[?:?]",
  556. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  557. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:308) ~[?:?]",
  558. "at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.2.0.jar:7.2.0]",
  559. "at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.2.0.jar:7.2.0]",
  560. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.2.0.jar:7.2.0]",
  561. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]",
  562. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
  563. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
  564. "at java.lang.Thread.run(Thread.java:835) [?:?]"] }
  565. {"type": "server", "timestamp": "2019-07-01T18:19:27,366+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "master not discovered or elected yet, an election requires 2 nodes with ids [NtLoCKHvSVWEO5hUT6jT2Q, lwLc0G2zQEW3-bZDvnsKmg], have discovered [{elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}] which is a quorum; discovery will continue using [172.31.33.10:9300, 172.31.11.195:9300] from hosts providers and [{elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 34, last-accepted version 4 in term 28"  }
  566. {"type": "server", "timestamp": "2019-07-01T18:19:37,368+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "master not discovered or elected yet, an election requires 2 nodes with ids [NtLoCKHvSVWEO5hUT6jT2Q, lwLc0G2zQEW3-bZDvnsKmg], have discovered [{elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}] which is a quorum; discovery will continue using [172.31.33.10:9300, 172.31.11.195:9300] from hosts providers and [{elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 34, last-accepted version 4 in term 28"  }
  567. {"type": "server", "timestamp": "2019-07-01T18:19:42,680+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "failed to join {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=44, lastAcceptedTerm=28, lastAcceptedVersion=4, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}}]}" ,
  568. "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-0][172.17.0.2:9300][internal:cluster/coordination/join]",
  569. "Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming last accepted term 28 of join higher than current last accepted term 0",
  570. "at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:230) ~[elasticsearch-7.2.0.jar:7.2.0]",
  571. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:942) ~[elasticsearch-7.2.0.jar:7.2.0]",
  572. "at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]",
  573. "at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:511) ~[elasticsearch-7.2.0.jar:7.2.0]",
  574. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:478) ~[elasticsearch-7.2.0.jar:7.2.0]",
  575. "at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:124) ~[elasticsearch-7.2.0.jar:7.2.0]",
  576. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) ~[?:?]",
  577. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  578. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:308) ~[?:?]",
  579. "at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.2.0.jar:7.2.0]",
  580. "at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.2.0.jar:7.2.0]",
  581. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.2.0.jar:7.2.0]",
  582. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  583. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]",
  584. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]",
  585. "at java.lang.Thread.run(Thread.java:835) [?:?]"] }
  586. {"type": "server", "timestamp": "2019-07-01T18:19:47,369+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "last failed join attempt was 4.6s ago, failed to join {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=44, lastAcceptedTerm=28, lastAcceptedVersion=4, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}}]}" ,
  587. "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-0][172.17.0.2:9300][internal:cluster/coordination/join]",
  588. "Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming last accepted term 28 of join higher than current last accepted term 0",
  589. "at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:230) ~[elasticsearch-7.2.0.jar:7.2.0]",
  590. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:942) ~[elasticsearch-7.2.0.jar:7.2.0]",
  591. "at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]",
  592. "at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:511) ~[elasticsearch-7.2.0.jar:7.2.0]",
  593. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:478) ~[elasticsearch-7.2.0.jar:7.2.0]",
  594. "at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:124) ~[elasticsearch-7.2.0.jar:7.2.0]",
  595. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) ~[?:?]",
  596. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  597. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:308) ~[?:?]",
  598. "at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.2.0.jar:7.2.0]",
  599. "at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.2.0.jar:7.2.0]",
  600. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.2.0.jar:7.2.0]",
  601. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]",
  602. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
  603. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
  604. "at java.lang.Thread.run(Thread.java:835) [?:?]"] }
  605. {"type": "server", "timestamp": "2019-07-01T18:19:47,372+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "master not discovered or elected yet, an election requires 2 nodes with ids [NtLoCKHvSVWEO5hUT6jT2Q, lwLc0G2zQEW3-bZDvnsKmg], have discovered [{elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}] which is a quorum; discovery will continue using [172.31.33.10:9300, 172.31.11.195:9300] from hosts providers and [{elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 44, last-accepted version 4 in term 28"  }
  606. {"type": "server", "timestamp": "2019-07-01T18:19:57,374+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "master not discovered or elected yet, an election requires 2 nodes with ids [NtLoCKHvSVWEO5hUT6jT2Q, lwLc0G2zQEW3-bZDvnsKmg], have discovered [{elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}] which is a quorum; discovery will continue using [172.31.33.10:9300, 172.31.11.195:9300] from hosts providers and [{elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 44, last-accepted version 4 in term 28"  }
  607. {"type": "server", "timestamp": "2019-07-01T18:20:03,379+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "failed to join {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=48, lastAcceptedTerm=28, lastAcceptedVersion=4, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}}]}" ,
  608. "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-0][172.17.0.2:9300][internal:cluster/coordination/join]",
  609. "Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming last accepted term 28 of join higher than current last accepted term 0",
  610. "at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:230) ~[elasticsearch-7.2.0.jar:7.2.0]",
  611. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:942) ~[elasticsearch-7.2.0.jar:7.2.0]",
  612. "at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]",
  613. "at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:511) ~[elasticsearch-7.2.0.jar:7.2.0]",
  614. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:478) ~[elasticsearch-7.2.0.jar:7.2.0]",
  615. "at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:124) ~[elasticsearch-7.2.0.jar:7.2.0]",
  616. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) ~[?:?]",
  617. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  618. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:308) ~[?:?]",
  619. "at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.2.0.jar:7.2.0]",
  620. "at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.2.0.jar:7.2.0]",
  621. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.2.0.jar:7.2.0]",
  622. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  623. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]",
  624. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]",
  625. "at java.lang.Thread.run(Thread.java:835) [?:?]"] }
  626. {"type": "server", "timestamp": "2019-07-01T18:20:07,376+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "last failed join attempt was 3.9s ago, failed to join {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=48, lastAcceptedTerm=28, lastAcceptedVersion=4, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}}]}" ,
  627. "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-0][172.17.0.2:9300][internal:cluster/coordination/join]",
  628. "Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming last accepted term 28 of join higher than current last accepted term 0",
  629. "at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:230) ~[elasticsearch-7.2.0.jar:7.2.0]",
  630. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:942) ~[elasticsearch-7.2.0.jar:7.2.0]",
  631. "at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]",
  632. "at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:511) ~[elasticsearch-7.2.0.jar:7.2.0]",
  633. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:478) ~[elasticsearch-7.2.0.jar:7.2.0]",
  634. "at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:124) ~[elasticsearch-7.2.0.jar:7.2.0]",
  635. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) ~[?:?]",
  636. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  637. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:308) ~[?:?]",
  638. "at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.2.0.jar:7.2.0]",
  639. "at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.2.0.jar:7.2.0]",
  640. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.2.0.jar:7.2.0]",
  641. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]",
  642. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
  643. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
  644. "at java.lang.Thread.run(Thread.java:835) [?:?]"] }
  645. {"type": "server", "timestamp": "2019-07-01T18:20:07,383+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "master not discovered or elected yet, an election requires 2 nodes with ids [NtLoCKHvSVWEO5hUT6jT2Q, lwLc0G2zQEW3-bZDvnsKmg], have discovered [{elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}] which is a quorum; discovery will continue using [172.31.33.10:9300, 172.31.11.195:9300] from hosts providers and [{elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 48, last-accepted version 4 in term 28"  }
  646. {"type": "server", "timestamp": "2019-07-01T18:20:17,384+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "master not discovered or elected yet, an election requires 2 nodes with ids [NtLoCKHvSVWEO5hUT6jT2Q, lwLc0G2zQEW3-bZDvnsKmg], have discovered [{elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}] which is a quorum; discovery will continue using [172.31.33.10:9300, 172.31.11.195:9300] from hosts providers and [{elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 50, last-accepted version 4 in term 28"  }
  647. {"type": "server", "timestamp": "2019-07-01T18:20:27,387+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "master not discovered or elected yet, an election requires 2 nodes with ids [NtLoCKHvSVWEO5hUT6jT2Q, lwLc0G2zQEW3-bZDvnsKmg], have discovered [{elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}] which is a quorum; discovery will continue using [172.31.33.10:9300, 172.31.11.195:9300] from hosts providers and [{elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 50, last-accepted version 4 in term 28"  }
  648. {"type": "server", "timestamp": "2019-07-01T18:20:35,640+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "failed to join {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=50, lastAcceptedTerm=28, lastAcceptedVersion=4, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}}]}" ,
  649. "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-2][172.17.0.2:9300][internal:cluster/coordination/join]",
  650. "Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: received a newer join from {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}",
  651. "at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.handleJoinRequest(JoinHelper.java:451) ~[elasticsearch-7.2.0.jar:7.2.0]",
  652. "at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:512) ~[elasticsearch-7.2.0.jar:7.2.0]",
  653. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:478) ~[elasticsearch-7.2.0.jar:7.2.0]",
  654. "at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:124) ~[elasticsearch-7.2.0.jar:7.2.0]",
  655. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) ~[?:?]",
  656. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  657. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:308) ~[?:?]",
  658. "at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.2.0.jar:7.2.0]",
  659. "at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.2.0.jar:7.2.0]",
  660. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.2.0.jar:7.2.0]",
  661. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  662. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]",
  663. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]",
  664. "at java.lang.Thread.run(Thread.java:835) [?:?]"] }
  665. {"type": "server", "timestamp": "2019-07-01T18:20:37,388+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "last failed join attempt was 1.7s ago, failed to join {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=50, lastAcceptedTerm=28, lastAcceptedVersion=4, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}}]}" ,
  666. "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-2][172.17.0.2:9300][internal:cluster/coordination/join]",
  667. "Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: received a newer join from {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}",
  668. "at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.handleJoinRequest(JoinHelper.java:451) ~[elasticsearch-7.2.0.jar:7.2.0]",
  669. "at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:512) ~[elasticsearch-7.2.0.jar:7.2.0]",
  670. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:478) ~[elasticsearch-7.2.0.jar:7.2.0]",
  671. "at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:124) ~[elasticsearch-7.2.0.jar:7.2.0]",
  672. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) ~[?:?]",
  673. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  674. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:308) ~[?:?]",
  675. "at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.2.0.jar:7.2.0]",
  676. "at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.2.0.jar:7.2.0]",
  677. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.2.0.jar:7.2.0]",
  678. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]",
  679. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
  680. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
  681. "at java.lang.Thread.run(Thread.java:835) [?:?]"] }
  682. {"type": "server", "timestamp": "2019-07-01T18:20:37,390+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "master not discovered or elected yet, an election requires 2 nodes with ids [NtLoCKHvSVWEO5hUT6jT2Q, lwLc0G2zQEW3-bZDvnsKmg], have discovered [{elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}] which is a quorum; discovery will continue using [172.31.33.10:9300, 172.31.11.195:9300] from hosts providers and [{elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 57, last-accepted version 4 in term 28"  }
  683. {"type": "server", "timestamp": "2019-07-01T18:20:47,391+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "master not discovered or elected yet, an election requires 2 nodes with ids [NtLoCKHvSVWEO5hUT6jT2Q, lwLc0G2zQEW3-bZDvnsKmg], have discovered [{elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}] which is a quorum; discovery will continue using [172.31.33.10:9300, 172.31.11.195:9300] from hosts providers and [{elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 57, last-accepted version 4 in term 28"  }
  684. {"type": "server", "timestamp": "2019-07-01T18:20:56,721+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "failed to join {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=57, lastAcceptedTerm=28, lastAcceptedVersion=4, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}}]}" ,
  685. "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-2][172.17.0.2:9300][internal:cluster/coordination/join]",
  686. "Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: received a newer join from {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}",
  687. "at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.handleJoinRequest(JoinHelper.java:451) ~[elasticsearch-7.2.0.jar:7.2.0]",
  688. "at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:512) ~[elasticsearch-7.2.0.jar:7.2.0]",
  689. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:478) ~[elasticsearch-7.2.0.jar:7.2.0]",
  690. "at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:124) ~[elasticsearch-7.2.0.jar:7.2.0]",
  691. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) ~[?:?]",
  692. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  693. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:308) ~[?:?]",
  694. "at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.2.0.jar:7.2.0]",
  695. "at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.2.0.jar:7.2.0]",
  696. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.2.0.jar:7.2.0]",
  697. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  698. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]",
  699. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]",
  700. "at java.lang.Thread.run(Thread.java:835) [?:?]"] }
  701. {"type": "server", "timestamp": "2019-07-01T18:20:57,395+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "last failed join attempt was 669ms ago, failed to join {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=57, lastAcceptedTerm=28, lastAcceptedVersion=4, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}}]}" ,
  702. "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-2][172.17.0.2:9300][internal:cluster/coordination/join]",
  703. "Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: received a newer join from {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}",
  704. "at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.handleJoinRequest(JoinHelper.java:451) ~[elasticsearch-7.2.0.jar:7.2.0]",
  705. "at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:512) ~[elasticsearch-7.2.0.jar:7.2.0]",
  706. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:478) ~[elasticsearch-7.2.0.jar:7.2.0]",
  707. "at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:124) ~[elasticsearch-7.2.0.jar:7.2.0]",
  708. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) ~[?:?]",
  709. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  710. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:308) ~[?:?]",
  711. "at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.2.0.jar:7.2.0]",
  712. "at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.2.0.jar:7.2.0]",
  713. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.2.0.jar:7.2.0]",
  714. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]",
  715. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
  716. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
  717. "at java.lang.Thread.run(Thread.java:835) [?:?]"] }
  718. {"type": "server", "timestamp": "2019-07-01T18:20:57,397+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "master not discovered or elected yet, an election requires 2 nodes with ids [NtLoCKHvSVWEO5hUT6jT2Q, lwLc0G2zQEW3-bZDvnsKmg], have discovered [{elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}] which is a quorum; discovery will continue using [172.31.33.10:9300, 172.31.11.195:9300] from hosts providers and [{elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 60, last-accepted version 4 in term 28"  }
  719. {"type": "server", "timestamp": "2019-07-01T18:21:07,399+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "master not discovered or elected yet, an election requires 2 nodes with ids [NtLoCKHvSVWEO5hUT6jT2Q, lwLc0G2zQEW3-bZDvnsKmg], have discovered [{elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}] which is a quorum; discovery will continue using [172.31.33.10:9300, 172.31.11.195:9300] from hosts providers and [{elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 60, last-accepted version 4 in term 28"  }
  720. {"type": "server", "timestamp": "2019-07-01T18:21:17,401+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "master not discovered or elected yet, an election requires 2 nodes with ids [NtLoCKHvSVWEO5hUT6jT2Q, lwLc0G2zQEW3-bZDvnsKmg], have discovered [{elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}] which is a quorum; discovery will continue using [172.31.33.10:9300, 172.31.11.195:9300] from hosts providers and [{elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 60, last-accepted version 4 in term 28"  }
  721. {"type": "server", "timestamp": "2019-07-01T18:21:27,403+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "master not discovered or elected yet, an election requires 2 nodes with ids [NtLoCKHvSVWEO5hUT6jT2Q, lwLc0G2zQEW3-bZDvnsKmg], have discovered [{elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}] which is a quorum; discovery will continue using [172.31.33.10:9300, 172.31.11.195:9300] from hosts providers and [{elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 60, last-accepted version 4 in term 28"  }
  722. {"type": "server", "timestamp": "2019-07-01T18:21:34,365+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "failed to join {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=66, lastAcceptedTerm=28, lastAcceptedVersion=4, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}}]}" ,
  723. "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-0][172.17.0.2:9300][internal:cluster/coordination/join]",
  724. "Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming last accepted term 28 of join higher than current last accepted term 0",
  725. "at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:230) ~[elasticsearch-7.2.0.jar:7.2.0]",
  726. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:942) ~[elasticsearch-7.2.0.jar:7.2.0]",
  727. "at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]",
  728. "at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:511) ~[elasticsearch-7.2.0.jar:7.2.0]",
  729. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:478) ~[elasticsearch-7.2.0.jar:7.2.0]",
  730. "at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:124) ~[elasticsearch-7.2.0.jar:7.2.0]",
  731. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) ~[?:?]",
  732. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  733. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:308) ~[?:?]",
  734. "at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.2.0.jar:7.2.0]",
  735. "at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.2.0.jar:7.2.0]",
  736. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.2.0.jar:7.2.0]",
  737. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  738. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]",
  739. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]",
  740. "at java.lang.Thread.run(Thread.java:835) [?:?]"] }
  741. {"type": "server", "timestamp": "2019-07-01T18:21:37,404+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "last failed join attempt was 3s ago, failed to join {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=66, lastAcceptedTerm=28, lastAcceptedVersion=4, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}}]}" ,
  742. "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-0][172.17.0.2:9300][internal:cluster/coordination/join]",
  743. "Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming last accepted term 28 of join higher than current last accepted term 0",
  744. "at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:230) ~[elasticsearch-7.2.0.jar:7.2.0]",
  745. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:942) ~[elasticsearch-7.2.0.jar:7.2.0]",
  746. "at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]",
  747. "at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:511) ~[elasticsearch-7.2.0.jar:7.2.0]",
  748. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:478) ~[elasticsearch-7.2.0.jar:7.2.0]",
  749. "at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:124) ~[elasticsearch-7.2.0.jar:7.2.0]",
  750. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) ~[?:?]",
  751. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  752. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:308) ~[?:?]",
  753. "at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.2.0.jar:7.2.0]",
  754. "at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.2.0.jar:7.2.0]",
  755. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.2.0.jar:7.2.0]",
  756. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]",
  757. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
  758. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
  759. "at java.lang.Thread.run(Thread.java:835) [?:?]"] }
  760. {"type": "server", "timestamp": "2019-07-01T18:21:37,407+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "master not discovered or elected yet, an election requires 2 nodes with ids [NtLoCKHvSVWEO5hUT6jT2Q, lwLc0G2zQEW3-bZDvnsKmg], have discovered [{elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}] which is a quorum; discovery will continue using [172.31.33.10:9300, 172.31.11.195:9300] from hosts providers and [{elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 66, last-accepted version 4 in term 28"  }
  761. {"type": "server", "timestamp": "2019-07-01T18:21:42,303+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "failed to join {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=67, lastAcceptedTerm=28, lastAcceptedVersion=4, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}}]}" ,
  762. "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-0][172.17.0.2:9300][internal:cluster/coordination/join]",
  763. "Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming last accepted term 28 of join higher than current last accepted term 0",
  764. "at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:230) ~[elasticsearch-7.2.0.jar:7.2.0]",
  765. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:942) ~[elasticsearch-7.2.0.jar:7.2.0]",
  766. "at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]",
  767. "at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:511) ~[elasticsearch-7.2.0.jar:7.2.0]",
  768. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:478) ~[elasticsearch-7.2.0.jar:7.2.0]",
  769. "at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:124) ~[elasticsearch-7.2.0.jar:7.2.0]",
  770. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) ~[?:?]",
  771. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  772. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:308) ~[?:?]",
  773. "at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.2.0.jar:7.2.0]",
  774. "at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.2.0.jar:7.2.0]",
  775. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.2.0.jar:7.2.0]",
  776. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  777. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]",
  778. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]",
  779. "at java.lang.Thread.run(Thread.java:835) [?:?]"] }
  780. {"type": "server", "timestamp": "2019-07-01T18:21:47,408+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "last failed join attempt was 5.1s ago, failed to join {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=67, lastAcceptedTerm=28, lastAcceptedVersion=4, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}}]}" ,
  781. "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-0][172.17.0.2:9300][internal:cluster/coordination/join]",
  782. "Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming last accepted term 28 of join higher than current last accepted term 0",
  783. "at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:230) ~[elasticsearch-7.2.0.jar:7.2.0]",
  784. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:942) ~[elasticsearch-7.2.0.jar:7.2.0]",
  785. "at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]",
  786. "at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:511) ~[elasticsearch-7.2.0.jar:7.2.0]",
  787. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:478) ~[elasticsearch-7.2.0.jar:7.2.0]",
  788. "at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:124) ~[elasticsearch-7.2.0.jar:7.2.0]",
  789. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) ~[?:?]",
  790. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  791. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:308) ~[?:?]",
  792. "at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.2.0.jar:7.2.0]",
  793. "at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.2.0.jar:7.2.0]",
  794. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.2.0.jar:7.2.0]",
  795. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]",
  796. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
  797. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
  798. "at java.lang.Thread.run(Thread.java:835) [?:?]"] }
  799. {"type": "server", "timestamp": "2019-07-01T18:21:47,410+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "master not discovered or elected yet, an election requires 2 nodes with ids [NtLoCKHvSVWEO5hUT6jT2Q, lwLc0G2zQEW3-bZDvnsKmg], have discovered [{elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}] which is a quorum; discovery will continue using [172.31.33.10:9300, 172.31.11.195:9300] from hosts providers and [{elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 67, last-accepted version 4 in term 28"  }
  800. {"type": "server", "timestamp": "2019-07-01T18:21:56,719+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "failed to join {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=60, lastAcceptedTerm=28, lastAcceptedVersion=4, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}}]}" ,
  801. "stacktrace": ["org.elasticsearch.transport.ReceiveTimeoutTransportException: [elasticsearch-master-0][172.31.33.10:9300][internal:cluster/coordination/join] request_id [392] timed out after [60029ms]",
  802. "at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:1013) [elasticsearch-7.2.0.jar:7.2.0]",
  803. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688) [elasticsearch-7.2.0.jar:7.2.0]",
  804. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
  805. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
  806. "at java.lang.Thread.run(Thread.java:835) [?:?]"] }
  807. {"type": "server", "timestamp": "2019-07-01T18:21:56,720+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "failed to join {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=60, lastAcceptedTerm=28, lastAcceptedVersion=4, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}}]}" ,
  808. "stacktrace": ["org.elasticsearch.transport.ReceiveTimeoutTransportException: [elasticsearch-master-0][172.31.33.10:9300][internal:cluster/coordination/join] request_id [392] timed out after [60029ms]",
  809. "at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:1013) [elasticsearch-7.2.0.jar:7.2.0]",
  810. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688) [elasticsearch-7.2.0.jar:7.2.0]",
  811. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
  812. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
  813. "at java.lang.Thread.run(Thread.java:835) [?:?]"] }
  814. {"type": "server", "timestamp": "2019-07-01T18:21:57,411+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "last failed join attempt was 690ms ago, failed to join {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=60, lastAcceptedTerm=28, lastAcceptedVersion=4, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}}]}" ,
  815. "stacktrace": ["org.elasticsearch.transport.ReceiveTimeoutTransportException: [elasticsearch-master-0][172.31.33.10:9300][internal:cluster/coordination/join] request_id [392] timed out after [60029ms]",
  816. "at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:1013) ~[elasticsearch-7.2.0.jar:7.2.0]",
  817. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688) ~[elasticsearch-7.2.0.jar:7.2.0]",
  818. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
  819. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
  820. "at java.lang.Thread.run(Thread.java:835) [?:?]"] }
  821. {"type": "server", "timestamp": "2019-07-01T18:21:57,413+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "master not discovered or elected yet, an election requires 2 nodes with ids [NtLoCKHvSVWEO5hUT6jT2Q, lwLc0G2zQEW3-bZDvnsKmg], have discovered [{elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}] which is a quorum; discovery will continue using [172.31.33.10:9300, 172.31.11.195:9300] from hosts providers and [{elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 67, last-accepted version 4 in term 28"  }
  822. {"type": "server", "timestamp": "2019-07-01T18:22:07,414+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "master not discovered or elected yet, an election requires 2 nodes with ids [NtLoCKHvSVWEO5hUT6jT2Q, lwLc0G2zQEW3-bZDvnsKmg], have discovered [{elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}] which is a quorum; discovery will continue using [172.31.33.10:9300, 172.31.11.195:9300] from hosts providers and [{elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 67, last-accepted version 4 in term 28"  }
  823. {"type": "server", "timestamp": "2019-07-01T18:22:09,224+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "failed to join {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=70, lastAcceptedTerm=28, lastAcceptedVersion=4, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}}]}" ,
  824. "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-0][172.17.0.2:9300][internal:cluster/coordination/join]",
  825. "Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming last accepted term 28 of join higher than current last accepted term 0",
  826. "at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:230) ~[elasticsearch-7.2.0.jar:7.2.0]",
  827. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:942) ~[elasticsearch-7.2.0.jar:7.2.0]",
  828. "at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]",
  829. "at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:511) ~[elasticsearch-7.2.0.jar:7.2.0]",
  830. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:478) ~[elasticsearch-7.2.0.jar:7.2.0]",
  831. "at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:124) ~[elasticsearch-7.2.0.jar:7.2.0]",
  832. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) ~[?:?]",
  833. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  834. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:308) ~[?:?]",
  835. "at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.2.0.jar:7.2.0]",
  836. "at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.2.0.jar:7.2.0]",
  837. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.2.0.jar:7.2.0]",
  838. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  839. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]",
  840. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]",
  841. "at java.lang.Thread.run(Thread.java:835) [?:?]"] }
  842. {"type": "server", "timestamp": "2019-07-01T18:22:16,605+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "failed to join {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=71, lastAcceptedTerm=28, lastAcceptedVersion=4, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}}]}" ,
  843. "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-0][172.17.0.2:9300][internal:cluster/coordination/join]",
  844. "Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming last accepted term 28 of join higher than current last accepted term 0",
  845. "at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:230) ~[elasticsearch-7.2.0.jar:7.2.0]",
  846. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:942) ~[elasticsearch-7.2.0.jar:7.2.0]",
  847. "at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]",
  848. "at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:511) ~[elasticsearch-7.2.0.jar:7.2.0]",
  849. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:478) ~[elasticsearch-7.2.0.jar:7.2.0]",
  850. "at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:124) ~[elasticsearch-7.2.0.jar:7.2.0]",
  851. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) ~[?:?]",
  852. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  853. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:308) ~[?:?]",
  854. "at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.2.0.jar:7.2.0]",
  855. "at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.2.0.jar:7.2.0]",
  856. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.2.0.jar:7.2.0]",
  857. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  858. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]",
  859. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]",
  860. "at java.lang.Thread.run(Thread.java:835) [?:?]"] }
  861. {"type": "server", "timestamp": "2019-07-01T18:22:17,416+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "last failed join attempt was 808ms ago, failed to join {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=71, lastAcceptedTerm=28, lastAcceptedVersion=4, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}}]}" ,
  862. "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-0][172.17.0.2:9300][internal:cluster/coordination/join]",
  863. "Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming last accepted term 28 of join higher than current last accepted term 0",
  864. "at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:230) ~[elasticsearch-7.2.0.jar:7.2.0]",
  865. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:942) ~[elasticsearch-7.2.0.jar:7.2.0]",
  866. "at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]",
  867. "at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:511) ~[elasticsearch-7.2.0.jar:7.2.0]",
  868. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:478) ~[elasticsearch-7.2.0.jar:7.2.0]",
  869. "at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:124) ~[elasticsearch-7.2.0.jar:7.2.0]",
  870. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) ~[?:?]",
  871. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  872. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:308) ~[?:?]",
  873. "at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.2.0.jar:7.2.0]",
  874. "at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.2.0.jar:7.2.0]",
  875. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.2.0.jar:7.2.0]",
  876. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]",
  877. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
  878. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
  879. "at java.lang.Thread.run(Thread.java:835) [?:?]"] }
  880. {"type": "server", "timestamp": "2019-07-01T18:22:17,419+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "master not discovered or elected yet, an election requires 2 nodes with ids [NtLoCKHvSVWEO5hUT6jT2Q, lwLc0G2zQEW3-bZDvnsKmg], have discovered [{elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}] which is a quorum; discovery will continue using [172.31.33.10:9300, 172.31.11.195:9300] from hosts providers and [{elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 71, last-accepted version 4 in term 28"  }
  881. {"type": "server", "timestamp": "2019-07-01T18:22:27,421+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "master not discovered or elected yet, an election requires 2 nodes with ids [NtLoCKHvSVWEO5hUT6jT2Q, lwLc0G2zQEW3-bZDvnsKmg], have discovered [{elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}] which is a quorum; discovery will continue using [172.31.33.10:9300, 172.31.11.195:9300] from hosts providers and [{elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 71, last-accepted version 4 in term 28"  }
  882. {"type": "server", "timestamp": "2019-07-01T18:22:29,994+0000", "level": "WARN", "component": "o.e.t.TransportService", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "Received response for a request that has timed out, sent [93245ms] ago, timed out [33216ms] ago, action [internal:cluster/coordination/join], node [{elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}], id [392]"  }
  883. {"type": "server", "timestamp": "2019-07-01T18:22:35,093+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "failed to join {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=73, lastAcceptedTerm=28, lastAcceptedVersion=4, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}}]}" ,
  884. "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-0][172.17.0.2:9300][internal:cluster/coordination/join]",
  885. "Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming last accepted term 28 of join higher than current last accepted term 0",
  886. "at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:230) ~[elasticsearch-7.2.0.jar:7.2.0]",
  887. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:942) ~[elasticsearch-7.2.0.jar:7.2.0]",
  888. "at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]",
  889. "at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:511) ~[elasticsearch-7.2.0.jar:7.2.0]",
  890. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:478) ~[elasticsearch-7.2.0.jar:7.2.0]",
  891. "at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:124) ~[elasticsearch-7.2.0.jar:7.2.0]",
  892. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) ~[?:?]",
  893. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  894. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:308) ~[?:?]",
  895. "at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.2.0.jar:7.2.0]",
  896. "at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.2.0.jar:7.2.0]",
  897. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.2.0.jar:7.2.0]",
  898. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  899. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]",
  900. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]",
  901. "at java.lang.Thread.run(Thread.java:835) [?:?]"] }
  902. {"type": "server", "timestamp": "2019-07-01T18:22:37,422+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "last failed join attempt was 2.3s ago, failed to join {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=73, lastAcceptedTerm=28, lastAcceptedVersion=4, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}}]}" ,
  903. "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-0][172.17.0.2:9300][internal:cluster/coordination/join]",
  904. "Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming last accepted term 28 of join higher than current last accepted term 0",
  905. "at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:230) ~[elasticsearch-7.2.0.jar:7.2.0]",
  906. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:942) ~[elasticsearch-7.2.0.jar:7.2.0]",
  907. "at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]",
  908. "at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:511) ~[elasticsearch-7.2.0.jar:7.2.0]",
  909. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:478) ~[elasticsearch-7.2.0.jar:7.2.0]",
  910. "at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:124) ~[elasticsearch-7.2.0.jar:7.2.0]",
  911. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) ~[?:?]",
  912. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  913. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:308) ~[?:?]",
  914. "at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.2.0.jar:7.2.0]",
  915. "at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.2.0.jar:7.2.0]",
  916. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.2.0.jar:7.2.0]",
  917. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]",
  918. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
  919. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
  920. "at java.lang.Thread.run(Thread.java:835) [?:?]"] }
  921. {"type": "server", "timestamp": "2019-07-01T18:22:37,426+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "master not discovered or elected yet, an election requires 2 nodes with ids [NtLoCKHvSVWEO5hUT6jT2Q, lwLc0G2zQEW3-bZDvnsKmg], have discovered [{elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}] which is a quorum; discovery will continue using [172.31.33.10:9300, 172.31.11.195:9300] from hosts providers and [{elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 73, last-accepted version 4 in term 28"  }
  922. {"type": "server", "timestamp": "2019-07-01T18:22:44,341+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "failed to join {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=72, lastAcceptedTerm=28, lastAcceptedVersion=4, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}}]}" ,
  923. "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-2][172.17.0.2:9300][internal:cluster/coordination/join]",
  924. "Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: received a newer join from {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}",
  925. "at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.handleJoinRequest(JoinHelper.java:451) ~[elasticsearch-7.2.0.jar:7.2.0]",
  926. "at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:512) ~[elasticsearch-7.2.0.jar:7.2.0]",
  927. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:478) ~[elasticsearch-7.2.0.jar:7.2.0]",
  928. "at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:124) ~[elasticsearch-7.2.0.jar:7.2.0]",
  929. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) ~[?:?]",
  930. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  931. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:308) ~[?:?]",
  932. "at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.2.0.jar:7.2.0]",
  933. "at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.2.0.jar:7.2.0]",
  934. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.2.0.jar:7.2.0]",
  935. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  936. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]",
  937. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]",
  938. "at java.lang.Thread.run(Thread.java:835) [?:?]"] }
  939. {"type": "server", "timestamp": "2019-07-01T18:22:46,365+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "failed to join {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=76, lastAcceptedTerm=28, lastAcceptedVersion=4, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}}]}" ,
  940. "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-0][172.17.0.2:9300][internal:cluster/coordination/join]",
  941. "Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming last accepted term 28 of join higher than current last accepted term 0",
  942. "at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:230) ~[elasticsearch-7.2.0.jar:7.2.0]",
  943. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:942) ~[elasticsearch-7.2.0.jar:7.2.0]",
  944. "at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]",
  945. "at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:511) ~[elasticsearch-7.2.0.jar:7.2.0]",
  946. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:478) ~[elasticsearch-7.2.0.jar:7.2.0]",
  947. "at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:124) ~[elasticsearch-7.2.0.jar:7.2.0]",
  948. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) ~[?:?]",
  949. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  950. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:308) ~[?:?]",
  951. "at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.2.0.jar:7.2.0]",
  952. "at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.2.0.jar:7.2.0]",
  953. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.2.0.jar:7.2.0]",
  954. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  955. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]",
  956. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]",
  957. "at java.lang.Thread.run(Thread.java:835) [?:?]"] }
  958. {"type": "server", "timestamp": "2019-07-01T18:22:47,427+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "last failed join attempt was 1s ago, failed to join {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=76, lastAcceptedTerm=28, lastAcceptedVersion=4, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}}]}" ,
  959. "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-0][172.17.0.2:9300][internal:cluster/coordination/join]",
  960. "Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming last accepted term 28 of join higher than current last accepted term 0",
  961. "at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:230) ~[elasticsearch-7.2.0.jar:7.2.0]",
  962. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:942) ~[elasticsearch-7.2.0.jar:7.2.0]",
  963. "at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]",
  964. "at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:511) ~[elasticsearch-7.2.0.jar:7.2.0]",
  965. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:478) ~[elasticsearch-7.2.0.jar:7.2.0]",
  966. "at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:124) ~[elasticsearch-7.2.0.jar:7.2.0]",
  967. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) ~[?:?]",
  968. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  969. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:308) ~[?:?]",
  970. "at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.2.0.jar:7.2.0]",
  971. "at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.2.0.jar:7.2.0]",
  972. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.2.0.jar:7.2.0]",
  973. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]",
  974. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
  975. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
  976. "at java.lang.Thread.run(Thread.java:835) [?:?]"] }
  977. {"type": "server", "timestamp": "2019-07-01T18:22:47,431+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "master not discovered or elected yet, an election requires 2 nodes with ids [NtLoCKHvSVWEO5hUT6jT2Q, lwLc0G2zQEW3-bZDvnsKmg], have discovered [{elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}] which is a quorum; discovery will continue using [172.31.33.10:9300, 172.31.11.195:9300] from hosts providers and [{elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 76, last-accepted version 4 in term 28"  }
  978. {"type": "server", "timestamp": "2019-07-01T18:22:57,433+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "master not discovered or elected yet, an election requires 2 nodes with ids [NtLoCKHvSVWEO5hUT6jT2Q, lwLc0G2zQEW3-bZDvnsKmg], have discovered [{elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}] which is a quorum; discovery will continue using [172.31.33.10:9300, 172.31.11.195:9300] from hosts providers and [{elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 76, last-accepted version 4 in term 28"  }
  979. {"type": "server", "timestamp": "2019-07-01T18:23:07,434+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "master not discovered or elected yet, an election requires 2 nodes with ids [NtLoCKHvSVWEO5hUT6jT2Q, lwLc0G2zQEW3-bZDvnsKmg], have discovered [{elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}] which is a quorum; discovery will continue using [172.31.33.10:9300, 172.31.11.195:9300] from hosts providers and [{elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 76, last-accepted version 4 in term 28"  }
  980. {"type": "server", "timestamp": "2019-07-01T18:23:17,436+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "master not discovered or elected yet, an election requires 2 nodes with ids [NtLoCKHvSVWEO5hUT6jT2Q, lwLc0G2zQEW3-bZDvnsKmg], have discovered [{elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}] which is a quorum; discovery will continue using [172.31.33.10:9300, 172.31.11.195:9300] from hosts providers and [{elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 76, last-accepted version 4 in term 28"  }
  981. {"type": "server", "timestamp": "2019-07-01T18:23:27,437+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "master not discovered or elected yet, an election requires 2 nodes with ids [NtLoCKHvSVWEO5hUT6jT2Q, lwLc0G2zQEW3-bZDvnsKmg], have discovered [{elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}] which is a quorum; discovery will continue using [172.31.33.10:9300, 172.31.11.195:9300] from hosts providers and [{elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 76, last-accepted version 4 in term 28"  }
  982. {"type": "server", "timestamp": "2019-07-01T18:23:37,438+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "master not discovered or elected yet, an election requires 2 nodes with ids [NtLoCKHvSVWEO5hUT6jT2Q, lwLc0G2zQEW3-bZDvnsKmg], have discovered [{elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}] which is a quorum; discovery will continue using [172.31.33.10:9300, 172.31.11.195:9300] from hosts providers and [{elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 76, last-accepted version 4 in term 28"  }
  983. {"type": "server", "timestamp": "2019-07-01T18:23:44,338+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "failed to join {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=75, lastAcceptedTerm=28, lastAcceptedVersion=4, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}}]}" ,
  984. "stacktrace": ["org.elasticsearch.transport.ReceiveTimeoutTransportException: [elasticsearch-master-0][172.31.33.10:9300][internal:cluster/coordination/join] request_id [613] timed out after [60028ms]",
  985. "at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:1013) [elasticsearch-7.2.0.jar:7.2.0]",
  986. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688) [elasticsearch-7.2.0.jar:7.2.0]",
  987. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
  988. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
  989. "at java.lang.Thread.run(Thread.java:835) [?:?]"] }
  990. {"type": "server", "timestamp": "2019-07-01T18:23:44,339+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "failed to join {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=75, lastAcceptedTerm=28, lastAcceptedVersion=4, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}}]}" ,
  991. "stacktrace": ["org.elasticsearch.transport.ReceiveTimeoutTransportException: [elasticsearch-master-0][172.31.33.10:9300][internal:cluster/coordination/join] request_id [613] timed out after [60028ms]",
  992. "at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:1013) [elasticsearch-7.2.0.jar:7.2.0]",
  993. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688) [elasticsearch-7.2.0.jar:7.2.0]",
  994. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
  995. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
  996. "at java.lang.Thread.run(Thread.java:835) [?:?]"] }
  997. {"type": "server", "timestamp": "2019-07-01T18:23:44,504+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "failed to join {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=83, lastAcceptedTerm=28, lastAcceptedVersion=4, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}}]}" ,
  998. "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-0][172.17.0.2:9300][internal:cluster/coordination/join]",
  999. "Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming last accepted term 28 of join higher than current last accepted term 0",
  1000. "at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:230) ~[elasticsearch-7.2.0.jar:7.2.0]",
  1001. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:942) ~[elasticsearch-7.2.0.jar:7.2.0]",
  1002. "at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]",
  1003. "at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:511) ~[elasticsearch-7.2.0.jar:7.2.0]",
  1004. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:478) ~[elasticsearch-7.2.0.jar:7.2.0]",
  1005. "at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:124) ~[elasticsearch-7.2.0.jar:7.2.0]",
  1006. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) ~[?:?]",
  1007. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  1008. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:308) ~[?:?]",
  1009. "at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.2.0.jar:7.2.0]",
  1010. "at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.2.0.jar:7.2.0]",
  1011. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.2.0.jar:7.2.0]",
  1012. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  1013. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]",
  1014. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]",
  1015. "at java.lang.Thread.run(Thread.java:835) [?:?]"] }
  1016. {"type": "server", "timestamp": "2019-07-01T18:23:47,439+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "last failed join attempt was 2.9s ago, failed to join {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=83, lastAcceptedTerm=28, lastAcceptedVersion=4, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}}]}" ,
  1017. "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-0][172.17.0.2:9300][internal:cluster/coordination/join]",
  1018. "Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming last accepted term 28 of join higher than current last accepted term 0",
  1019. "at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:230) ~[elasticsearch-7.2.0.jar:7.2.0]",
  1020. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:942) ~[elasticsearch-7.2.0.jar:7.2.0]",
  1021. "at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]",
  1022. "at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:511) ~[elasticsearch-7.2.0.jar:7.2.0]",
  1023. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:478) ~[elasticsearch-7.2.0.jar:7.2.0]",
  1024. "at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:124) ~[elasticsearch-7.2.0.jar:7.2.0]",
  1025. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) ~[?:?]",
  1026. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  1027. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:308) ~[?:?]",
  1028. "at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.2.0.jar:7.2.0]",
  1029. "at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.2.0.jar:7.2.0]",
  1030. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.2.0.jar:7.2.0]",
  1031. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]",
  1032. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
  1033. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
  1034. "at java.lang.Thread.run(Thread.java:835) [?:?]"] }
  1035. {"type": "server", "timestamp": "2019-07-01T18:23:47,442+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "master not discovered or elected yet, an election requires 2 nodes with ids [NtLoCKHvSVWEO5hUT6jT2Q, lwLc0G2zQEW3-bZDvnsKmg], have discovered [{elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}] which is a quorum; discovery will continue using [172.31.33.10:9300, 172.31.11.195:9300] from hosts providers and [{elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 83, last-accepted version 4 in term 28"  }
  1036. {"type": "server", "timestamp": "2019-07-01T18:23:50,006+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "failed to join {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=84, lastAcceptedTerm=28, lastAcceptedVersion=4, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}}]}" ,
  1037. "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-0][172.17.0.2:9300][internal:cluster/coordination/join]",
  1038. "Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming last accepted term 28 of join higher than current last accepted term 0",
  1039. "at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:230) ~[elasticsearch-7.2.0.jar:7.2.0]",
  1040. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:942) ~[elasticsearch-7.2.0.jar:7.2.0]",
  1041. "at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]",
  1042. "at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:511) ~[elasticsearch-7.2.0.jar:7.2.0]",
  1043. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:478) ~[elasticsearch-7.2.0.jar:7.2.0]",
  1044. "at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:124) ~[elasticsearch-7.2.0.jar:7.2.0]",
  1045. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) ~[?:?]",
  1046. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  1047. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:308) ~[?:?]",
  1048. "at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.2.0.jar:7.2.0]",
  1049. "at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.2.0.jar:7.2.0]",
  1050. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.2.0.jar:7.2.0]",
  1051. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  1052. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]",
  1053. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]",
  1054. "at java.lang.Thread.run(Thread.java:835) [?:?]"] }
  1055. {"type": "server", "timestamp": "2019-07-01T18:23:57,443+0000", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "last failed join attempt was 7.4s ago, failed to join {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=84, lastAcceptedTerm=28, lastAcceptedVersion=4, sourceNode={elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}}]}" ,
  1056. "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-0][172.17.0.2:9300][internal:cluster/coordination/join]",
  1057. "Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming last accepted term 28 of join higher than current last accepted term 0",
  1058. "at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:230) ~[elasticsearch-7.2.0.jar:7.2.0]",
  1059. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:942) ~[elasticsearch-7.2.0.jar:7.2.0]",
  1060. "at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]",
  1061. "at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:511) ~[elasticsearch-7.2.0.jar:7.2.0]",
  1062. "at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:478) ~[elasticsearch-7.2.0.jar:7.2.0]",
  1063. "at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:124) ~[elasticsearch-7.2.0.jar:7.2.0]",
  1064. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) ~[?:?]",
  1065. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]",
  1066. "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:308) ~[?:?]",
  1067. "at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.2.0.jar:7.2.0]",
  1068. "at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.2.0.jar:7.2.0]",
  1069. "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.2.0.jar:7.2.0]",
  1070. "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]",
  1071. "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
  1072. "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
  1073. "at java.lang.Thread.run(Thread.java:835) [?:?]"] }
  1074. {"type": "server", "timestamp": "2019-07-01T18:23:57,445+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "master not discovered or elected yet, an election requires 2 nodes with ids [NtLoCKHvSVWEO5hUT6jT2Q, lwLc0G2zQEW3-bZDvnsKmg], have discovered [{elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}] which is a quorum; discovery will continue using [172.31.33.10:9300, 172.31.11.195:9300] from hosts providers and [{elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 84, last-accepted version 4 in term 28"  }
  1075. {"type": "server", "timestamp": "2019-07-01T18:24:00,128+0000", "level": "WARN", "component": "o.e.t.TransportService", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "Received response for a request that has timed out, sent [75835ms] ago, timed out [15807ms] ago, action [internal:cluster/coordination/join], node [{elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}], id [613]"  }
  1076. {"type": "server", "timestamp": "2019-07-01T18:24:07,446+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "master not discovered or elected yet, an election requires 2 nodes with ids [NtLoCKHvSVWEO5hUT6jT2Q, lwLc0G2zQEW3-bZDvnsKmg], have discovered [{elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}] which is a quorum; discovery will continue using [172.31.33.10:9300, 172.31.11.195:9300] from hosts providers and [{elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 86, last-accepted version 4 in term 28"  }
  1077. {"type": "server", "timestamp": "2019-07-01T18:24:17,448+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "master not discovered or elected yet, an election requires 2 nodes with ids [NtLoCKHvSVWEO5hUT6jT2Q, lwLc0G2zQEW3-bZDvnsKmg], have discovered [{elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}] which is a quorum; discovery will continue using [172.31.33.10:9300, 172.31.11.195:9300] from hosts providers and [{elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 86, last-accepted version 4 in term 28"  }
  1078. {"type": "server", "timestamp": "2019-07-01T18:24:27,449+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "clevyr-elk-cluster", "node.name": "elasticsearch-master-2",  "message": "master not discovered or elected yet, an election requires 2 nodes with ids [NtLoCKHvSVWEO5hUT6jT2Q, lwLc0G2zQEW3-bZDvnsKmg], have discovered [{elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}] which is a quorum; discovery will continue using [172.31.33.10:9300, 172.31.11.195:9300] from hosts providers and [{elasticsearch-master-0}{lwLc0G2zQEW3-bZDvnsKmg}{2DxTor0QSJ2gHFisJNBqzQ}{172.31.33.10}{172.31.33.10:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-1}{NtLoCKHvSVWEO5hUT6jT2Q}{PtMglaGTQKW2Svl4ECwthQ}{172.31.11.195}{172.31.11.195:9300}{ml.machine_memory=8061341696, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-2}{scM0pZ9MTnGum8-xyt37mQ}{P9Yb7lCDShCtQJLOJaBgag}{172.31.29.199}{172.31.29.199:9300}{ml.machine_memory=8061341696, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 86, last-accepted version 4 in term 28"  }
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!