API tools faq
paste
Advertisement
Guest User

Untitled

a guest
May 23rd, 2019
241
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 420.27 KB | None | 0 0
raw download clone embed print report
  1. [2019-05-23T16:45:09,266][WARN ][o.e.c.c.ClusterFormationFailureHelper] [awselsdevlap01.est1933.com-esnode01] master not discovered or elected yet, an election requires at least 2 nodes with ids from [UWgnBPsHQ1aW4xXEZVKyJQ, T_8EuKpaTqWg2oP3TAnAaA, YZ6m2ioDQWqi1cNnOteB6w], have discovered [{awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}] which is a quorum; discovery will continue using [10.173.148.65:9300, 10.173.148.73:9300, 10.173.148.58:9300, 10.173.148.50:9300, 10.173.148.67:9300] from hosts providers and [{awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{WHxGPT4gTWGe6rHamIfbUA}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 22, last-accepted version 2867 in term 21
  2. [2019-05-23T16:45:14,821][INFO ][o.e.n.Node ] [awselsdevlap01.est1933.com-esnode01] stopping ...
  3. [2019-05-23T16:45:14,832][INFO ][o.e.x.w.WatcherService ] [awselsdevlap01.est1933.com-esnode01] stopping watch service, reason [shutdown initiated]
  4. [2019-05-23T16:45:15,067][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [awselsdevlap01.est1933.com-esnode01] [controller/8665] [Main.cc@148] Ml controller exiting
  5. [2019-05-23T16:45:15,068][INFO ][o.e.x.m.p.NativeController] [awselsdevlap01.est1933.com-esnode01] Native controller process has stopped - no new native processes can be started
  6. [2019-05-23T16:45:15,079][INFO ][o.e.n.Node ] [awselsdevlap01.est1933.com-esnode01] stopped
  7. [2019-05-23T16:45:15,079][INFO ][o.e.n.Node ] [awselsdevlap01.est1933.com-esnode01] closing ...
  8. [2019-05-23T16:45:15,090][INFO ][o.e.n.Node ] [awselsdevlap01.est1933.com-esnode01] closed
  9. [2019-05-23T16:45:19,285][WARN ][o.e.c.l.LogConfigurator ] [awselsdevlap01.est1933.com-esnode01] Some logging configurations have %marker but don't have %node_name. We will automatically add %node_name to the pattern to ease the migration for users who customize log4j2.properties but will stop this behavior in 7.0. You should manually replace `%node_name` with `[%node_name]%marker ` in these locations:
  10. /etc/elasticsearch/esnode01/log4j2.properties
  11. [2019-05-23T16:45:19,680][INFO ][o.e.e.NodeEnvironment ] [awselsdevlap01.est1933.com-esnode01] using [5] data paths, mounts [[/ (rootfs)]], net usable_space [194.2gb], net total_space [199.9gb], types [rootfs]
  12. [2019-05-23T16:45:19,681][INFO ][o.e.e.NodeEnvironment ] [awselsdevlap01.est1933.com-esnode01] heap size [3.9gb], compressed ordinary object pointers [true]
  13. [2019-05-23T16:45:19,755][INFO ][o.e.n.Node ] [awselsdevlap01.est1933.com-esnode01] node name [awselsdevlap01.est1933.com-esnode01], node ID [YZ6m2ioDQWqi1cNnOteB6w], cluster name [es_gallo_dev]
  14. [2019-05-23T16:45:19,755][INFO ][o.e.n.Node ] [awselsdevlap01.est1933.com-esnode01] version[7.1.0], pid[21203], build[default/rpm/606a173/2019-05-16T00:43:15.323135Z], OS[Linux/3.10.0-957.10.1.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/12.0.1/12.0.1+12]
  15. [2019-05-23T16:45:19,755][INFO ][o.e.n.Node ] [awselsdevlap01.est1933.com-esnode01] JVM home [/usr/share/elasticsearch/jdk]
  16. [2019-05-23T16:45:19,755][INFO ][o.e.n.Node ] [awselsdevlap01.est1933.com-esnode01] JVM arguments [-Xms4g, -Xmx4g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Dio.netty.allocator.type=pooled, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch/esnode01, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
  17. [2019-05-23T16:45:21,453][INFO ][o.e.p.PluginsService ] [awselsdevlap01.est1933.com-esnode01] loaded module [aggs-matrix-stats]
  18. [2019-05-23T16:45:21,453][INFO ][o.e.p.PluginsService ] [awselsdevlap01.est1933.com-esnode01] loaded module [analysis-common]
  19. [2019-05-23T16:45:21,453][INFO ][o.e.p.PluginsService ] [awselsdevlap01.est1933.com-esnode01] loaded module [ingest-common]
  20. [2019-05-23T16:45:21,453][INFO ][o.e.p.PluginsService ] [awselsdevlap01.est1933.com-esnode01] loaded module [ingest-geoip]
  21. [2019-05-23T16:45:21,453][INFO ][o.e.p.PluginsService ] [awselsdevlap01.est1933.com-esnode01] loaded module [ingest-user-agent]
  22. [2019-05-23T16:45:21,453][INFO ][o.e.p.PluginsService ] [awselsdevlap01.est1933.com-esnode01] loaded module [lang-expression]
  23. [2019-05-23T16:45:21,453][INFO ][o.e.p.PluginsService ] [awselsdevlap01.est1933.com-esnode01] loaded module [lang-mustache]
  24. [2019-05-23T16:45:21,453][INFO ][o.e.p.PluginsService ] [awselsdevlap01.est1933.com-esnode01] loaded module [lang-painless]
  25. [2019-05-23T16:45:21,454][INFO ][o.e.p.PluginsService ] [awselsdevlap01.est1933.com-esnode01] loaded module [mapper-extras]
  26. [2019-05-23T16:45:21,454][INFO ][o.e.p.PluginsService ] [awselsdevlap01.est1933.com-esnode01] loaded module [parent-join]
  27. [2019-05-23T16:45:21,454][INFO ][o.e.p.PluginsService ] [awselsdevlap01.est1933.com-esnode01] loaded module [percolator]
  28. [2019-05-23T16:45:21,454][INFO ][o.e.p.PluginsService ] [awselsdevlap01.est1933.com-esnode01] loaded module [rank-eval]
  29. [2019-05-23T16:45:21,454][INFO ][o.e.p.PluginsService ] [awselsdevlap01.est1933.com-esnode01] loaded module [reindex]
  30. [2019-05-23T16:45:21,454][INFO ][o.e.p.PluginsService ] [awselsdevlap01.est1933.com-esnode01] loaded module [repository-url]
  31. [2019-05-23T16:45:21,454][INFO ][o.e.p.PluginsService ] [awselsdevlap01.est1933.com-esnode01] loaded module [transport-netty4]
  32. [2019-05-23T16:45:21,454][INFO ][o.e.p.PluginsService ] [awselsdevlap01.est1933.com-esnode01] loaded module [x-pack-ccr]
  33. [2019-05-23T16:45:21,455][INFO ][o.e.p.PluginsService ] [awselsdevlap01.est1933.com-esnode01] loaded module [x-pack-core]
  34. [2019-05-23T16:45:21,455][INFO ][o.e.p.PluginsService ] [awselsdevlap01.est1933.com-esnode01] loaded module [x-pack-deprecation]
  35. [2019-05-23T16:45:21,455][INFO ][o.e.p.PluginsService ] [awselsdevlap01.est1933.com-esnode01] loaded module [x-pack-graph]
  36. [2019-05-23T16:45:21,455][INFO ][o.e.p.PluginsService ] [awselsdevlap01.est1933.com-esnode01] loaded module [x-pack-ilm]
  37. [2019-05-23T16:45:21,455][INFO ][o.e.p.PluginsService ] [awselsdevlap01.est1933.com-esnode01] loaded module [x-pack-logstash]
  38. [2019-05-23T16:45:21,455][INFO ][o.e.p.PluginsService ] [awselsdevlap01.est1933.com-esnode01] loaded module [x-pack-ml]
  39. [2019-05-23T16:45:21,455][INFO ][o.e.p.PluginsService ] [awselsdevlap01.est1933.com-esnode01] loaded module [x-pack-monitoring]
  40. [2019-05-23T16:45:21,455][INFO ][o.e.p.PluginsService ] [awselsdevlap01.est1933.com-esnode01] loaded module [x-pack-rollup]
  41. [2019-05-23T16:45:21,456][INFO ][o.e.p.PluginsService ] [awselsdevlap01.est1933.com-esnode01] loaded module [x-pack-security]
  42. [2019-05-23T16:45:21,456][INFO ][o.e.p.PluginsService ] [awselsdevlap01.est1933.com-esnode01] loaded module [x-pack-sql]
  43. [2019-05-23T16:45:21,456][INFO ][o.e.p.PluginsService ] [awselsdevlap01.est1933.com-esnode01] loaded module [x-pack-watcher]
  44. [2019-05-23T16:45:21,456][INFO ][o.e.p.PluginsService ] [awselsdevlap01.est1933.com-esnode01] no plugins loaded
  45. [2019-05-23T16:45:24,726][DEBUG][o.e.d.z.ElectMasterService] [awselsdevlap01.est1933.com-esnode01] using minimum_master_nodes [-1]
  46. [2019-05-23T16:45:26,423][INFO ][o.e.x.s.a.l.LdapUserSearchSessionFactory] [awselsdevlap01.est1933.com-esnode01] Realm [ldap1] is in user-search mode - base_dn=[ou=people,o=ejgallo.com], search filter=[(uid={0})]
  47. [2019-05-23T16:45:26,426][WARN ][o.e.x.s.a.s.DnRoleMapper ] [awselsdevlap01.est1933.com-esnode01] Role mapping file [/etc/elasticsearch/esnode01/role_mapping.yml] for realm [ldap1] does not exist. Role mapping will be skipped.
  48. [2019-05-23T16:45:27,174][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [awselsdevlap01.est1933.com-esnode01] [controller/21283] [Main.cc@109] controller (64 bit): Version 7.1.0 (Build a8ee6de8087169) Copyright (c) 2019 Elasticsearch BV
  49. [2019-05-23T16:45:27,745][DEBUG][o.e.a.ActionModule ] [awselsdevlap01.est1933.com-esnode01] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
  50. [2019-05-23T16:45:28,273][DEBUG][o.e.d.SettingsBasedSeedHostsProvider] [awselsdevlap01.est1933.com-esnode01] using initial hosts [awselsdevlap01.est1933.com, awselsdevlap02.est1933.com, awselsdevlap03.est1933.com, awselsdevlap04.est1933.com, awselsdevlap05.est1933.com, awselsdevlap06.est1933.com]
  51. [2019-05-23T16:45:28,305][INFO ][o.e.d.DiscoveryModule ] [awselsdevlap01.est1933.com-esnode01] using discovery type [zen] and seed hosts providers [settings]
  52. [2019-05-23T16:45:29,350][INFO ][o.e.n.Node ] [awselsdevlap01.est1933.com-esnode01] initialized
  53. [2019-05-23T16:45:29,351][INFO ][o.e.n.Node ] [awselsdevlap01.est1933.com-esnode01] starting ...
  54. [2019-05-23T16:45:29,630][INFO ][o.e.t.TransportService ] [awselsdevlap01.est1933.com-esnode01] publish_address {10.173.148.143:9300}, bound_addresses {10.173.148.143:9300}
  55. [2019-05-23T16:45:29,642][INFO ][o.e.b.BootstrapChecks ] [awselsdevlap01.est1933.com-esnode01] bound or publishing to a non-loopback address, enforcing bootstrap checks
  56. [2019-05-23T16:45:29,668][DEBUG][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] using max_concurrent_resolvers [10], resolver timeout [5s]
  57. [2019-05-23T16:45:29,668][INFO ][o.e.c.c.Coordinator ] [awselsdevlap01.est1933.com-esnode01] cluster UUID [kVxvHoPPSRaadVGgF0-0Ag]
  58. [2019-05-23T16:45:29,672][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] activating with nodes:
  59. {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, xpack.installed=true, ml.max_open_jobs=20}, local
  60.  
  61. [2019-05-23T16:45:29,673][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing master nodes from cluster state: nodes:
  62. {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, xpack.installed=true, ml.max_open_jobs=20}, local
  63.  
  64. [2019-05-23T16:45:29,673][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  65. [2019-05-23T16:45:29,736][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap01.est1933.com] to [10.173.148.143:9300]
  66. [2019-05-23T16:45:29,736][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap02.est1933.com] to [10.173.148.65:9300]
  67. [2019-05-23T16:45:29,736][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap03.est1933.com] to [10.173.148.73:9300]
  68. [2019-05-23T16:45:29,737][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap04.est1933.com] to [10.173.148.58:9300]
  69. [2019-05-23T16:45:29,737][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap05.est1933.com] to [10.173.148.50:9300]
  70. [2019-05-23T16:45:29,737][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap06.est1933.com] to [10.173.148.67:9300]
  71. [2019-05-23T16:45:29,737][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing resolved transport addresses [10.173.148.65:9300, 10.173.148.73:9300, 10.173.148.58:9300, 10.173.148.50:9300, 10.173.148.67:9300]
  72. [2019-05-23T16:45:29,738][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  73. [2019-05-23T16:45:29,739][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  74. [2019-05-23T16:45:29,740][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  75. [2019-05-23T16:45:29,740][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  76. [2019-05-23T16:45:29,741][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  77. [2019-05-23T16:45:29,741][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.65:9300]] opening probe connection
  78. [2019-05-23T16:45:29,741][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.73:9300]] opening probe connection
  79. [2019-05-23T16:45:29,741][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.58:9300]] opening probe connection
  80. [2019-05-23T16:45:29,743][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.50:9300]] opening probe connection
  81. [2019-05-23T16:45:29,745][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.67:9300]] opening probe connection
  82. [2019-05-23T16:45:29,789][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  83. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.58:9300] connect_exception
  84. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  85. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  86. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  87. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  88. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  89. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  90. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  91. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  92. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  93. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  94. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  95. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  96. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  97. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  98. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  99. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  100. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  101. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  102. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  103. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  104. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  105. at java.lang.Thread.run(Thread.java:835) [?:?]
  106. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap04.est1933.com/10.173.148.58:9300
  107. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  108. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  109. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  110. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  111. ... 6 more
  112. Caused by: java.net.ConnectException: Connection refused
  113. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  114. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  115. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  116. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  117. ... 6 more
  118. [2019-05-23T16:45:29,789][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  119. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.67:9300] connect_exception
  120. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  121. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  122. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  123. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  124. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  125. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  126. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  127. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  128. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  129. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  130. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  131. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  132. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  133. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  134. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  135. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  136. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  137. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  138. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  139. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  140. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  141. at java.lang.Thread.run(Thread.java:835) [?:?]
  142. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap06.est1933.com/10.173.148.67:9300
  143. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  144. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  145. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  146. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  147. ... 6 more
  148. Caused by: java.net.ConnectException: Connection refused
  149. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  150. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  151. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  152. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  153. ... 6 more
  154. [2019-05-23T16:45:29,794][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  155. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.50:9300] connect_exception
  156. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  157. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  158. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  159. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  160. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  161. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  162. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  163. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  164. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  165. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  166. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  167. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  168. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  169. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  170. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  171. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  172. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  173. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  174. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  175. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  176. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  177. at java.lang.Thread.run(Thread.java:835) [?:?]
  178. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap05.est1933.com/10.173.148.50:9300
  179. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  180. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  181. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  182. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  183. ... 6 more
  184. Caused by: java.net.ConnectException: Connection refused
  185. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  186. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  187. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  188. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  189. ... 6 more
  190. [2019-05-23T16:45:29,844][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.65:9300]] opened probe connection
  191. [2019-05-23T16:45:29,845][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.73:9300]] opened probe connection
  192. [2019-05-23T16:45:29,875][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.65:9300]] handshake successful: {awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}
  193. [2019-05-23T16:45:29,910][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.73:9300]] handshake successful: {awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}
  194. [2019-05-23T16:45:29,929][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  195. [2019-05-23T16:45:29,944][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.73:9300]] full connection successful: {awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}
  196. [2019-05-23T16:45:29,944][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  197. [2019-05-23T16:45:29,950][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.65:9300]] full connection successful: {awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}
  198. [2019-05-23T16:45:29,950][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  199. [2019-05-23T16:45:30,003][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  200. [2019-05-23T16:45:30,005][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  201. [2019-05-23T16:45:30,005][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  202. [2019-05-23T16:45:30,019][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  203. [2019-05-23T16:45:30,601][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  204. [2019-05-23T16:45:30,674][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  205. [2019-05-23T16:45:30,675][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  206. [2019-05-23T16:45:30,676][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing master nodes from cluster state: nodes:
  207. {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, xpack.installed=true, ml.max_open_jobs=20}, local
  208.  
  209. [2019-05-23T16:45:30,676][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  210. [2019-05-23T16:45:30,677][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  211. [2019-05-23T16:45:30,677][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  212. [2019-05-23T16:45:30,677][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  213. [2019-05-23T16:45:30,677][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap01.est1933.com] to [10.173.148.143:9300]
  214. [2019-05-23T16:45:30,677][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  215. [2019-05-23T16:45:30,677][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap02.est1933.com] to [10.173.148.65:9300]
  216. [2019-05-23T16:45:30,678][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap03.est1933.com] to [10.173.148.73:9300]
  217. [2019-05-23T16:45:30,678][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap04.est1933.com] to [10.173.148.58:9300]
  218. [2019-05-23T16:45:30,678][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap05.est1933.com] to [10.173.148.50:9300]
  219. [2019-05-23T16:45:30,678][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap06.est1933.com] to [10.173.148.67:9300]
  220. [2019-05-23T16:45:30,678][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing resolved transport addresses [10.173.148.65:9300, 10.173.148.73:9300, 10.173.148.58:9300, 10.173.148.50:9300, 10.173.148.67:9300]
  221. [2019-05-23T16:45:30,678][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  222. [2019-05-23T16:45:30,679][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  223. [2019-05-23T16:45:30,679][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.58:9300]] opening probe connection
  224. [2019-05-23T16:45:30,681][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.50:9300]] opening probe connection
  225. [2019-05-23T16:45:30,679][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  226. [2019-05-23T16:45:30,682][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.67:9300]] opening probe connection
  227. [2019-05-23T16:45:30,691][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  228. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.58:9300] connect_exception
  229. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  230. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  231. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  232. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  233. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  234. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  235. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  236. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  237. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  238. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  239. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  240. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  241. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  242. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  243. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  244. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  245. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  246. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  247. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  248. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  249. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  250. at java.lang.Thread.run(Thread.java:835) [?:?]
  251. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap04.est1933.com/10.173.148.58:9300
  252. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  253. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  254. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  255. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  256. ... 6 more
  257. Caused by: java.net.ConnectException: Connection refused
  258. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  259. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  260. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  261. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  262. ... 6 more
  263. [2019-05-23T16:45:30,694][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  264. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.67:9300] connect_exception
  265. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  266. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  267. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  268. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  269. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  270. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  271. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  272. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  273. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  274. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  275. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  276. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  277. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  278. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  279. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  280. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  281. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  282. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  283. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  284. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  285. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  286. at java.lang.Thread.run(Thread.java:835) [?:?]
  287. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap06.est1933.com/10.173.148.67:9300
  288. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  289. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  290. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  291. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  292. ... 6 more
  293. Caused by: java.net.ConnectException: Connection refused
  294. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  295. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  296. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  297. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  298. ... 6 more
  299. [2019-05-23T16:45:30,694][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  300. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.50:9300] connect_exception
  301. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  302. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  303. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  304. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  305. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  306. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  307. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  308. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  309. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  310. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  311. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  312. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  313. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  314. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  315. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  316. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  317. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  318. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  319. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  320. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  321. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  322. at java.lang.Thread.run(Thread.java:835) [?:?]
  323. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap05.est1933.com/10.173.148.50:9300
  324. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  325. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  326. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  327. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  328. ... 6 more
  329. Caused by: java.net.ConnectException: Connection refused
  330. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  331. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  332. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  333. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  334. ... 6 more
  335. [2019-05-23T16:45:30,833][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  336. [2019-05-23T16:45:31,599][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  337. [2019-05-23T16:45:31,676][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  338. [2019-05-23T16:45:31,677][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  339. [2019-05-23T16:45:31,677][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing master nodes from cluster state: nodes:
  340. {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, xpack.installed=true, ml.max_open_jobs=20}, local
  341.  
  342. [2019-05-23T16:45:31,677][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  343. [2019-05-23T16:45:31,679][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  344. [2019-05-23T16:45:31,679][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  345. [2019-05-23T16:45:31,679][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  346. [2019-05-23T16:45:31,679][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap01.est1933.com] to [10.173.148.143:9300]
  347. [2019-05-23T16:45:31,679][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap02.est1933.com] to [10.173.148.65:9300]
  348. [2019-05-23T16:45:31,679][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  349. [2019-05-23T16:45:31,679][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap03.est1933.com] to [10.173.148.73:9300]
  350. [2019-05-23T16:45:31,679][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap04.est1933.com] to [10.173.148.58:9300]
  351. [2019-05-23T16:45:31,680][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap05.est1933.com] to [10.173.148.50:9300]
  352. [2019-05-23T16:45:31,680][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap06.est1933.com] to [10.173.148.67:9300]
  353. [2019-05-23T16:45:31,680][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing resolved transport addresses [10.173.148.65:9300, 10.173.148.73:9300, 10.173.148.58:9300, 10.173.148.50:9300, 10.173.148.67:9300]
  354. [2019-05-23T16:45:31,680][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  355. [2019-05-23T16:45:31,680][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  356. [2019-05-23T16:45:31,680][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.58:9300]] opening probe connection
  357. [2019-05-23T16:45:31,681][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  358. [2019-05-23T16:45:31,681][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.50:9300]] opening probe connection
  359. [2019-05-23T16:45:31,681][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.67:9300]] opening probe connection
  360. [2019-05-23T16:45:31,684][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  361. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.50:9300] connect_exception
  362. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  363. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  364. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  365. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  366. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  367. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  368. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  369. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  370. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  371. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  372. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  373. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  374. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  375. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  376. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  377. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  378. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  379. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  380. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  381. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  382. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  383. at java.lang.Thread.run(Thread.java:835) [?:?]
  384. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap05.est1933.com/10.173.148.50:9300
  385. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  386. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  387. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  388. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  389. ... 6 more
  390. Caused by: java.net.ConnectException: Connection refused
  391. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  392. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  393. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  394. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  395. ... 6 more
  396. [2019-05-23T16:45:31,685][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  397. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.58:9300] connect_exception
  398. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  399. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  400. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  401. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  402. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  403. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  404. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  405. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  406. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  407. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  408. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  409. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  410. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  411. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  412. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  413. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  414. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  415. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  416. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  417. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  418. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  419. at java.lang.Thread.run(Thread.java:835) [?:?]
  420. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap04.est1933.com/10.173.148.58:9300
  421. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  422. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  423. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  424. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  425. ... 6 more
  426. Caused by: java.net.ConnectException: Connection refused
  427. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  428. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  429. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  430. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  431. ... 6 more
  432. [2019-05-23T16:45:31,684][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  433. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.67:9300] connect_exception
  434. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  435. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  436. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  437. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  438. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  439. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  440. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  441. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  442. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  443. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  444. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  445. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  446. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  447. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  448. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  449. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  450. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  451. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  452. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  453. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  454. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  455. at java.lang.Thread.run(Thread.java:835) [?:?]
  456. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap06.est1933.com/10.173.148.67:9300
  457. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  458. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  459. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  460. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  461. ... 6 more
  462. Caused by: java.net.ConnectException: Connection refused
  463. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  464. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  465. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  466. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  467. ... 6 more
  468. [2019-05-23T16:45:31,833][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  469. [2019-05-23T16:45:32,599][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  470. [2019-05-23T16:45:32,678][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  471. [2019-05-23T16:45:32,678][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  472. [2019-05-23T16:45:32,679][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing master nodes from cluster state: nodes:
  473. {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, xpack.installed=true, ml.max_open_jobs=20}, local
  474.  
  475. [2019-05-23T16:45:32,679][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  476. [2019-05-23T16:45:32,680][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  477. [2019-05-23T16:45:32,680][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  478. [2019-05-23T16:45:32,680][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  479. [2019-05-23T16:45:32,680][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap01.est1933.com] to [10.173.148.143:9300]
  480. [2019-05-23T16:45:32,680][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  481. [2019-05-23T16:45:32,681][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap02.est1933.com] to [10.173.148.65:9300]
  482. [2019-05-23T16:45:32,681][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap03.est1933.com] to [10.173.148.73:9300]
  483. [2019-05-23T16:45:32,681][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap04.est1933.com] to [10.173.148.58:9300]
  484. [2019-05-23T16:45:32,681][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap05.est1933.com] to [10.173.148.50:9300]
  485. [2019-05-23T16:45:32,681][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap06.est1933.com] to [10.173.148.67:9300]
  486. [2019-05-23T16:45:32,681][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing resolved transport addresses [10.173.148.65:9300, 10.173.148.73:9300, 10.173.148.58:9300, 10.173.148.50:9300, 10.173.148.67:9300]
  487. [2019-05-23T16:45:32,681][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  488. [2019-05-23T16:45:32,682][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  489. [2019-05-23T16:45:32,682][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  490. [2019-05-23T16:45:32,682][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.50:9300]] opening probe connection
  491. [2019-05-23T16:45:32,682][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.67:9300]] opening probe connection
  492. [2019-05-23T16:45:32,682][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.58:9300]] opening probe connection
  493. [2019-05-23T16:45:32,686][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  494. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.58:9300] connect_exception
  495. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  496. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  497. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  498. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  499. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  500. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  501. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  502. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  503. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  504. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  505. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  506. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  507. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  508. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  509. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  510. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  511. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  512. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  513. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  514. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  515. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  516. at java.lang.Thread.run(Thread.java:835) [?:?]
  517. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap04.est1933.com/10.173.148.58:9300
  518. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  519. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  520. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  521. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  522. ... 6 more
  523. Caused by: java.net.ConnectException: Connection refused
  524. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  525. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  526. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  527. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  528. ... 6 more
  529. [2019-05-23T16:45:32,687][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  530. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.67:9300] connect_exception
  531. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  532. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  533. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  534. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  535. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  536. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  537. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  538. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  539. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  540. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  541. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  542. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  543. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  544. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  545. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  546. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  547. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  548. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  549. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  550. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  551. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  552. at java.lang.Thread.run(Thread.java:835) [?:?]
  553. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap06.est1933.com/10.173.148.67:9300
  554. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  555. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  556. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  557. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  558. ... 6 more
  559. Caused by: java.net.ConnectException: Connection refused
  560. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  561. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  562. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  563. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  564. ... 6 more
  565. [2019-05-23T16:45:32,687][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  566. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.50:9300] connect_exception
  567. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  568. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  569. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  570. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  571. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  572. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  573. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  574. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  575. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  576. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  577. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  578. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  579. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  580. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  581. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  582. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  583. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  584. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  585. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  586. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  587. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  588. at java.lang.Thread.run(Thread.java:835) [?:?]
  589. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap05.est1933.com/10.173.148.50:9300
  590. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  591. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  592. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  593. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  594. ... 6 more
  595. Caused by: java.net.ConnectException: Connection refused
  596. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  597. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  598. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  599. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  600. ... 6 more
  601. [2019-05-23T16:45:32,834][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  602. [2019-05-23T16:45:33,599][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  603. [2019-05-23T16:45:33,679][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  604. [2019-05-23T16:45:33,680][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  605. [2019-05-23T16:45:33,681][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing master nodes from cluster state: nodes:
  606. {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, xpack.installed=true, ml.max_open_jobs=20}, local
  607.  
  608. [2019-05-23T16:45:33,681][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  609. [2019-05-23T16:45:33,682][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap01.est1933.com] to [10.173.148.143:9300]
  610. [2019-05-23T16:45:33,682][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap02.est1933.com] to [10.173.148.65:9300]
  611. [2019-05-23T16:45:33,682][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap03.est1933.com] to [10.173.148.73:9300]
  612. [2019-05-23T16:45:33,682][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap04.est1933.com] to [10.173.148.58:9300]
  613. [2019-05-23T16:45:33,682][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap05.est1933.com] to [10.173.148.50:9300]
  614. [2019-05-23T16:45:33,683][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  615. [2019-05-23T16:45:33,683][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  616. [2019-05-23T16:45:33,683][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap06.est1933.com] to [10.173.148.67:9300]
  617. [2019-05-23T16:45:33,683][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  618. [2019-05-23T16:45:33,683][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing resolved transport addresses [10.173.148.65:9300, 10.173.148.73:9300, 10.173.148.58:9300, 10.173.148.50:9300, 10.173.148.67:9300]
  619. [2019-05-23T16:45:33,683][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  620. [2019-05-23T16:45:33,683][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  621. [2019-05-23T16:45:33,683][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.58:9300]] opening probe connection
  622. [2019-05-23T16:45:33,683][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  623. [2019-05-23T16:45:33,684][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.67:9300]] opening probe connection
  624. [2019-05-23T16:45:33,684][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.50:9300]] opening probe connection
  625. [2019-05-23T16:45:33,685][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  626. [2019-05-23T16:45:33,687][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  627. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.58:9300] connect_exception
  628. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  629. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  630. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  631. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  632. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  633. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  634. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  635. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  636. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  637. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  638. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  639. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  640. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  641. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  642. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  643. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  644. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  645. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  646. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  647. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  648. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  649. at java.lang.Thread.run(Thread.java:835) [?:?]
  650. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap04.est1933.com/10.173.148.58:9300
  651. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  652. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  653. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  654. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  655. ... 6 more
  656. Caused by: java.net.ConnectException: Connection refused
  657. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  658. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  659. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  660. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  661. ... 6 more
  662. [2019-05-23T16:45:33,687][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  663. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.67:9300] connect_exception
  664. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  665. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  666. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  667. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  668. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  669. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  670. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  671. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  672. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  673. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  674. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  675. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  676. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  677. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  678. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  679. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  680. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  681. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  682. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  683. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  684. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  685. at java.lang.Thread.run(Thread.java:835) [?:?]
  686. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap06.est1933.com/10.173.148.67:9300
  687. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  688. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  689. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  690. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  691. ... 6 more
  692. Caused by: java.net.ConnectException: Connection refused
  693. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  694. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  695. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  696. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  697. ... 6 more
  698. [2019-05-23T16:45:33,690][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  699. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.50:9300] connect_exception
  700. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  701. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  702. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  703. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  704. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  705. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  706. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  707. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  708. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  709. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  710. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  711. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  712. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  713. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  714. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  715. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  716. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  717. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  718. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  719. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  720. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  721. at java.lang.Thread.run(Thread.java:835) [?:?]
  722. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap05.est1933.com/10.173.148.50:9300
  723. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  724. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  725. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  726. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  727. ... 6 more
  728. Caused by: java.net.ConnectException: Connection refused
  729. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  730. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  731. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  732. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  733. ... 6 more
  734. [2019-05-23T16:45:33,834][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  735. [2019-05-23T16:45:34,600][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  736. [2019-05-23T16:45:34,681][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  737. [2019-05-23T16:45:34,682][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  738. [2019-05-23T16:45:34,683][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing master nodes from cluster state: nodes:
  739. {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, xpack.installed=true, ml.max_open_jobs=20}, local
  740.  
  741. [2019-05-23T16:45:34,683][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  742. [2019-05-23T16:45:34,683][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap01.est1933.com] to [10.173.148.143:9300]
  743. [2019-05-23T16:45:34,684][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  744. [2019-05-23T16:45:34,684][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap02.est1933.com] to [10.173.148.65:9300]
  745. [2019-05-23T16:45:34,684][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  746. [2019-05-23T16:45:34,684][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap03.est1933.com] to [10.173.148.73:9300]
  747. [2019-05-23T16:45:34,684][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap04.est1933.com] to [10.173.148.58:9300]
  748. [2019-05-23T16:45:34,684][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  749. [2019-05-23T16:45:34,684][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap05.est1933.com] to [10.173.148.50:9300]
  750. [2019-05-23T16:45:34,684][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  751. [2019-05-23T16:45:34,684][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap06.est1933.com] to [10.173.148.67:9300]
  752. [2019-05-23T16:45:34,684][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing resolved transport addresses [10.173.148.65:9300, 10.173.148.73:9300, 10.173.148.58:9300, 10.173.148.50:9300, 10.173.148.67:9300]
  753. [2019-05-23T16:45:34,684][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  754. [2019-05-23T16:45:34,685][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  755. [2019-05-23T16:45:34,685][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.58:9300]] opening probe connection
  756. [2019-05-23T16:45:34,685][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  757. [2019-05-23T16:45:34,685][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.67:9300]] opening probe connection
  758. [2019-05-23T16:45:34,686][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.50:9300]] opening probe connection
  759. [2019-05-23T16:45:34,688][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  760. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.67:9300] connect_exception
  761. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  762. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  763. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  764. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  765. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  766. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  767. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  768. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  769. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  770. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  771. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  772. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  773. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  774. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  775. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  776. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  777. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  778. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  779. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  780. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  781. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  782. at java.lang.Thread.run(Thread.java:835) [?:?]
  783. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap06.est1933.com/10.173.148.67:9300
  784. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  785. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  786. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  787. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  788. ... 6 more
  789. Caused by: java.net.ConnectException: Connection refused
  790. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  791. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  792. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  793. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  794. ... 6 more
  795. [2019-05-23T16:45:34,690][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  796. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.58:9300] connect_exception
  797. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  798. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  799. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  800. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  801. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  802. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  803. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  804. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  805. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  806. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  807. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  808. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  809. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  810. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  811. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  812. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  813. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  814. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  815. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  816. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  817. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  818. at java.lang.Thread.run(Thread.java:835) [?:?]
  819. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap04.est1933.com/10.173.148.58:9300
  820. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  821. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  822. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  823. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  824. ... 6 more
  825. Caused by: java.net.ConnectException: Connection refused
  826. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  827. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  828. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  829. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  830. ... 6 more
  831. [2019-05-23T16:45:34,692][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  832. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.50:9300] connect_exception
  833. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  834. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  835. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  836. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  837. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  838. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  839. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  840. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  841. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  842. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  843. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  844. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  845. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  846. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  847. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  848. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  849. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  850. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  851. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  852. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  853. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  854. at java.lang.Thread.run(Thread.java:835) [?:?]
  855. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap05.est1933.com/10.173.148.50:9300
  856. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  857. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  858. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  859. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  860. ... 6 more
  861. Caused by: java.net.ConnectException: Connection refused
  862. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  863. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  864. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  865. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  866. ... 6 more
  867. [2019-05-23T16:45:34,834][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  868. [2019-05-23T16:45:35,600][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  869. [2019-05-23T16:45:35,683][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  870. [2019-05-23T16:45:35,684][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  871. [2019-05-23T16:45:35,684][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing master nodes from cluster state: nodes:
  872. {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, xpack.installed=true, ml.max_open_jobs=20}, local
  873.  
  874. [2019-05-23T16:45:35,684][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  875. [2019-05-23T16:45:35,685][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  876. [2019-05-23T16:45:35,685][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap01.est1933.com] to [10.173.148.143:9300]
  877. [2019-05-23T16:45:35,685][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  878. [2019-05-23T16:45:35,685][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap02.est1933.com] to [10.173.148.65:9300]
  879. [2019-05-23T16:45:35,685][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap03.est1933.com] to [10.173.148.73:9300]
  880. [2019-05-23T16:45:35,685][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap04.est1933.com] to [10.173.148.58:9300]
  881. [2019-05-23T16:45:35,686][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap05.est1933.com] to [10.173.148.50:9300]
  882. [2019-05-23T16:45:35,686][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap06.est1933.com] to [10.173.148.67:9300]
  883. [2019-05-23T16:45:35,686][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing resolved transport addresses [10.173.148.65:9300, 10.173.148.73:9300, 10.173.148.58:9300, 10.173.148.50:9300, 10.173.148.67:9300]
  884. [2019-05-23T16:45:35,686][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  885. [2019-05-23T16:45:35,686][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  886. [2019-05-23T16:45:35,686][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  887. [2019-05-23T16:45:35,686][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.58:9300]] opening probe connection
  888. [2019-05-23T16:45:35,686][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  889. [2019-05-23T16:45:35,686][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.50:9300]] opening probe connection
  890. [2019-05-23T16:45:35,687][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.67:9300]] opening probe connection
  891. [2019-05-23T16:45:35,687][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  892. [2019-05-23T16:45:35,691][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  893. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.50:9300] connect_exception
  894. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  895. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  896. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  897. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  898. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  899. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  900. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  901. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  902. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  903. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  904. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  905. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  906. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  907. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  908. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  909. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  910. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  911. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  912. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  913. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  914. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  915. at java.lang.Thread.run(Thread.java:835) [?:?]
  916. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap05.est1933.com/10.173.148.50:9300
  917. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  918. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  919. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  920. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  921. ... 6 more
  922. Caused by: java.net.ConnectException: Connection refused
  923. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  924. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  925. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  926. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  927. ... 6 more
  928. [2019-05-23T16:45:35,693][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  929. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.67:9300] connect_exception
  930. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  931. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  932. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  933. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  934. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  935. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  936. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  937. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  938. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  939. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  940. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  941. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  942. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  943. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  944. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  945. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  946. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  947. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  948. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  949. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  950. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  951. at java.lang.Thread.run(Thread.java:835) [?:?]
  952. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap06.est1933.com/10.173.148.67:9300
  953. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  954. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  955. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  956. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  957. ... 6 more
  958. Caused by: java.net.ConnectException: Connection refused
  959. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  960. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  961. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  962. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  963. ... 6 more
  964. [2019-05-23T16:45:35,691][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  965. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.58:9300] connect_exception
  966. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  967. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  968. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  969. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  970. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  971. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  972. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  973. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  974. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  975. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  976. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  977. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  978. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  979. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  980. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  981. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  982. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  983. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  984. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  985. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  986. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  987. at java.lang.Thread.run(Thread.java:835) [?:?]
  988. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap04.est1933.com/10.173.148.58:9300
  989. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  990. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  991. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  992. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  993. ... 6 more
  994. Caused by: java.net.ConnectException: Connection refused
  995. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  996. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  997. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  998. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  999. ... 6 more
  1000. [2019-05-23T16:45:35,834][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  1001. [2019-05-23T16:45:36,600][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  1002. [2019-05-23T16:45:36,684][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  1003. [2019-05-23T16:45:36,685][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  1004. [2019-05-23T16:45:36,685][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing master nodes from cluster state: nodes:
  1005. {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, xpack.installed=true, ml.max_open_jobs=20}, local
  1006.  
  1007. [2019-05-23T16:45:36,685][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  1008. [2019-05-23T16:45:36,686][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap01.est1933.com] to [10.173.148.143:9300]
  1009. [2019-05-23T16:45:36,686][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap02.est1933.com] to [10.173.148.65:9300]
  1010. [2019-05-23T16:45:36,686][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  1011. [2019-05-23T16:45:36,686][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap03.est1933.com] to [10.173.148.73:9300]
  1012. [2019-05-23T16:45:36,686][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  1013. [2019-05-23T16:45:36,687][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap04.est1933.com] to [10.173.148.58:9300]
  1014. [2019-05-23T16:45:36,687][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap05.est1933.com] to [10.173.148.50:9300]
  1015. [2019-05-23T16:45:36,687][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap06.est1933.com] to [10.173.148.67:9300]
  1016. [2019-05-23T16:45:36,686][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  1017. [2019-05-23T16:45:36,687][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing resolved transport addresses [10.173.148.65:9300, 10.173.148.73:9300, 10.173.148.58:9300, 10.173.148.50:9300, 10.173.148.67:9300]
  1018. [2019-05-23T16:45:36,687][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  1019. [2019-05-23T16:45:36,687][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  1020. [2019-05-23T16:45:36,687][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.58:9300]] opening probe connection
  1021. [2019-05-23T16:45:36,688][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.50:9300]] opening probe connection
  1022. [2019-05-23T16:45:36,687][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  1023. [2019-05-23T16:45:36,689][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.67:9300]] opening probe connection
  1024. [2019-05-23T16:45:36,690][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  1025. [2019-05-23T16:45:36,693][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  1026. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.50:9300] connect_exception
  1027. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  1028. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  1029. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  1030. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  1031. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  1032. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  1033. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  1034. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  1035. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  1036. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  1037. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  1038. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  1039. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  1040. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  1041. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  1042. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  1043. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  1044. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  1045. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  1046. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  1047. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  1048. at java.lang.Thread.run(Thread.java:835) [?:?]
  1049. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap05.est1933.com/10.173.148.50:9300
  1050. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  1051. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  1052. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  1053. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  1054. ... 6 more
  1055. Caused by: java.net.ConnectException: Connection refused
  1056. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  1057. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  1058. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  1059. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  1060. ... 6 more
  1061. [2019-05-23T16:45:36,693][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  1062. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.58:9300] connect_exception
  1063. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  1064. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  1065. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  1066. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  1067. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  1068. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  1069. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  1070. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  1071. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  1072. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  1073. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  1074. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  1075. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  1076. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  1077. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  1078. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  1079. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  1080. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  1081. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  1082. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  1083. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  1084. at java.lang.Thread.run(Thread.java:835) [?:?]
  1085. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap04.est1933.com/10.173.148.58:9300
  1086. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  1087. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  1088. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  1089. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  1090. ... 6 more
  1091. Caused by: java.net.ConnectException: Connection refused
  1092. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  1093. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  1094. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  1095. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  1096. ... 6 more
  1097. [2019-05-23T16:45:36,696][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  1098. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.67:9300] connect_exception
  1099. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  1100. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  1101. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  1102. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  1103. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  1104. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  1105. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  1106. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  1107. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  1108. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  1109. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  1110. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  1111. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  1112. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  1113. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  1114. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  1115. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  1116. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  1117. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  1118. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  1119. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  1120. at java.lang.Thread.run(Thread.java:835) [?:?]
  1121. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap06.est1933.com/10.173.148.67:9300
  1122. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  1123. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  1124. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  1125. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  1126. ... 6 more
  1127. Caused by: java.net.ConnectException: Connection refused
  1128. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  1129. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  1130. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  1131. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  1132. ... 6 more
  1133. [2019-05-23T16:45:36,835][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  1134. [2019-05-23T16:45:37,601][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  1135. [2019-05-23T16:45:37,686][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  1136. [2019-05-23T16:45:37,686][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  1137. [2019-05-23T16:45:37,687][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing master nodes from cluster state: nodes:
  1138. {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, xpack.installed=true, ml.max_open_jobs=20}, local
  1139.  
  1140. [2019-05-23T16:45:37,687][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  1141. [2019-05-23T16:45:37,688][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  1142. [2019-05-23T16:45:37,688][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  1143. [2019-05-23T16:45:37,688][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap01.est1933.com] to [10.173.148.143:9300]
  1144. [2019-05-23T16:45:37,688][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap02.est1933.com] to [10.173.148.65:9300]
  1145. [2019-05-23T16:45:37,688][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap03.est1933.com] to [10.173.148.73:9300]
  1146. [2019-05-23T16:45:37,688][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap04.est1933.com] to [10.173.148.58:9300]
  1147. [2019-05-23T16:45:37,689][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap05.est1933.com] to [10.173.148.50:9300]
  1148. [2019-05-23T16:45:37,689][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap06.est1933.com] to [10.173.148.67:9300]
  1149. [2019-05-23T16:45:37,689][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing resolved transport addresses [10.173.148.65:9300, 10.173.148.73:9300, 10.173.148.58:9300, 10.173.148.50:9300, 10.173.148.67:9300]
  1150. [2019-05-23T16:45:37,689][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  1151. [2019-05-23T16:45:37,689][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  1152. [2019-05-23T16:45:37,689][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.58:9300]] opening probe connection
  1153. [2019-05-23T16:45:37,689][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  1154. [2019-05-23T16:45:37,690][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.67:9300]] opening probe connection
  1155. [2019-05-23T16:45:37,690][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.50:9300]] opening probe connection
  1156. [2019-05-23T16:45:37,690][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  1157. [2019-05-23T16:45:37,690][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  1158. [2019-05-23T16:45:37,692][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  1159. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.50:9300] connect_exception
  1160. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  1161. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  1162. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  1163. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  1164. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  1165. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  1166. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  1167. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  1168. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  1169. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  1170. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  1171. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  1172. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  1173. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  1174. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  1175. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  1176. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  1177. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  1178. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  1179. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  1180. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  1181. at java.lang.Thread.run(Thread.java:835) [?:?]
  1182. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap05.est1933.com/10.173.148.50:9300
  1183. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  1184. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  1185. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  1186. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  1187. ... 6 more
  1188. Caused by: java.net.ConnectException: Connection refused
  1189. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  1190. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  1191. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  1192. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  1193. ... 6 more
  1194. [2019-05-23T16:45:37,692][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  1195. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.67:9300] connect_exception
  1196. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  1197. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  1198. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  1199. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  1200. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  1201. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  1202. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  1203. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  1204. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  1205. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  1206. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  1207. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  1208. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  1209. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  1210. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  1211. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  1212. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  1213. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  1214. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  1215. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  1216. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  1217. at java.lang.Thread.run(Thread.java:835) [?:?]
  1218. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap06.est1933.com/10.173.148.67:9300
  1219. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  1220. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  1221. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  1222. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  1223. ... 6 more
  1224. Caused by: java.net.ConnectException: Connection refused
  1225. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  1226. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  1227. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  1228. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  1229. ... 6 more
  1230. [2019-05-23T16:45:37,696][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  1231. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.58:9300] connect_exception
  1232. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  1233. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  1234. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  1235. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  1236. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  1237. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  1238. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  1239. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  1240. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  1241. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  1242. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  1243. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  1244. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  1245. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  1246. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  1247. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  1248. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  1249. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  1250. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  1251. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  1252. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  1253. at java.lang.Thread.run(Thread.java:835) [?:?]
  1254. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap04.est1933.com/10.173.148.58:9300
  1255. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  1256. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  1257. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  1258. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  1259. ... 6 more
  1260. Caused by: java.net.ConnectException: Connection refused
  1261. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  1262. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  1263. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  1264. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  1265. ... 6 more
  1266. [2019-05-23T16:45:37,835][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  1267. [2019-05-23T16:45:38,601][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  1268. [2019-05-23T16:45:38,687][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  1269. [2019-05-23T16:45:38,687][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  1270. [2019-05-23T16:45:38,688][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing master nodes from cluster state: nodes:
  1271. {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, xpack.installed=true, ml.max_open_jobs=20}, local
  1272.  
  1273. [2019-05-23T16:45:38,688][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  1274. [2019-05-23T16:45:38,689][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  1275. [2019-05-23T16:45:38,689][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  1276. [2019-05-23T16:45:38,689][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap01.est1933.com] to [10.173.148.143:9300]
  1277. [2019-05-23T16:45:38,689][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  1278. [2019-05-23T16:45:38,689][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap02.est1933.com] to [10.173.148.65:9300]
  1279. [2019-05-23T16:45:38,689][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  1280. [2019-05-23T16:45:38,689][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap03.est1933.com] to [10.173.148.73:9300]
  1281. [2019-05-23T16:45:38,689][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap04.est1933.com] to [10.173.148.58:9300]
  1282. [2019-05-23T16:45:38,689][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap05.est1933.com] to [10.173.148.50:9300]
  1283. [2019-05-23T16:45:38,689][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap06.est1933.com] to [10.173.148.67:9300]
  1284. [2019-05-23T16:45:38,689][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing resolved transport addresses [10.173.148.65:9300, 10.173.148.73:9300, 10.173.148.58:9300, 10.173.148.50:9300, 10.173.148.67:9300]
  1285. [2019-05-23T16:45:38,690][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  1286. [2019-05-23T16:45:38,690][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  1287. [2019-05-23T16:45:38,690][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.58:9300]] opening probe connection
  1288. [2019-05-23T16:45:38,690][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  1289. [2019-05-23T16:45:38,690][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.50:9300]] opening probe connection
  1290. [2019-05-23T16:45:38,690][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.67:9300]] opening probe connection
  1291. [2019-05-23T16:45:38,694][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  1292. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.58:9300] connect_exception
  1293. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  1294. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  1295. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  1296. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  1297. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  1298. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  1299. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  1300. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  1301. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  1302. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  1303. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  1304. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  1305. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  1306. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  1307. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  1308. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  1309. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  1310. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  1311. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  1312. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  1313. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  1314. at java.lang.Thread.run(Thread.java:835) [?:?]
  1315. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap04.est1933.com/10.173.148.58:9300
  1316. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  1317. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  1318. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  1319. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  1320. ... 6 more
  1321. Caused by: java.net.ConnectException: Connection refused
  1322. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  1323. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  1324. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  1325. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  1326. ... 6 more
  1327. [2019-05-23T16:45:38,692][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  1328. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.50:9300] connect_exception
  1329. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  1330. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  1331. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  1332. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  1333. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  1334. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  1335. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  1336. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  1337. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  1338. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  1339. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  1340. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  1341. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  1342. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  1343. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  1344. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  1345. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  1346. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  1347. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  1348. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  1349. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  1350. at java.lang.Thread.run(Thread.java:835) [?:?]
  1351. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap05.est1933.com/10.173.148.50:9300
  1352. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  1353. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  1354. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  1355. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  1356. ... 6 more
  1357. Caused by: java.net.ConnectException: Connection refused
  1358. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  1359. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  1360. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  1361. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  1362. ... 6 more
  1363. [2019-05-23T16:45:38,696][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  1364. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.67:9300] connect_exception
  1365. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  1366. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  1367. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  1368. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  1369. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  1370. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  1371. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  1372. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  1373. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  1374. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  1375. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  1376. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  1377. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  1378. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  1379. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  1380. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  1381. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  1382. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  1383. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  1384. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  1385. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  1386. at java.lang.Thread.run(Thread.java:835) [?:?]
  1387. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap06.est1933.com/10.173.148.67:9300
  1388. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  1389. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  1390. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  1391. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  1392. ... 6 more
  1393. Caused by: java.net.ConnectException: Connection refused
  1394. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  1395. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  1396. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  1397. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  1398. ... 6 more
  1399. [2019-05-23T16:45:38,835][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  1400. [2019-05-23T16:45:39,602][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  1401. [2019-05-23T16:45:39,678][WARN ][o.e.c.c.ClusterFormationFailureHelper] [awselsdevlap01.est1933.com-esnode01] master not discovered or elected yet, an election requires at least 2 nodes with ids from [UWgnBPsHQ1aW4xXEZVKyJQ, T_8EuKpaTqWg2oP3TAnAaA, YZ6m2ioDQWqi1cNnOteB6w], have discovered [{awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}] which is a quorum; discovery will continue using [10.173.148.65:9300, 10.173.148.73:9300, 10.173.148.58:9300, 10.173.148.50:9300, 10.173.148.67:9300] from hosts providers and [{awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 22, last-accepted version 2867 in term 21
  1402. [2019-05-23T16:45:39,688][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  1403. [2019-05-23T16:45:39,689][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  1404. [2019-05-23T16:45:39,690][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing master nodes from cluster state: nodes:
  1405. {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, xpack.installed=true, ml.max_open_jobs=20}, local
  1406.  
  1407. [2019-05-23T16:45:39,690][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  1408. [2019-05-23T16:45:39,691][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap01.est1933.com] to [10.173.148.143:9300]
  1409. [2019-05-23T16:45:39,691][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  1410. [2019-05-23T16:45:39,692][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  1411. [2019-05-23T16:45:39,691][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap02.est1933.com] to [10.173.148.65:9300]
  1412. [2019-05-23T16:45:39,692][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  1413. [2019-05-23T16:45:39,692][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  1414. [2019-05-23T16:45:39,692][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap03.est1933.com] to [10.173.148.73:9300]
  1415. [2019-05-23T16:45:39,692][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap04.est1933.com] to [10.173.148.58:9300]
  1416. [2019-05-23T16:45:39,692][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap05.est1933.com] to [10.173.148.50:9300]
  1417. [2019-05-23T16:45:39,693][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap06.est1933.com] to [10.173.148.67:9300]
  1418. [2019-05-23T16:45:39,693][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing resolved transport addresses [10.173.148.65:9300, 10.173.148.73:9300, 10.173.148.58:9300, 10.173.148.50:9300, 10.173.148.67:9300]
  1419. [2019-05-23T16:45:39,693][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  1420. [2019-05-23T16:45:39,693][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  1421. [2019-05-23T16:45:39,693][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  1422. [2019-05-23T16:45:39,694][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.50:9300]] opening probe connection
  1423. [2019-05-23T16:45:39,694][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.58:9300]] opening probe connection
  1424. [2019-05-23T16:45:39,694][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.67:9300]] opening probe connection
  1425. [2019-05-23T16:45:39,697][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  1426. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.67:9300] connect_exception
  1427. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  1428. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  1429. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  1430. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  1431. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  1432. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  1433. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  1434. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  1435. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  1436. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  1437. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  1438. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  1439. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  1440. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  1441. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  1442. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  1443. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  1444. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  1445. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  1446. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  1447. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  1448. at java.lang.Thread.run(Thread.java:835) [?:?]
  1449. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap06.est1933.com/10.173.148.67:9300
  1450. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  1451. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  1452. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  1453. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  1454. ... 6 more
  1455. Caused by: java.net.ConnectException: Connection refused
  1456. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  1457. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  1458. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  1459. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  1460. ... 6 more
  1461. [2019-05-23T16:45:39,699][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  1462. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.50:9300] connect_exception
  1463. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  1464. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  1465. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  1466. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  1467. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  1468. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  1469. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  1470. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  1471. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  1472. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  1473. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  1474. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  1475. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  1476. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  1477. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  1478. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  1479. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  1480. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  1481. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  1482. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  1483. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  1484. at java.lang.Thread.run(Thread.java:835) [?:?]
  1485. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap05.est1933.com/10.173.148.50:9300
  1486. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  1487. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  1488. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  1489. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  1490. ... 6 more
  1491. Caused by: java.net.ConnectException: Connection refused
  1492. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  1493. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  1494. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  1495. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  1496. ... 6 more
  1497. [2019-05-23T16:45:39,699][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  1498. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.58:9300] connect_exception
  1499. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  1500. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  1501. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  1502. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  1503. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  1504. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  1505. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  1506. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  1507. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  1508. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  1509. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  1510. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  1511. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  1512. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  1513. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  1514. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  1515. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  1516. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  1517. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  1518. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  1519. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  1520. at java.lang.Thread.run(Thread.java:835) [?:?]
  1521. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap04.est1933.com/10.173.148.58:9300
  1522. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  1523. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  1524. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  1525. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  1526. ... 6 more
  1527. Caused by: java.net.ConnectException: Connection refused
  1528. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  1529. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  1530. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  1531. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  1532. ... 6 more
  1533. [2019-05-23T16:45:39,836][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  1534. [2019-05-23T16:45:40,602][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  1535. [2019-05-23T16:45:40,690][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  1536. [2019-05-23T16:45:40,691][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  1537. [2019-05-23T16:45:40,691][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing master nodes from cluster state: nodes:
  1538. {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, xpack.installed=true, ml.max_open_jobs=20}, local
  1539.  
  1540. [2019-05-23T16:45:40,691][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  1541. [2019-05-23T16:45:40,692][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  1542. [2019-05-23T16:45:40,692][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  1543. [2019-05-23T16:45:40,692][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap01.est1933.com] to [10.173.148.143:9300]
  1544. [2019-05-23T16:45:40,692][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  1545. [2019-05-23T16:45:40,693][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap02.est1933.com] to [10.173.148.65:9300]
  1546. [2019-05-23T16:45:40,693][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  1547. [2019-05-23T16:45:40,693][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap03.est1933.com] to [10.173.148.73:9300]
  1548. [2019-05-23T16:45:40,693][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap04.est1933.com] to [10.173.148.58:9300]
  1549. [2019-05-23T16:45:40,693][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap05.est1933.com] to [10.173.148.50:9300]
  1550. [2019-05-23T16:45:40,693][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap06.est1933.com] to [10.173.148.67:9300]
  1551. [2019-05-23T16:45:40,693][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing resolved transport addresses [10.173.148.65:9300, 10.173.148.73:9300, 10.173.148.58:9300, 10.173.148.50:9300, 10.173.148.67:9300]
  1552. [2019-05-23T16:45:40,693][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  1553. [2019-05-23T16:45:40,693][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  1554. [2019-05-23T16:45:40,693][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  1555. [2019-05-23T16:45:40,693][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.50:9300]] opening probe connection
  1556. [2019-05-23T16:45:40,694][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.58:9300]] opening probe connection
  1557. [2019-05-23T16:45:40,694][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.67:9300]] opening probe connection
  1558. [2019-05-23T16:45:40,696][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  1559. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.50:9300] connect_exception
  1560. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  1561. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  1562. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  1563. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  1564. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  1565. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  1566. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  1567. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  1568. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  1569. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  1570. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  1571. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  1572. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  1573. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  1574. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  1575. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  1576. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  1577. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  1578. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  1579. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  1580. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  1581. at java.lang.Thread.run(Thread.java:835) [?:?]
  1582. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap05.est1933.com/10.173.148.50:9300
  1583. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  1584. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  1585. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  1586. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  1587. ... 6 more
  1588. Caused by: java.net.ConnectException: Connection refused
  1589. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  1590. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  1591. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  1592. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  1593. ... 6 more
  1594. [2019-05-23T16:45:40,699][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  1595. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.58:9300] connect_exception
  1596. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  1597. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  1598. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  1599. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  1600. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  1601. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  1602. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  1603. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  1604. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  1605. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  1606. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  1607. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  1608. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  1609. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  1610. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  1611. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  1612. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  1613. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  1614. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  1615. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  1616. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  1617. at java.lang.Thread.run(Thread.java:835) [?:?]
  1618. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap04.est1933.com/10.173.148.58:9300
  1619. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  1620. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  1621. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  1622. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  1623. ... 6 more
  1624. Caused by: java.net.ConnectException: Connection refused
  1625. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  1626. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  1627. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  1628. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  1629. ... 6 more
  1630. [2019-05-23T16:45:40,701][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  1631. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.67:9300] connect_exception
  1632. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  1633. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  1634. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  1635. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  1636. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  1637. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  1638. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  1639. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  1640. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  1641. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  1642. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  1643. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  1644. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  1645. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  1646. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  1647. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  1648. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  1649. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  1650. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  1651. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  1652. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  1653. at java.lang.Thread.run(Thread.java:835) [?:?]
  1654. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap06.est1933.com/10.173.148.67:9300
  1655. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  1656. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  1657. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  1658. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  1659. ... 6 more
  1660. Caused by: java.net.ConnectException: Connection refused
  1661. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  1662. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  1663. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  1664. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  1665. ... 6 more
  1666. [2019-05-23T16:45:40,836][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  1667. [2019-05-23T16:45:41,603][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  1668. [2019-05-23T16:45:41,692][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  1669. [2019-05-23T16:45:41,693][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  1670. [2019-05-23T16:45:41,693][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing master nodes from cluster state: nodes:
  1671. {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, xpack.installed=true, ml.max_open_jobs=20}, local
  1672.  
  1673. [2019-05-23T16:45:41,693][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  1674. [2019-05-23T16:45:41,694][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  1675. [2019-05-23T16:45:41,695][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  1676. [2019-05-23T16:45:41,695][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap01.est1933.com] to [10.173.148.143:9300]
  1677. [2019-05-23T16:45:41,696][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap02.est1933.com] to [10.173.148.65:9300]
  1678. [2019-05-23T16:45:41,696][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap03.est1933.com] to [10.173.148.73:9300]
  1679. [2019-05-23T16:45:41,696][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap04.est1933.com] to [10.173.148.58:9300]
  1680. [2019-05-23T16:45:41,696][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap05.est1933.com] to [10.173.148.50:9300]
  1681. [2019-05-23T16:45:41,696][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap06.est1933.com] to [10.173.148.67:9300]
  1682. [2019-05-23T16:45:41,696][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing resolved transport addresses [10.173.148.65:9300, 10.173.148.73:9300, 10.173.148.58:9300, 10.173.148.50:9300, 10.173.148.67:9300]
  1683. [2019-05-23T16:45:41,697][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  1684. [2019-05-23T16:45:41,697][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  1685. [2019-05-23T16:45:41,697][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  1686. [2019-05-23T16:45:41,697][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.58:9300]] opening probe connection
  1687. [2019-05-23T16:45:41,697][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.50:9300]] opening probe connection
  1688. [2019-05-23T16:45:41,697][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  1689. [2019-05-23T16:45:41,699][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.67:9300]] opening probe connection
  1690. [2019-05-23T16:45:41,700][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  1691. [2019-05-23T16:45:41,701][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  1692. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.50:9300] connect_exception
  1693. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  1694. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  1695. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  1696. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  1697. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  1698. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  1699. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  1700. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  1701. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  1702. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  1703. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  1704. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  1705. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  1706. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  1707. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  1708. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  1709. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  1710. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  1711. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  1712. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  1713. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  1714. at java.lang.Thread.run(Thread.java:835) [?:?]
  1715. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap05.est1933.com/10.173.148.50:9300
  1716. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  1717. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  1718. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  1719. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  1720. ... 6 more
  1721. Caused by: java.net.ConnectException: Connection refused
  1722. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  1723. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  1724. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  1725. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  1726. ... 6 more
  1727. [2019-05-23T16:45:41,706][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  1728. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.58:9300] connect_exception
  1729. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  1730. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  1731. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  1732. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  1733. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  1734. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  1735. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  1736. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  1737. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  1738. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  1739. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  1740. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  1741. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  1742. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  1743. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  1744. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  1745. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  1746. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  1747. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  1748. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  1749. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  1750. at java.lang.Thread.run(Thread.java:835) [?:?]
  1751. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap04.est1933.com/10.173.148.58:9300
  1752. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  1753. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  1754. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  1755. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  1756. ... 6 more
  1757. Caused by: java.net.ConnectException: Connection refused
  1758. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  1759. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  1760. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  1761. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  1762. ... 6 more
  1763. [2019-05-23T16:45:41,706][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  1764. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.67:9300] connect_exception
  1765. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  1766. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  1767. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  1768. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  1769. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  1770. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  1771. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  1772. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  1773. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  1774. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  1775. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  1776. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  1777. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  1778. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  1779. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  1780. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  1781. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  1782. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  1783. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  1784. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  1785. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  1786. at java.lang.Thread.run(Thread.java:835) [?:?]
  1787. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap06.est1933.com/10.173.148.67:9300
  1788. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  1789. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  1790. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  1791. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  1792. ... 6 more
  1793. Caused by: java.net.ConnectException: Connection refused
  1794. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  1795. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  1796. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  1797. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  1798. ... 6 more
  1799. [2019-05-23T16:45:41,836][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  1800. [2019-05-23T16:45:42,603][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  1801. [2019-05-23T16:45:42,694][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  1802. [2019-05-23T16:45:42,694][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  1803. [2019-05-23T16:45:42,695][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing master nodes from cluster state: nodes:
  1804. {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, xpack.installed=true, ml.max_open_jobs=20}, local
  1805.  
  1806. [2019-05-23T16:45:42,695][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  1807. [2019-05-23T16:45:42,696][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap01.est1933.com] to [10.173.148.143:9300]
  1808. [2019-05-23T16:45:42,696][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap02.est1933.com] to [10.173.148.65:9300]
  1809. [2019-05-23T16:45:42,696][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap03.est1933.com] to [10.173.148.73:9300]
  1810. [2019-05-23T16:45:42,696][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap04.est1933.com] to [10.173.148.58:9300]
  1811. [2019-05-23T16:45:42,696][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap05.est1933.com] to [10.173.148.50:9300]
  1812. [2019-05-23T16:45:42,697][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap06.est1933.com] to [10.173.148.67:9300]
  1813. [2019-05-23T16:45:42,697][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing resolved transport addresses [10.173.148.65:9300, 10.173.148.73:9300, 10.173.148.58:9300, 10.173.148.50:9300, 10.173.148.67:9300]
  1814. [2019-05-23T16:45:42,697][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  1815. [2019-05-23T16:45:42,697][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  1816. [2019-05-23T16:45:42,697][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  1817. [2019-05-23T16:45:42,697][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  1818. [2019-05-23T16:45:42,697][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.50:9300]] opening probe connection
  1819. [2019-05-23T16:45:42,697][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.67:9300]] opening probe connection
  1820. [2019-05-23T16:45:42,698][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  1821. [2019-05-23T16:45:42,698][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.58:9300]] opening probe connection
  1822. [2019-05-23T16:45:42,698][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  1823. [2019-05-23T16:45:42,699][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  1824. [2019-05-23T16:45:42,700][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  1825. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.67:9300] connect_exception
  1826. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  1827. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  1828. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  1829. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  1830. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  1831. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  1832. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  1833. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  1834. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  1835. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  1836. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  1837. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  1838. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  1839. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  1840. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  1841. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  1842. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  1843. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  1844. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  1845. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  1846. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  1847. at java.lang.Thread.run(Thread.java:835) [?:?]
  1848. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap06.est1933.com/10.173.148.67:9300
  1849. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  1850. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  1851. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  1852. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  1853. ... 6 more
  1854. Caused by: java.net.ConnectException: Connection refused
  1855. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  1856. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  1857. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  1858. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  1859. ... 6 more
  1860. [2019-05-23T16:45:42,702][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  1861. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.58:9300] connect_exception
  1862. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  1863. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  1864. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  1865. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  1866. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  1867. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  1868. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  1869. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  1870. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  1871. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  1872. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  1873. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  1874. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  1875. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  1876. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  1877. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  1878. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  1879. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  1880. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  1881. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  1882. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  1883. at java.lang.Thread.run(Thread.java:835) [?:?]
  1884. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap04.est1933.com/10.173.148.58:9300
  1885. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  1886. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  1887. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  1888. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  1889. ... 6 more
  1890. Caused by: java.net.ConnectException: Connection refused
  1891. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  1892. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  1893. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  1894. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  1895. ... 6 more
  1896. [2019-05-23T16:45:42,702][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  1897. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.50:9300] connect_exception
  1898. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  1899. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  1900. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  1901. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  1902. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  1903. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  1904. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  1905. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  1906. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  1907. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  1908. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  1909. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  1910. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  1911. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  1912. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  1913. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  1914. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  1915. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  1916. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  1917. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  1918. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  1919. at java.lang.Thread.run(Thread.java:835) [?:?]
  1920. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap05.est1933.com/10.173.148.50:9300
  1921. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  1922. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  1923. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  1924. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  1925. ... 6 more
  1926. Caused by: java.net.ConnectException: Connection refused
  1927. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  1928. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  1929. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  1930. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  1931. ... 6 more
  1932. [2019-05-23T16:45:42,837][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  1933. [2019-05-23T16:45:43,603][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  1934. [2019-05-23T16:45:43,695][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  1935. [2019-05-23T16:45:43,696][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  1936. [2019-05-23T16:45:43,697][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing master nodes from cluster state: nodes:
  1937. {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, xpack.installed=true, ml.max_open_jobs=20}, local
  1938.  
  1939. [2019-05-23T16:45:43,697][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  1940. [2019-05-23T16:45:43,698][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap01.est1933.com] to [10.173.148.143:9300]
  1941. [2019-05-23T16:45:43,698][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap02.est1933.com] to [10.173.148.65:9300]
  1942. [2019-05-23T16:45:43,698][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap03.est1933.com] to [10.173.148.73:9300]
  1943. [2019-05-23T16:45:43,698][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  1944. [2019-05-23T16:45:43,698][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap04.est1933.com] to [10.173.148.58:9300]
  1945. [2019-05-23T16:45:43,698][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  1946. [2019-05-23T16:45:43,698][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap05.est1933.com] to [10.173.148.50:9300]
  1947. [2019-05-23T16:45:43,699][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap06.est1933.com] to [10.173.148.67:9300]
  1948. [2019-05-23T16:45:43,699][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing resolved transport addresses [10.173.148.65:9300, 10.173.148.73:9300, 10.173.148.58:9300, 10.173.148.50:9300, 10.173.148.67:9300]
  1949. [2019-05-23T16:45:43,699][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  1950. [2019-05-23T16:45:43,699][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  1951. [2019-05-23T16:45:43,699][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  1952. [2019-05-23T16:45:43,699][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  1953. [2019-05-23T16:45:43,700][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.67:9300]] opening probe connection
  1954. [2019-05-23T16:45:43,700][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  1955. [2019-05-23T16:45:43,700][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.58:9300]] opening probe connection
  1956. [2019-05-23T16:45:43,701][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.50:9300]] opening probe connection
  1957. [2019-05-23T16:45:43,702][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  1958. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.58:9300] connect_exception
  1959. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  1960. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  1961. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  1962. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  1963. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  1964. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  1965. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  1966. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  1967. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  1968. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  1969. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  1970. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  1971. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  1972. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  1973. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  1974. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  1975. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  1976. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  1977. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  1978. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  1979. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  1980. at java.lang.Thread.run(Thread.java:835) [?:?]
  1981. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap04.est1933.com/10.173.148.58:9300
  1982. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  1983. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  1984. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  1985. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  1986. ... 6 more
  1987. Caused by: java.net.ConnectException: Connection refused
  1988. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  1989. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  1990. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  1991. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  1992. ... 6 more
  1993. [2019-05-23T16:45:43,702][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  1994. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.67:9300] connect_exception
  1995. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  1996. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  1997. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  1998. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  1999. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  2000. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  2001. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  2002. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  2003. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  2004. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  2005. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  2006. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  2007. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  2008. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  2009. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  2010. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  2011. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  2012. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  2013. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  2014. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  2015. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  2016. at java.lang.Thread.run(Thread.java:835) [?:?]
  2017. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap06.est1933.com/10.173.148.67:9300
  2018. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2019. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2020. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2021. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2022. ... 6 more
  2023. Caused by: java.net.ConnectException: Connection refused
  2024. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2025. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2026. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2027. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2028. ... 6 more
  2029. [2019-05-23T16:45:43,706][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  2030. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.50:9300] connect_exception
  2031. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  2032. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  2033. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  2034. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  2035. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  2036. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  2037. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  2038. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  2039. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  2040. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  2041. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  2042. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  2043. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  2044. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  2045. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  2046. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  2047. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  2048. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  2049. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  2050. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  2051. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  2052. at java.lang.Thread.run(Thread.java:835) [?:?]
  2053. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap05.est1933.com/10.173.148.50:9300
  2054. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2055. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2056. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2057. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2058. ... 6 more
  2059. Caused by: java.net.ConnectException: Connection refused
  2060. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2061. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2062. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2063. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2064. ... 6 more
  2065. [2019-05-23T16:45:43,837][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  2066. [2019-05-23T16:45:44,603][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  2067. [2019-05-23T16:45:44,697][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  2068. [2019-05-23T16:45:44,698][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  2069. [2019-05-23T16:45:44,698][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing master nodes from cluster state: nodes:
  2070. {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, xpack.installed=true, ml.max_open_jobs=20}, local
  2071.  
  2072. [2019-05-23T16:45:44,698][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  2073. [2019-05-23T16:45:44,699][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap01.est1933.com] to [10.173.148.143:9300]
  2074. [2019-05-23T16:45:44,699][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap02.est1933.com] to [10.173.148.65:9300]
  2075. [2019-05-23T16:45:44,699][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap03.est1933.com] to [10.173.148.73:9300]
  2076. [2019-05-23T16:45:44,699][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap04.est1933.com] to [10.173.148.58:9300]
  2077. [2019-05-23T16:45:44,699][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap05.est1933.com] to [10.173.148.50:9300]
  2078. [2019-05-23T16:45:44,699][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap06.est1933.com] to [10.173.148.67:9300]
  2079. [2019-05-23T16:45:44,699][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing resolved transport addresses [10.173.148.65:9300, 10.173.148.73:9300, 10.173.148.58:9300, 10.173.148.50:9300, 10.173.148.67:9300]
  2080. [2019-05-23T16:45:44,699][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  2081. [2019-05-23T16:45:44,700][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  2082. [2019-05-23T16:45:44,700][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  2083. [2019-05-23T16:45:44,700][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.58:9300]] opening probe connection
  2084. [2019-05-23T16:45:44,700][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.50:9300]] opening probe connection
  2085. [2019-05-23T16:45:44,700][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  2086. [2019-05-23T16:45:44,700][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  2087. [2019-05-23T16:45:44,700][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.67:9300]] opening probe connection
  2088. [2019-05-23T16:45:44,701][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  2089. [2019-05-23T16:45:44,701][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  2090. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.58:9300] connect_exception
  2091. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  2092. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  2093. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  2094. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  2095. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  2096. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  2097. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  2098. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  2099. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  2100. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  2101. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  2102. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  2103. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  2104. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  2105. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  2106. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  2107. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  2108. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  2109. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  2110. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  2111. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  2112. at java.lang.Thread.run(Thread.java:835) [?:?]
  2113. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap04.est1933.com/10.173.148.58:9300
  2114. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2115. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2116. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2117. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2118. ... 6 more
  2119. Caused by: java.net.ConnectException: Connection refused
  2120. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2121. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2122. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2123. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2124. ... 6 more
  2125. [2019-05-23T16:45:44,702][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  2126. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.50:9300] connect_exception
  2127. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  2128. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  2129. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  2130. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  2131. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  2132. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  2133. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  2134. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  2135. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  2136. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  2137. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  2138. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  2139. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  2140. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  2141. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  2142. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  2143. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  2144. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  2145. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  2146. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  2147. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  2148. at java.lang.Thread.run(Thread.java:835) [?:?]
  2149. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap05.est1933.com/10.173.148.50:9300
  2150. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2151. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2152. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2153. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2154. ... 6 more
  2155. Caused by: java.net.ConnectException: Connection refused
  2156. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2157. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2158. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2159. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2160. ... 6 more
  2161. [2019-05-23T16:45:44,704][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  2162. [2019-05-23T16:45:44,705][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  2163. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.67:9300] connect_exception
  2164. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  2165. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  2166. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  2167. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  2168. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  2169. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  2170. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  2171. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  2172. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  2173. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  2174. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  2175. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  2176. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  2177. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  2178. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  2179. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  2180. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  2181. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  2182. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  2183. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  2184. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  2185. at java.lang.Thread.run(Thread.java:835) [?:?]
  2186. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap06.est1933.com/10.173.148.67:9300
  2187. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2188. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2189. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2190. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2191. ... 6 more
  2192. Caused by: java.net.ConnectException: Connection refused
  2193. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2194. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2195. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2196. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2197. ... 6 more
  2198. [2019-05-23T16:45:44,837][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  2199. [2019-05-23T16:45:45,604][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  2200. [2019-05-23T16:45:45,699][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  2201. [2019-05-23T16:45:45,699][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  2202. [2019-05-23T16:45:45,700][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing master nodes from cluster state: nodes:
  2203. {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, xpack.installed=true, ml.max_open_jobs=20}, local
  2204.  
  2205. [2019-05-23T16:45:45,700][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  2206. [2019-05-23T16:45:45,701][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  2207. [2019-05-23T16:45:45,701][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  2208. [2019-05-23T16:45:45,701][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  2209. [2019-05-23T16:45:45,701][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  2210. [2019-05-23T16:45:45,701][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap01.est1933.com] to [10.173.148.143:9300]
  2211. [2019-05-23T16:45:45,701][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap02.est1933.com] to [10.173.148.65:9300]
  2212. [2019-05-23T16:45:45,702][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap03.est1933.com] to [10.173.148.73:9300]
  2213. [2019-05-23T16:45:45,702][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap04.est1933.com] to [10.173.148.58:9300]
  2214. [2019-05-23T16:45:45,702][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap05.est1933.com] to [10.173.148.50:9300]
  2215. [2019-05-23T16:45:45,702][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap06.est1933.com] to [10.173.148.67:9300]
  2216. [2019-05-23T16:45:45,702][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing resolved transport addresses [10.173.148.65:9300, 10.173.148.73:9300, 10.173.148.58:9300, 10.173.148.50:9300, 10.173.148.67:9300]
  2217. [2019-05-23T16:45:45,702][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  2218. [2019-05-23T16:45:45,703][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  2219. [2019-05-23T16:45:45,703][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.58:9300]] opening probe connection
  2220. [2019-05-23T16:45:45,703][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  2221. [2019-05-23T16:45:45,703][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.50:9300]] opening probe connection
  2222. [2019-05-23T16:45:45,703][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.67:9300]] opening probe connection
  2223. [2019-05-23T16:45:45,705][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  2224. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.67:9300] connect_exception
  2225. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  2226. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  2227. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  2228. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  2229. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  2230. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  2231. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  2232. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  2233. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  2234. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  2235. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  2236. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  2237. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  2238. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  2239. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  2240. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  2241. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  2242. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  2243. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  2244. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  2245. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  2246. at java.lang.Thread.run(Thread.java:835) [?:?]
  2247. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap06.est1933.com/10.173.148.67:9300
  2248. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2249. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2250. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2251. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2252. ... 6 more
  2253. Caused by: java.net.ConnectException: Connection refused
  2254. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2255. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2256. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2257. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2258. ... 6 more
  2259. [2019-05-23T16:45:45,705][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  2260. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.50:9300] connect_exception
  2261. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  2262. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  2263. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  2264. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  2265. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  2266. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  2267. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  2268. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  2269. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  2270. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  2271. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  2272. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  2273. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  2274. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  2275. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  2276. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  2277. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  2278. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  2279. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  2280. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  2281. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  2282. at java.lang.Thread.run(Thread.java:835) [?:?]
  2283. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap05.est1933.com/10.173.148.50:9300
  2284. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2285. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2286. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2287. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2288. ... 6 more
  2289. Caused by: java.net.ConnectException: Connection refused
  2290. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2291. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2292. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2293. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2294. ... 6 more
  2295. [2019-05-23T16:45:45,706][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  2296. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.58:9300] connect_exception
  2297. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  2298. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  2299. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  2300. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  2301. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  2302. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  2303. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  2304. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  2305. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  2306. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  2307. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  2308. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  2309. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  2310. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  2311. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  2312. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  2313. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  2314. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  2315. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  2316. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  2317. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  2318. at java.lang.Thread.run(Thread.java:835) [?:?]
  2319. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap04.est1933.com/10.173.148.58:9300
  2320. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2321. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2322. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2323. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2324. ... 6 more
  2325. Caused by: java.net.ConnectException: Connection refused
  2326. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2327. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2328. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2329. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2330. ... 6 more
  2331. [2019-05-23T16:45:45,837][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  2332. [2019-05-23T16:45:46,604][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  2333. [2019-05-23T16:45:46,700][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  2334. [2019-05-23T16:45:46,701][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  2335. [2019-05-23T16:45:46,701][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing master nodes from cluster state: nodes:
  2336. {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, xpack.installed=true, ml.max_open_jobs=20}, local
  2337.  
  2338. [2019-05-23T16:45:46,701][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  2339. [2019-05-23T16:45:46,702][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  2340. [2019-05-23T16:45:46,702][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap01.est1933.com] to [10.173.148.143:9300]
  2341. [2019-05-23T16:45:46,702][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  2342. [2019-05-23T16:45:46,702][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap02.est1933.com] to [10.173.148.65:9300]
  2343. [2019-05-23T16:45:46,702][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  2344. [2019-05-23T16:45:46,702][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap03.est1933.com] to [10.173.148.73:9300]
  2345. [2019-05-23T16:45:46,702][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  2346. [2019-05-23T16:45:46,703][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap04.est1933.com] to [10.173.148.58:9300]
  2347. [2019-05-23T16:45:46,703][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap05.est1933.com] to [10.173.148.50:9300]
  2348. [2019-05-23T16:45:46,703][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap06.est1933.com] to [10.173.148.67:9300]
  2349. [2019-05-23T16:45:46,703][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing resolved transport addresses [10.173.148.65:9300, 10.173.148.73:9300, 10.173.148.58:9300, 10.173.148.50:9300, 10.173.148.67:9300]
  2350. [2019-05-23T16:45:46,703][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  2351. [2019-05-23T16:45:46,703][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  2352. [2019-05-23T16:45:46,703][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.58:9300]] opening probe connection
  2353. [2019-05-23T16:45:46,703][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  2354. [2019-05-23T16:45:46,703][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.67:9300]] opening probe connection
  2355. [2019-05-23T16:45:46,703][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.50:9300]] opening probe connection
  2356. [2019-05-23T16:45:46,704][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  2357. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.67:9300] connect_exception
  2358. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  2359. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  2360. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  2361. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  2362. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  2363. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  2364. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  2365. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  2366. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  2367. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  2368. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  2369. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  2370. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  2371. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  2372. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  2373. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  2374. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  2375. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  2376. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  2377. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  2378. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  2379. at java.lang.Thread.run(Thread.java:835) [?:?]
  2380. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap06.est1933.com/10.173.148.67:9300
  2381. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2382. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2383. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2384. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2385. ... 6 more
  2386. Caused by: java.net.ConnectException: Connection refused
  2387. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2388. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2389. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2390. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2391. ... 6 more
  2392. [2019-05-23T16:45:46,705][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  2393. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.58:9300] connect_exception
  2394. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  2395. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  2396. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  2397. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  2398. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  2399. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  2400. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  2401. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  2402. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  2403. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  2404. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  2405. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  2406. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  2407. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  2408. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  2409. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  2410. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  2411. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  2412. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  2413. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  2414. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  2415. at java.lang.Thread.run(Thread.java:835) [?:?]
  2416. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap04.est1933.com/10.173.148.58:9300
  2417. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2418. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2419. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2420. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2421. ... 6 more
  2422. Caused by: java.net.ConnectException: Connection refused
  2423. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2424. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2425. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2426. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2427. ... 6 more
  2428. [2019-05-23T16:45:46,706][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  2429. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.50:9300] connect_exception
  2430. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  2431. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  2432. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  2433. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  2434. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  2435. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  2436. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  2437. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  2438. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  2439. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  2440. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  2441. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  2442. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  2443. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  2444. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  2445. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  2446. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  2447. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  2448. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  2449. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  2450. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  2451. at java.lang.Thread.run(Thread.java:835) [?:?]
  2452. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap05.est1933.com/10.173.148.50:9300
  2453. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2454. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2455. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2456. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2457. ... 6 more
  2458. Caused by: java.net.ConnectException: Connection refused
  2459. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2460. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2461. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2462. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2463. ... 6 more
  2464. [2019-05-23T16:45:46,838][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  2465. [2019-05-23T16:45:47,604][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  2466. [2019-05-23T16:45:47,701][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  2467. [2019-05-23T16:45:47,702][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  2468. [2019-05-23T16:45:47,702][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing master nodes from cluster state: nodes:
  2469. {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, xpack.installed=true, ml.max_open_jobs=20}, local
  2470.  
  2471. [2019-05-23T16:45:47,702][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  2472. [2019-05-23T16:45:47,703][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap01.est1933.com] to [10.173.148.143:9300]
  2473. [2019-05-23T16:45:47,703][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap02.est1933.com] to [10.173.148.65:9300]
  2474. [2019-05-23T16:45:47,703][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap03.est1933.com] to [10.173.148.73:9300]
  2475. [2019-05-23T16:45:47,703][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap04.est1933.com] to [10.173.148.58:9300]
  2476. [2019-05-23T16:45:47,703][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap05.est1933.com] to [10.173.148.50:9300]
  2477. [2019-05-23T16:45:47,703][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap06.est1933.com] to [10.173.148.67:9300]
  2478. [2019-05-23T16:45:47,703][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  2479. [2019-05-23T16:45:47,703][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing resolved transport addresses [10.173.148.65:9300, 10.173.148.73:9300, 10.173.148.58:9300, 10.173.148.50:9300, 10.173.148.67:9300]
  2480. [2019-05-23T16:45:47,704][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  2481. [2019-05-23T16:45:47,704][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  2482. [2019-05-23T16:45:47,704][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  2483. [2019-05-23T16:45:47,704][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.58:9300]] opening probe connection
  2484. [2019-05-23T16:45:47,704][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.50:9300]] opening probe connection
  2485. [2019-05-23T16:45:47,704][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  2486. [2019-05-23T16:45:47,704][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  2487. [2019-05-23T16:45:47,704][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  2488. [2019-05-23T16:45:47,704][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.67:9300]] opening probe connection
  2489. [2019-05-23T16:45:47,705][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  2490. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.58:9300] connect_exception
  2491. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  2492. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  2493. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  2494. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  2495. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  2496. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  2497. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  2498. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  2499. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  2500. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  2501. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  2502. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  2503. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  2504. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  2505. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  2506. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  2507. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  2508. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  2509. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  2510. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  2511. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  2512. at java.lang.Thread.run(Thread.java:835) [?:?]
  2513. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap04.est1933.com/10.173.148.58:9300
  2514. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2515. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2516. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2517. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2518. ... 6 more
  2519. Caused by: java.net.ConnectException: Connection refused
  2520. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2521. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2522. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2523. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2524. ... 6 more
  2525. [2019-05-23T16:45:47,705][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  2526. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.50:9300] connect_exception
  2527. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  2528. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  2529. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  2530. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  2531. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  2532. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  2533. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  2534. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  2535. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  2536. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  2537. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  2538. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  2539. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  2540. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  2541. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  2542. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  2543. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  2544. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  2545. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  2546. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  2547. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  2548. at java.lang.Thread.run(Thread.java:835) [?:?]
  2549. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap05.est1933.com/10.173.148.50:9300
  2550. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2551. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2552. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2553. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2554. ... 6 more
  2555. Caused by: java.net.ConnectException: Connection refused
  2556. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2557. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2558. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2559. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2560. ... 6 more
  2561. [2019-05-23T16:45:47,706][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  2562. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.67:9300] connect_exception
  2563. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  2564. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  2565. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  2566. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  2567. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  2568. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  2569. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  2570. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  2571. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  2572. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  2573. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  2574. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  2575. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  2576. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  2577. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  2578. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  2579. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  2580. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  2581. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  2582. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  2583. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  2584. at java.lang.Thread.run(Thread.java:835) [?:?]
  2585. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap06.est1933.com/10.173.148.67:9300
  2586. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2587. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2588. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2589. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2590. ... 6 more
  2591. Caused by: java.net.ConnectException: Connection refused
  2592. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2593. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2594. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2595. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2596. ... 6 more
  2597. [2019-05-23T16:45:47,838][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  2598. [2019-05-23T16:45:48,605][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  2599. [2019-05-23T16:45:48,703][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  2600. [2019-05-23T16:45:48,703][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  2601. [2019-05-23T16:45:48,704][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing master nodes from cluster state: nodes:
  2602. {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, xpack.installed=true, ml.max_open_jobs=20}, local
  2603.  
  2604. [2019-05-23T16:45:48,704][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  2605. [2019-05-23T16:45:48,705][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  2606. [2019-05-23T16:45:48,705][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  2607. [2019-05-23T16:45:48,705][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap01.est1933.com] to [10.173.148.143:9300]
  2608. [2019-05-23T16:45:48,705][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap02.est1933.com] to [10.173.148.65:9300]
  2609. [2019-05-23T16:45:48,705][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap03.est1933.com] to [10.173.148.73:9300]
  2610. [2019-05-23T16:45:48,705][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap04.est1933.com] to [10.173.148.58:9300]
  2611. [2019-05-23T16:45:48,705][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap05.est1933.com] to [10.173.148.50:9300]
  2612. [2019-05-23T16:45:48,705][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap06.est1933.com] to [10.173.148.67:9300]
  2613. [2019-05-23T16:45:48,705][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing resolved transport addresses [10.173.148.65:9300, 10.173.148.73:9300, 10.173.148.58:9300, 10.173.148.50:9300, 10.173.148.67:9300]
  2614. [2019-05-23T16:45:48,705][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  2615. [2019-05-23T16:45:48,705][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  2616. [2019-05-23T16:45:48,706][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  2617. [2019-05-23T16:45:48,706][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.58:9300]] opening probe connection
  2618. [2019-05-23T16:45:48,706][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  2619. [2019-05-23T16:45:48,706][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.50:9300]] opening probe connection
  2620. [2019-05-23T16:45:48,706][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  2621. [2019-05-23T16:45:48,706][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.67:9300]] opening probe connection
  2622. [2019-05-23T16:45:48,707][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  2623. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.58:9300] connect_exception
  2624. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  2625. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  2626. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  2627. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  2628. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  2629. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  2630. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  2631. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  2632. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  2633. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  2634. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  2635. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  2636. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  2637. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  2638. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  2639. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  2640. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  2641. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  2642. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  2643. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  2644. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  2645. at java.lang.Thread.run(Thread.java:835) [?:?]
  2646. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap04.est1933.com/10.173.148.58:9300
  2647. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2648. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2649. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2650. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2651. ... 6 more
  2652. Caused by: java.net.ConnectException: Connection refused
  2653. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2654. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2655. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2656. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2657. ... 6 more
  2658. [2019-05-23T16:45:48,707][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  2659. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.50:9300] connect_exception
  2660. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  2661. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  2662. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  2663. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  2664. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  2665. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  2666. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  2667. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  2668. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  2669. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  2670. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  2671. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  2672. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  2673. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  2674. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  2675. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  2676. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  2677. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  2678. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  2679. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  2680. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  2681. at java.lang.Thread.run(Thread.java:835) [?:?]
  2682. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap05.est1933.com/10.173.148.50:9300
  2683. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2684. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2685. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2686. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2687. ... 6 more
  2688. Caused by: java.net.ConnectException: Connection refused
  2689. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2690. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2691. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2692. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2693. ... 6 more
  2694. [2019-05-23T16:45:48,707][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  2695. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.67:9300] connect_exception
  2696. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  2697. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  2698. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  2699. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  2700. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  2701. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  2702. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  2703. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  2704. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  2705. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  2706. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  2707. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  2708. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  2709. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  2710. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  2711. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  2712. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  2713. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  2714. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  2715. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  2716. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  2717. at java.lang.Thread.run(Thread.java:835) [?:?]
  2718. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap06.est1933.com/10.173.148.67:9300
  2719. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2720. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2721. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2722. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2723. ... 6 more
  2724. Caused by: java.net.ConnectException: Connection refused
  2725. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2726. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2727. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2728. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2729. ... 6 more
  2730. [2019-05-23T16:45:48,838][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  2731. [2019-05-23T16:45:49,605][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  2732. [2019-05-23T16:45:49,679][WARN ][o.e.c.c.ClusterFormationFailureHelper] [awselsdevlap01.est1933.com-esnode01] master not discovered or elected yet, an election requires at least 2 nodes with ids from [UWgnBPsHQ1aW4xXEZVKyJQ, T_8EuKpaTqWg2oP3TAnAaA, YZ6m2ioDQWqi1cNnOteB6w], have discovered [{awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}] which is a quorum; discovery will continue using [10.173.148.65:9300, 10.173.148.73:9300, 10.173.148.58:9300, 10.173.148.50:9300, 10.173.148.67:9300] from hosts providers and [{awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 22, last-accepted version 2867 in term 21
  2733. [2019-05-23T16:45:49,704][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  2734. [2019-05-23T16:45:49,704][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  2735. [2019-05-23T16:45:49,705][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing master nodes from cluster state: nodes:
  2736. {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, xpack.installed=true, ml.max_open_jobs=20}, local
  2737.  
  2738. [2019-05-23T16:45:49,705][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  2739. [2019-05-23T16:45:49,706][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap01.est1933.com] to [10.173.148.143:9300]
  2740. [2019-05-23T16:45:49,707][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap02.est1933.com] to [10.173.148.65:9300]
  2741. [2019-05-23T16:45:49,707][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap03.est1933.com] to [10.173.148.73:9300]
  2742. [2019-05-23T16:45:49,707][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap04.est1933.com] to [10.173.148.58:9300]
  2743. [2019-05-23T16:45:49,707][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap05.est1933.com] to [10.173.148.50:9300]
  2744. [2019-05-23T16:45:49,707][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap06.est1933.com] to [10.173.148.67:9300]
  2745. [2019-05-23T16:45:49,707][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing resolved transport addresses [10.173.148.65:9300, 10.173.148.73:9300, 10.173.148.58:9300, 10.173.148.50:9300, 10.173.148.67:9300]
  2746. [2019-05-23T16:45:49,707][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  2747. [2019-05-23T16:45:49,707][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  2748. [2019-05-23T16:45:49,707][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  2749. [2019-05-23T16:45:49,707][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  2750. [2019-05-23T16:45:49,707][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  2751. [2019-05-23T16:45:49,707][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.67:9300]] opening probe connection
  2752. [2019-05-23T16:45:49,707][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  2753. [2019-05-23T16:45:49,708][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  2754. [2019-05-23T16:45:49,708][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.50:9300]] opening probe connection
  2755. [2019-05-23T16:45:49,708][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.58:9300]] opening probe connection
  2756. [2019-05-23T16:45:49,710][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  2757. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.50:9300] connect_exception
  2758. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  2759. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  2760. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  2761. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  2762. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  2763. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  2764. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  2765. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  2766. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  2767. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  2768. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  2769. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  2770. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  2771. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  2772. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  2773. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  2774. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  2775. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  2776. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  2777. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  2778. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  2779. at java.lang.Thread.run(Thread.java:835) [?:?]
  2780. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap05.est1933.com/10.173.148.50:9300
  2781. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2782. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2783. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2784. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2785. ... 6 more
  2786. Caused by: java.net.ConnectException: Connection refused
  2787. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2788. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2789. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2790. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2791. ... 6 more
  2792. [2019-05-23T16:45:49,710][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  2793. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.67:9300] connect_exception
  2794. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  2795. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  2796. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  2797. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  2798. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  2799. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  2800. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  2801. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  2802. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  2803. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  2804. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  2805. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  2806. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  2807. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  2808. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  2809. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  2810. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  2811. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  2812. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  2813. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  2814. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  2815. at java.lang.Thread.run(Thread.java:835) [?:?]
  2816. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap06.est1933.com/10.173.148.67:9300
  2817. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2818. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2819. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2820. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2821. ... 6 more
  2822. Caused by: java.net.ConnectException: Connection refused
  2823. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2824. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2825. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2826. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2827. ... 6 more
  2828. [2019-05-23T16:45:49,710][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  2829. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.58:9300] connect_exception
  2830. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  2831. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  2832. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  2833. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  2834. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  2835. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  2836. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  2837. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  2838. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  2839. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  2840. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  2841. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  2842. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  2843. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  2844. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  2845. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  2846. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  2847. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  2848. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  2849. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  2850. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  2851. at java.lang.Thread.run(Thread.java:835) [?:?]
  2852. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap04.est1933.com/10.173.148.58:9300
  2853. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2854. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2855. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2856. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2857. ... 6 more
  2858. Caused by: java.net.ConnectException: Connection refused
  2859. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2860. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2861. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2862. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2863. ... 6 more
  2864. [2019-05-23T16:45:49,838][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  2865. [2019-05-23T16:45:50,605][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  2866. [2019-05-23T16:45:50,705][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  2867. [2019-05-23T16:45:50,706][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  2868. [2019-05-23T16:45:50,706][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing master nodes from cluster state: nodes:
  2869. {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, xpack.installed=true, ml.max_open_jobs=20}, local
  2870.  
  2871. [2019-05-23T16:45:50,706][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  2872. [2019-05-23T16:45:50,707][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  2873. [2019-05-23T16:45:50,707][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  2874. [2019-05-23T16:45:50,707][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap01.est1933.com] to [10.173.148.143:9300]
  2875. [2019-05-23T16:45:50,707][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  2876. [2019-05-23T16:45:50,707][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap02.est1933.com] to [10.173.148.65:9300]
  2877. [2019-05-23T16:45:50,707][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap03.est1933.com] to [10.173.148.73:9300]
  2878. [2019-05-23T16:45:50,708][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap04.est1933.com] to [10.173.148.58:9300]
  2879. [2019-05-23T16:45:50,707][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  2880. [2019-05-23T16:45:50,708][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap05.est1933.com] to [10.173.148.50:9300]
  2881. [2019-05-23T16:45:50,708][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap06.est1933.com] to [10.173.148.67:9300]
  2882. [2019-05-23T16:45:50,708][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing resolved transport addresses [10.173.148.65:9300, 10.173.148.73:9300, 10.173.148.58:9300, 10.173.148.50:9300, 10.173.148.67:9300]
  2883. [2019-05-23T16:45:50,708][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  2884. [2019-05-23T16:45:50,708][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  2885. [2019-05-23T16:45:50,708][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.58:9300]] opening probe connection
  2886. [2019-05-23T16:45:50,708][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  2887. [2019-05-23T16:45:50,708][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.67:9300]] opening probe connection
  2888. [2019-05-23T16:45:50,708][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.50:9300]] opening probe connection
  2889. [2019-05-23T16:45:50,709][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  2890. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.58:9300] connect_exception
  2891. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  2892. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  2893. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  2894. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  2895. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  2896. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  2897. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  2898. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  2899. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  2900. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  2901. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  2902. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  2903. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  2904. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  2905. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  2906. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  2907. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  2908. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  2909. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  2910. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  2911. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  2912. at java.lang.Thread.run(Thread.java:835) [?:?]
  2913. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap04.est1933.com/10.173.148.58:9300
  2914. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2915. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2916. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2917. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2918. ... 6 more
  2919. Caused by: java.net.ConnectException: Connection refused
  2920. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2921. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2922. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2923. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2924. ... 6 more
  2925. [2019-05-23T16:45:50,710][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  2926. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.67:9300] connect_exception
  2927. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  2928. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  2929. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  2930. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  2931. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  2932. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  2933. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  2934. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  2935. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  2936. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  2937. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  2938. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  2939. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  2940. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  2941. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  2942. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  2943. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  2944. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  2945. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  2946. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  2947. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  2948. at java.lang.Thread.run(Thread.java:835) [?:?]
  2949. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap06.est1933.com/10.173.148.67:9300
  2950. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2951. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2952. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2953. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2954. ... 6 more
  2955. Caused by: java.net.ConnectException: Connection refused
  2956. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2957. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2958. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2959. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2960. ... 6 more
  2961. [2019-05-23T16:45:50,711][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  2962. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.50:9300] connect_exception
  2963. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  2964. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  2965. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  2966. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  2967. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  2968. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  2969. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  2970. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  2971. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  2972. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  2973. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  2974. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  2975. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  2976. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  2977. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  2978. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  2979. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  2980. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  2981. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  2982. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  2983. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  2984. at java.lang.Thread.run(Thread.java:835) [?:?]
  2985. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap05.est1933.com/10.173.148.50:9300
  2986. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2987. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2988. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2989. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2990. ... 6 more
  2991. Caused by: java.net.ConnectException: Connection refused
  2992. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  2993. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  2994. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  2995. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  2996. ... 6 more
  2997. [2019-05-23T16:45:50,838][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  2998. [2019-05-23T16:45:51,606][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  2999. [2019-05-23T16:45:51,707][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  3000. [2019-05-23T16:45:51,707][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  3001. [2019-05-23T16:45:51,708][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing master nodes from cluster state: nodes:
  3002. {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, xpack.installed=true, ml.max_open_jobs=20}, local
  3003.  
  3004. [2019-05-23T16:45:51,708][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  3005. [2019-05-23T16:45:51,709][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap01.est1933.com] to [10.173.148.143:9300]
  3006. [2019-05-23T16:45:51,709][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap02.est1933.com] to [10.173.148.65:9300]
  3007. [2019-05-23T16:45:51,709][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  3008. [2019-05-23T16:45:51,709][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap03.est1933.com] to [10.173.148.73:9300]
  3009. [2019-05-23T16:45:51,709][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  3010. [2019-05-23T16:45:51,709][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap04.est1933.com] to [10.173.148.58:9300]
  3011. [2019-05-23T16:45:51,709][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap05.est1933.com] to [10.173.148.50:9300]
  3012. [2019-05-23T16:45:51,709][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap06.est1933.com] to [10.173.148.67:9300]
  3013. [2019-05-23T16:45:51,709][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  3014. [2019-05-23T16:45:51,709][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing resolved transport addresses [10.173.148.65:9300, 10.173.148.73:9300, 10.173.148.58:9300, 10.173.148.50:9300, 10.173.148.67:9300]
  3015. [2019-05-23T16:45:51,710][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  3016. [2019-05-23T16:45:51,710][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  3017. [2019-05-23T16:45:51,710][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  3018. [2019-05-23T16:45:51,710][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.50:9300]] opening probe connection
  3019. [2019-05-23T16:45:51,710][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.67:9300]] opening probe connection
  3020. [2019-05-23T16:45:51,711][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  3021. [2019-05-23T16:45:51,711][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.58:9300]] opening probe connection
  3022. [2019-05-23T16:45:51,712][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  3023. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.67:9300] connect_exception
  3024. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  3025. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  3026. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  3027. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  3028. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  3029. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  3030. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  3031. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  3032. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  3033. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  3034. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  3035. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  3036. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  3037. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  3038. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  3039. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  3040. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  3041. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  3042. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  3043. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  3044. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  3045. at java.lang.Thread.run(Thread.java:835) [?:?]
  3046. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap06.est1933.com/10.173.148.67:9300
  3047. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  3048. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  3049. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  3050. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  3051. ... 6 more
  3052. Caused by: java.net.ConnectException: Connection refused
  3053. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  3054. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  3055. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  3056. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  3057. ... 6 more
  3058. [2019-05-23T16:45:51,713][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  3059. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.58:9300] connect_exception
  3060. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  3061. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  3062. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  3063. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  3064. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  3065. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  3066. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  3067. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  3068. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  3069. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  3070. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  3071. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  3072. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  3073. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  3074. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  3075. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  3076. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  3077. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  3078. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  3079. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  3080. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  3081. at java.lang.Thread.run(Thread.java:835) [?:?]
  3082. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap04.est1933.com/10.173.148.58:9300
  3083. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  3084. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  3085. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  3086. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  3087. ... 6 more
  3088. Caused by: java.net.ConnectException: Connection refused
  3089. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  3090. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  3091. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  3092. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  3093. ... 6 more
  3094. [2019-05-23T16:45:51,712][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  3095. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.50:9300] connect_exception
  3096. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  3097. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  3098. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  3099. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  3100. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  3101. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  3102. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  3103. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  3104. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  3105. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  3106. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  3107. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  3108. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  3109. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  3110. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  3111. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  3112. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  3113. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  3114. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  3115. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  3116. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  3117. at java.lang.Thread.run(Thread.java:835) [?:?]
  3118. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap05.est1933.com/10.173.148.50:9300
  3119. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  3120. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  3121. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  3122. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  3123. ... 6 more
  3124. Caused by: java.net.ConnectException: Connection refused
  3125. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  3126. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  3127. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  3128. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  3129. ... 6 more
  3130. [2019-05-23T16:45:51,839][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  3131. [2019-05-23T16:45:52,606][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  3132. [2019-05-23T16:45:52,708][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  3133. [2019-05-23T16:45:52,709][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  3134. [2019-05-23T16:45:52,709][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing master nodes from cluster state: nodes:
  3135. {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, xpack.installed=true, ml.max_open_jobs=20}, local
  3136.  
  3137. [2019-05-23T16:45:52,709][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  3138. [2019-05-23T16:45:52,710][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap01.est1933.com] to [10.173.148.143:9300]
  3139. [2019-05-23T16:45:52,710][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  3140. [2019-05-23T16:45:52,710][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap02.est1933.com] to [10.173.148.65:9300]
  3141. [2019-05-23T16:45:52,711][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  3142. [2019-05-23T16:45:52,711][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap03.est1933.com] to [10.173.148.73:9300]
  3143. [2019-05-23T16:45:52,711][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap04.est1933.com] to [10.173.148.58:9300]
  3144. [2019-05-23T16:45:52,711][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap05.est1933.com] to [10.173.148.50:9300]
  3145. [2019-05-23T16:45:52,711][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap06.est1933.com] to [10.173.148.67:9300]
  3146. [2019-05-23T16:45:52,711][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  3147. [2019-05-23T16:45:52,711][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing resolved transport addresses [10.173.148.65:9300, 10.173.148.73:9300, 10.173.148.58:9300, 10.173.148.50:9300, 10.173.148.67:9300]
  3148. [2019-05-23T16:45:52,711][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  3149. [2019-05-23T16:45:52,711][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  3150. [2019-05-23T16:45:52,711][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  3151. [2019-05-23T16:45:52,711][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.50:9300]] opening probe connection
  3152. [2019-05-23T16:45:52,712][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.58:9300]] opening probe connection
  3153. [2019-05-23T16:45:52,712][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.67:9300]] opening probe connection
  3154. [2019-05-23T16:45:52,712][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  3155. [2019-05-23T16:45:52,713][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  3156. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.58:9300] connect_exception
  3157. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  3158. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  3159. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  3160. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  3161. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  3162. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  3163. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  3164. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  3165. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  3166. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  3167. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  3168. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  3169. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  3170. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  3171. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  3172. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  3173. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  3174. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  3175. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  3176. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  3177. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  3178. at java.lang.Thread.run(Thread.java:835) [?:?]
  3179. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap04.est1933.com/10.173.148.58:9300
  3180. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  3181. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  3182. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  3183. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  3184. ... 6 more
  3185. Caused by: java.net.ConnectException: Connection refused
  3186. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  3187. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  3188. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  3189. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  3190. ... 6 more
  3191. [2019-05-23T16:45:52,715][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  3192. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.50:9300] connect_exception
  3193. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  3194. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  3195. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  3196. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  3197. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  3198. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  3199. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  3200. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  3201. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  3202. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  3203. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  3204. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  3205. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  3206. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  3207. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  3208. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  3209. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  3210. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  3211. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  3212. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  3213. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  3214. at java.lang.Thread.run(Thread.java:835) [?:?]
  3215. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap05.est1933.com/10.173.148.50:9300
  3216. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  3217. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  3218. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  3219. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  3220. ... 6 more
  3221. Caused by: java.net.ConnectException: Connection refused
  3222. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  3223. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  3224. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  3225. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  3226. ... 6 more
  3227. [2019-05-23T16:45:52,713][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  3228. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.67:9300] connect_exception
  3229. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  3230. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  3231. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  3232. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  3233. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  3234. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  3235. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  3236. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  3237. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  3238. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  3239. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  3240. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  3241. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  3242. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  3243. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  3244. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  3245. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  3246. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  3247. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  3248. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  3249. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  3250. at java.lang.Thread.run(Thread.java:835) [?:?]
  3251. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap06.est1933.com/10.173.148.67:9300
  3252. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  3253. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  3254. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  3255. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  3256. ... 6 more
  3257. Caused by: java.net.ConnectException: Connection refused
  3258. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  3259. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  3260. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  3261. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  3262. ... 6 more
  3263. [2019-05-23T16:45:52,839][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  3264. [2019-05-23T16:45:53,607][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  3265. [2019-05-23T16:45:53,710][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  3266. [2019-05-23T16:45:53,710][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=false} requesting peers
  3267. [2019-05-23T16:45:53,711][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing master nodes from cluster state: nodes:
  3268. {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, xpack.installed=true, ml.max_open_jobs=20}, local
  3269.  
  3270. [2019-05-23T16:45:53,711][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  3271. [2019-05-23T16:45:53,712][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap01.est1933.com] to [10.173.148.143:9300]
  3272. [2019-05-23T16:45:53,712][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap02.est1933.com] to [10.173.148.65:9300]
  3273. [2019-05-23T16:45:53,712][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap03.est1933.com] to [10.173.148.73:9300]
  3274. [2019-05-23T16:45:53,712][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap04.est1933.com] to [10.173.148.58:9300]
  3275. [2019-05-23T16:45:53,712][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.73:9300, discoveryNode={awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  3276. [2019-05-23T16:45:53,712][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap05.est1933.com] to [10.173.148.50:9300]
  3277. [2019-05-23T16:45:53,713][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  3278. [2019-05-23T16:45:53,713][TRACE][o.e.d.SeedHostsResolver ] [awselsdevlap01.est1933.com-esnode01] resolved host [awselsdevlap06.est1933.com] to [10.173.148.67:9300]
  3279. [2019-05-23T16:45:53,713][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] probing resolved transport addresses [10.173.148.65:9300, 10.173.148.73:9300, 10.173.148.58:9300, 10.173.148.50:9300, 10.173.148.67:9300]
  3280. [2019-05-23T16:45:53,713][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  3281. [2019-05-23T16:45:53,713][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  3282. [2019-05-23T16:45:53,713][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.58:9300]] opening probe connection
  3283. [2019-05-23T16:45:53,713][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} attempting connection
  3284. [2019-05-23T16:45:53,713][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.65:9300, discoveryNode={awselsdevlap02.est1933.com-esnode02}{T_8EuKpaTqWg2oP3TAnAaA}{VzR0zqVzT9-tTqCePts5GA}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{awselsdevlap03.est1933.com-esnode03}{UWgnBPsHQ1aW4xXEZVKyJQ}{nB1UcJ2PQFiEeicwBR7rLw}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap01.est1933.com-esnode01}{YZ6m2ioDQWqi1cNnOteB6w}{r03UVukISCaTKx3tSWoM0g}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}], term=22}
  3285. [2019-05-23T16:45:53,713][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.50:9300]] opening probe connection
  3286. [2019-05-23T16:45:53,713][TRACE][o.e.d.HandshakingTransportAddressConnector] [awselsdevlap01.est1933.com-esnode01] [connectToRemoteMasterNode[10.173.148.67:9300]] opening probe connection
  3287. [2019-05-23T16:45:53,714][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  3288. [2019-05-23T16:45:53,715][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.58:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  3289. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.58:9300] connect_exception
  3290. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  3291. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  3292. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  3293. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  3294. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  3295. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  3296. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  3297. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  3298. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  3299. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  3300. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  3301. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  3302. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  3303. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  3304. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  3305. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  3306. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  3307. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  3308. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  3309. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  3310. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  3311. at java.lang.Thread.run(Thread.java:835) [?:?]
  3312. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap04.est1933.com/10.173.148.58:9300
  3313. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  3314. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  3315. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  3316. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  3317. ... 6 more
  3318. Caused by: java.net.ConnectException: Connection refused
  3319. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  3320. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  3321. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  3322. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  3323. ... 6 more
  3324. [2019-05-23T16:45:53,716][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.50:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  3325. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.50:9300] connect_exception
  3326. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  3327. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  3328. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  3329. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  3330. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  3331. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  3332. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  3333. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  3334. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  3335. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  3336. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  3337. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  3338. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  3339. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  3340. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  3341. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  3342. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  3343. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  3344. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  3345. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  3346. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  3347. at java.lang.Thread.run(Thread.java:835) [?:?]
  3348. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap05.est1933.com/10.173.148.50:9300
  3349. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  3350. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  3351. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  3352. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  3353. ... 6 more
  3354. Caused by: java.net.ConnectException: Connection refused
  3355. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  3356. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  3357. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  3358. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  3359. ... 6 more
  3360. [2019-05-23T16:45:53,717][DEBUG][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] Peer{transportAddress=10.173.148.67:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
  3361. org.elasticsearch.transport.ConnectTransportException: [][10.173.148.67:9300] connect_exception
  3362. at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1299) ~[elasticsearch-7.1.0.jar:7.1.0]
  3363. at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:99) ~[elasticsearch-7.1.0.jar:7.1.0]
  3364. at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  3365. at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
  3366. at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
  3367. at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
  3368. at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
  3369. at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-7.1.0.jar:7.1.0]
  3370. at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
  3371. at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
  3372. at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
  3373. at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
  3374. at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
  3375. at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
  3376. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
  3377. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
  3378. at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
  3379. at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
  3380. at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
  3381. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
  3382. at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
  3383. at java.lang.Thread.run(Thread.java:835) [?:?]
  3384. Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: awselsdevlap06.est1933.com/10.173.148.67:9300
  3385. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  3386. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  3387. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  3388. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  3389. ... 6 more
  3390. Caused by: java.net.ConnectException: Connection refused
  3391. at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
  3392. at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
  3393. at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
  3394. at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
  3395. ... 6 more
  3396. [2019-05-23T16:45:53,728][INFO ][o.e.n.Node ] [awselsdevlap01.est1933.com-esnode01] stopping ...
  3397. [2019-05-23T16:45:53,730][INFO ][o.e.x.w.WatcherService ] [awselsdevlap01.est1933.com-esnode01] stopping watch service, reason [shutdown initiated]
  3398. [2019-05-23T16:45:53,839][TRACE][o.e.d.PeerFinder ] [awselsdevlap01.est1933.com-esnode01] startProbe(10.173.148.143:9300) not probing local node
  3399. [2019-05-23T16:45:54,103][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [awselsdevlap01.est1933.com-esnode01] [controller/21283] [Main.cc@148] Ml controller exiting
  3400. [2019-05-23T16:45:54,104][INFO ][o.e.x.m.p.NativeController] [awselsdevlap01.est1933.com-esnode01] Native controller process has stopped - no new native processes can be started
  3401. [2019-05-23T16:45:54,124][INFO ][o.e.h.AbstractHttpServerTransport] [awselsdevlap01.est1933.com-esnode01] publish_address {10.173.148.143:9200}, bound_addresses {10.173.148.143:9200}
  3402. [2019-05-23T16:45:54,124][INFO ][o.e.n.Node ] [awselsdevlap01.est1933.com-esnode01] started
  3403. [2019-05-23T16:45:54,126][INFO ][o.e.n.Node ] [awselsdevlap01.est1933.com-esnode01] stopped
  3404. [2019-05-23T16:45:54,126][INFO ][o.e.n.Node ] [awselsdevlap01.est1933.com-esnode01] closing ...
  3405. [2019-05-23T16:45:54,147][INFO ][o.e.n.Node ] [awselsdevlap01.est1933.com-esnode01] closed
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!