API tools faq
paste
Guest User

Untitled

a guest
Feb 12th, 2020
322
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 15.90 KB | None | 0 0
raw download clone embed print report
  1. [2019-11-25T00:21:22,134][INFO ][o.e.m.j.JvmGcMonitorService] [austorelasticdata02.production.solutetech.net] [gc][897889] overhead, spent [417ms] collecting in the last [1s]
  2. [2019-11-25T01:28:56,332][INFO ][o.e.m.j.JvmGcMonitorService] [austorelasticdata02.production.solutetech.net] [gc][901918] overhead, spent [503ms] collecting in the last [1s]
  3. [2019-11-25T02:37:38,646][WARN ][o.e.m.j.JvmGcMonitorService] [austorelasticdata02.production.solutetech.net] [gc][906014] overhead, spent [515ms] collecting in the last [1s]
  4. [2019-11-25T04:48:52,470][INFO ][o.e.m.j.JvmGcMonitorService] [austorelasticdata02.production.solutetech.net] [gc][913840] overhead, spent [738ms] collecting in the last [1.5s]
  5. [2019-11-25T05:59:19,783][WARN ][o.e.m.j.JvmGcMonitorService] [austorelasticdata02.production.solutetech.net] [gc][918042] overhead, spent [562ms] collecting in the last [1s]
  6. [2019-11-25T07:57:30,991][WARN ][o.e.m.j.JvmGcMonitorService] [austorelasticdata02.production.solutetech.net] [gc][925090] overhead, spent [580ms] collecting in the last [1s]
  7. [2019-11-25T08:09:43,693][WARN ][o.e.i.f.SyncedFlushService] [austorelasticdata02.production.solutetech.net] [current_de_offer][1] can't to issue sync id [AW6hm_gNcLyQnSBsH9DL] for out of sync replica [[current_de_offer][1], node[pnogo7NXRkCyNE-AaWI_Mw], [R], s[STARTED], a[id=c2vCfCj8TtCf304lW6pUvg]] with num docs [56184433]; num docs on primary [56184436]
  8. [2019-11-25T08:09:43,694][WARN ][o.e.i.f.SyncedFlushService] [austorelasticdata02.production.solutetech.net] [current_de_offer][1] can't to issue sync id [AW6hm_gNcLyQnSBsH9DL] for out of sync replica [[current_de_offer][1], node[gnD0T30ZSWKvuXECBdujzw], [R], s[STARTED], a[id=0YkbnsPrS4mZII0y9wfFnw]] with num docs [56184433]; num docs on primary [56184436]
  9. [2019-11-25T08:09:44,203][WARN ][o.e.i.f.SyncedFlushService] [austorelasticdata02.production.solutetech.net] [current_de_offer][0] can't to issue sync id [AW6hm_oLcLyQnSBsH9uf] for out of sync replica [[current_de_offer][0], node[gnD0T30ZSWKvuXECBdujzw], [R], s[STARTED], a[id=u4vstyxcTdOr4Z1EFpPerg]] with num docs [56150525]; num docs on primary [56150516]
  10. [2019-11-25T08:09:44,321][WARN ][o.e.i.f.SyncedFlushService] [austorelasticdata02.production.solutetech.net] [delta_de_offer_2019-11-25-08-03-34-029262][1] can't to issue sync id [AW6hm_qBcLyQnSBsH9zm] for out of sync replica [[delta_de_offer_2019-11-25-08-03-34-029262][1], node[Omy7F981SauIEiRwuaPWnQ], [R], s[STARTED], a[id=c0YtFcNjQbuFnPsWaedjjA]] with num docs [66050]; num docs on primary [66054]
  11. [2019-11-25T08:09:44,442][WARN ][o.e.i.f.SyncedFlushService] [austorelasticdata02.production.solutetech.net] [delta_de_offer_2019-11-25-08-03-34-029262][0] can't to issue sync id [AW6hm_r6cLyQnSBsH99d] for out of sync replica [[delta_de_offer_2019-11-25-08-03-34-029262][0], node[rZ_TNyvMQ32tpVTfINv7Zg], [R], s[STARTED], a[id=yQ1ZbjDgSiSdWcFZo6krFw]] with num docs [66136]; num docs on primary [66124]
  12. [2019-11-25T08:09:44,555][WARN ][o.e.i.f.SyncedFlushService] [austorelasticdata02.production.solutetech.net] [current_de_offer][2] can't to issue sync id [AW6hm_trcLyQnSBsH-GK] for out of sync replica [[current_de_offer][2], node[rZ_TNyvMQ32tpVTfINv7Zg], [R], s[STARTED], a[id=Pjnudf7SQc-MWBkZx2LOsQ]] with num docs [56166037]; num docs on primary [56166019]
  13. [2019-11-25T08:09:59,518][INFO ][o.e.n.Node ] [austorelasticdata02.production.solutetech.net] stopping ...
  14. [2019-11-25T08:10:06,783][INFO ][o.e.n.Node ] [austorelasticdata02.production.solutetech.net] stopped
  15. [2019-11-25T08:10:06,783][INFO ][o.e.n.Node ] [austorelasticdata02.production.solutetech.net] closing ...
  16. [2019-11-25T08:10:06,827][INFO ][o.e.n.Node ] [austorelasticdata02.production.solutetech.net] closed
  17. [2019-11-25T08:10:28,304][INFO ][o.e.n.Node ] [austorelasticdata02.production.solutetech.net] initializing ...
  18. [2019-11-25T08:10:28,474][INFO ][o.e.e.NodeEnvironment ] [austorelasticdata02.production.solutetech.net] using [2] data paths, mounts [[/var/lib/elasticsearch2 (/dev/sdb1), /var/lib/elasticsearch (/dev/sdc1)]], net usable_space [1.1tb], net total_space [1.7tb], spins? [no], types [xfs]
  19. [2019-11-25T08:10:28,475][INFO ][o.e.e.NodeEnvironment ] [austorelasticdata02.production.solutetech.net] heap size [29.1gb], compressed ordinary object pointers [true]
  20. [2019-11-25T08:10:36,860][INFO ][o.e.n.Node ] [austorelasticdata02.production.solutetech.net] node name [austorelasticdata02.production.solutetech.net], node ID [SvCz-jFMR1a8TU2INDwD8g]
  21. [2019-11-25T08:10:36,861][INFO ][o.e.n.Node ] [austorelasticdata02.production.solutetech.net] version[5.6.16], pid[25942], build[3a740d1/2019-03-13T15:33:36.565Z], OS[Linux/3.16.0-10-amd64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_111/25.111-b14]
  22. [2019-11-25T08:10:36,861][INFO ][o.e.n.Node ] [austorelasticdata02.production.solutetech.net] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Xms30000M, -Xmx30000M, -Des.path.home=/usr/share/elasticsearch]
  23. [2019-11-25T08:10:37,601][INFO ][o.e.p.PluginsService ] [austorelasticdata02.production.solutetech.net] loaded module [aggs-matrix-stats]
  24. [2019-11-25T08:10:37,601][INFO ][o.e.p.PluginsService ] [austorelasticdata02.production.solutetech.net] loaded module [ingest-common]
  25. [2019-11-25T08:10:37,601][INFO ][o.e.p.PluginsService ] [austorelasticdata02.production.solutetech.net] loaded module [lang-expression]
  26. [2019-11-25T08:10:37,602][INFO ][o.e.p.PluginsService ] [austorelasticdata02.production.solutetech.net] loaded module [lang-groovy]
  27. [2019-11-25T08:10:37,602][INFO ][o.e.p.PluginsService ] [austorelasticdata02.production.solutetech.net] loaded module [lang-mustache]
  28. [2019-11-25T08:10:37,602][INFO ][o.e.p.PluginsService ] [austorelasticdata02.production.solutetech.net] loaded module [lang-painless]
  29. [2019-11-25T08:10:37,602][INFO ][o.e.p.PluginsService ] [austorelasticdata02.production.solutetech.net] loaded module [parent-join]
  30. [2019-11-25T08:10:37,602][INFO ][o.e.p.PluginsService ] [austorelasticdata02.production.solutetech.net] loaded module [percolator]
  31. [2019-11-25T08:10:37,602][INFO ][o.e.p.PluginsService ] [austorelasticdata02.production.solutetech.net] loaded module [reindex]
  32. [2019-11-25T08:10:37,602][INFO ][o.e.p.PluginsService ] [austorelasticdata02.production.solutetech.net] loaded module [transport-netty3]
  33. [2019-11-25T08:10:37,602][INFO ][o.e.p.PluginsService ] [austorelasticdata02.production.solutetech.net] loaded module [transport-netty4]
  34. [2019-11-25T08:10:37,602][INFO ][o.e.p.PluginsService ] [austorelasticdata02.production.solutetech.net] no plugins loaded
  35. [2019-11-25T08:10:39,878][INFO ][o.e.d.DiscoveryModule ] [austorelasticdata02.production.solutetech.net] using discovery type [zen]
  36. [2019-11-25T08:11:10,971][INFO ][o.e.n.Node ] [austorelasticdata02.production.solutetech.net] initialized
  37. [2019-11-25T08:11:10,971][INFO ][o.e.n.Node ] [austorelasticdata02.production.solutetech.net] starting ...
  38. [2019-11-25T08:11:11,252][INFO ][o.e.t.TransportService ] [austorelasticdata02.production.solutetech.net] publish_address {172.16.161.28:9300}, bound_addresses {[::]:9300}
  39. [2019-11-25T08:11:11,270][INFO ][o.e.b.BootstrapChecks ] [austorelasticdata02.production.solutetech.net] bound or publishing to a non-loopback address, enforcing bootstrap checks
  40. [2019-11-25T08:11:33,637][INFO ][o.e.c.s.ClusterService ] [austorelasticdata02.production.solutetech.net] detected_master {austorelasticmaster03.production.solutetech.net}{67vB6G0dQtGDKHs6cQI0Lw}{EwD5bHhLQT-7G1eItAz_Cw}{172.16.161.47}{172.16.161.47:9300}, added {{austorelasticdata04.production.solutetech.net}{pnogo7NXRkCyNE-AaWI_Mw}{BNYzbm2ZTcy3P1Wt7cwkxw}{172.16.162.205}{172.16.162.205:9300},{austorelasticdata01.production.solutetech.net}{eBm2LzatRpaiQihNPFhk5Q}{NuDRonMCTjCQY9hRPBNTiA}{172.16.161.23}{172.16.161.23:9300},{austorelasticmaster03.production.solutetech.net}{67vB6G0dQtGDKHs6cQI0Lw}{EwD5bHhLQT-7G1eItAz_Cw}{172.16.161.47}{172.16.161.47:9300},{austorelasticmaster01.production.solutetech.net}{65i1u-x7Qsy0s3PbUTUqBQ}{3X8oepxHQ6u8P_gG5vSh_w}{172.16.161.42}{172.16.161.42:9300},{austorelasticdata03.production.solutetech.net}{rZ_TNyvMQ32tpVTfINv7Zg}{6AGT2hflQpCdaNNvXK2C9Q}{172.16.161.25}{172.16.161.25:9300},{austorelasticmaster02.production.solutetech.net}{pFqA4PBTS4632jOTRlfRmg}{2bOh3VUdQp6igd4LU7kJoQ}{172.16.161.45}{172.16.161.45:9300},{austorelasticdata05.production.solutetech.net}{JbHVX0sWTHO0XeA_4QKxwg}{bl1LDHxSTjOW7XK9edSSKg}{172.16.162.206}{172.16.162.206:9300},{austorelasticdata07.production.solutetech.net}{Omy7F981SauIEiRwuaPWnQ}{IN4bFsjWRL6Sm_8yrbvGOw}{172.16.163.45}{172.16.163.45:9300},{austorelasticdata06.production.solutetech.net}{gnD0T30ZSWKvuXECBdujzw}{eLTtFhvgT0yT9_HW2HTQaw}{172.16.162.252}{172.16.162.252:9300},{austorelasticapi01.production.solutetech.net}{Z9IZUZFRTvy7nAAGUx7eWg}{lvSubrsbSg-vd66HJNOgGg}{172.16.161.24}{172.16.161.24:9300},{austorelasticapi02.production.solutetech.net}{1RO4OkkUS5iq91j-2tkS4Q}{dxcw7r1jSh6Ow5mIayQ7Qw}{172.16.161.27}{172.16.161.27:9300},}, reason: zen-disco-receive(from master [master {austorelasticmaster03.production.solutetech.net}{67vB6G0dQtGDKHs6cQI0Lw}{EwD5bHhLQT-7G1eItAz_Cw}{172.16.161.47}{172.16.161.47:9300} committed version [3944566]])
  41. [2019-11-25T08:11:33,829][INFO ][o.e.c.s.ClusterSettings ] [austorelasticdata02.production.solutetech.net] updating [cluster.routing.allocation.cluster_concurrent_rebalance] from [2] to [6]
  42. [2019-11-25T08:11:33,829][INFO ][o.e.c.s.ClusterSettings ] [austorelasticdata02.production.solutetech.net] updating [cluster.routing.allocation.enable] from [ALL] to [all]
  43. [2019-11-25T08:11:33,830][INFO ][o.e.c.s.ClusterSettings ] [austorelasticdata02.production.solutetech.net] updating [cluster.routing.allocation.exclude.] from [{}] to [{"_id":"","_host":"","_name":"","_ip":""}]
  44. [2019-11-25T08:11:33,830][INFO ][o.e.c.s.ClusterSettings ] [austorelasticdata02.production.solutetech.net] updating [cluster.routing.allocation.node_concurrent_incoming_recoveries] from [2] to [6]
  45. [2019-11-25T08:11:33,830][INFO ][o.e.c.s.ClusterSettings ] [austorelasticdata02.production.solutetech.net] updating [cluster.routing.allocation.node_concurrent_outgoing_recoveries] from [2] to [6]
  46. [2019-11-25T08:11:33,830][INFO ][o.e.c.s.ClusterSettings ] [austorelasticdata02.production.solutetech.net] updating [indices.recovery.max_bytes_per_sec] from [40mb] to [100mb]
  47. [2019-11-25T08:11:35,080][WARN ][o.e.m.j.JvmGcMonitorService] [austorelasticdata02.production.solutetech.net] [gc][24] overhead, spent [582ms] collecting in the last [1s]
  48. [2019-11-25T08:11:37,513][INFO ][o.e.h.n.Netty4HttpServerTransport] [austorelasticdata02.production.solutetech.net] publish_address {172.16.161.28:9200}, bound_addresses {[::]:9200}
  49. [2019-11-25T08:11:37,514][INFO ][o.e.n.Node ] [austorelasticdata02.production.solutetech.net] started
  50. [2019-11-25T10:28:38,024][WARN ][o.e.c.s.ClusterService ] [austorelasticdata02.production.solutetech.net] cluster state update task [zen-disco-receive(from master [master {austorelasticmaster03.production.solutetech.net}{67vB6G0dQtGDKHs6cQI0Lw}{EwD5bHhLQT-7G1eItAz_Cw}{172.16.161.47}{172.16.161.47:9300} committed version [3945763]])] took [37.8s] above the warn threshold of 30s
  51. [2019-11-25T10:59:54,054][WARN ][o.e.t.n.Netty4Transport ] [austorelasticdata02.production.solutetech.net] write and flush on the network layer failed (channel: [id: 0x261af416, L:/172.16.161.28:9300 ! R:/172.16.161.25:39892]) java.nio.channels.ClosedChannelException
  52. at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
  53. [2019-11-25T10:59:54,054][WARN ][o.e.t.n.Netty4Transport ] [austorelasticdata02.production.solutetech.net] write and flush on the network layer failed (channel: [id: 0x02088b18, L:/172.16.161.28:9300 ! R:/172.16.161.25:39889]) java.nio.channels.ClosedChannelException
  54. at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
  55. [2019-11-25T10:59:54,054][WARN ][o.e.t.n.Netty4Transport ] [austorelasticdata02.production.solutetech.net] write and flush on the network layer failed (channel: [id: 0x0bfae011, L:/172.16.161.28:9300 ! R:/172.16.161.25:39893]) java.nio.channels.ClosedChannelException
  56. at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
  57. [2019-11-25T10:59:55,827][WARN ][o.e.c.NodeConnectionsService] [austorelasticdata02.production.solutetech.net] failed to connect to node {austorelasticdata03.production.solutetech.net}{rZ_TNyvMQ32tpVTfINv7Zg}{6AGT2hflQpCdaNNvXK2C9Q}{172.16.161.25}{172.16.161.25:9300} (tried [1] times) ConnectTransportException[[austorelasticdata03.production.solutetech.net][172.16.161.25:9300] connect_timeout[30s]]; nested: AnnotatedConnectException[Connection refused: 172.16.161.25/172.16.161.25:9300]; nested: ConnectException[Connection refused];
  58. at org.elasticsearch.transport.netty4.Netty4Transport.connectToChannels(Netty4Transport.java:363)
  59. [2019-11-25T11:00:02,362][INFO ][o.e.c.s.ClusterService ] [austorelasticdata02.production.solutetech.net] removed {{austorelasticdata03.production.solutetech.net}{rZ_TNyvMQ32tpVTfINv7Zg}{6AGT2hflQpCdaNNvXK2C9Q}{172.16.161.25}{172.16.161.25:9300},}, reason: zen-disco-receive(from master [master {austorelasticmaster03.production.solutetech.net}{67vB6G0dQtGDKHs6cQI0Lw}{EwD5bHhLQT-7G1eItAz_Cw}{172.16.161.47}{172.16.161.47:9300} committed version [3946026]])
  60. [2019-11-25T11:01:51,214][INFO ][o.e.c.s.ClusterService ] [austorelasticdata02.production.solutetech.net] added {{austorelasticdata03.production.solutetech.net}{rZ_TNyvMQ32tpVTfINv7Zg}{Z0gt6TWNTuai77O3SUi3Eg}{172.16.161.25}{172.16.161.25:9300},}, reason: zen-disco-receive(from master [master {austorelasticmaster03.production.solutetech.net}{67vB6G0dQtGDKHs6cQI0Lw}{EwD5bHhLQT-7G1eItAz_Cw}{172.16.161.47}{172.16.161.47:9300} committed version [3946034]])
  61. [2019-11-25T13:19:22,009][INFO ][o.e.m.j.JvmGcMonitorService] [austorelasticdata02.production.solutetech.net] [gc][18409] overhead, spent [296ms] collecting in the last [1s]
  62. [2019-11-25T13:27:03,018][INFO ][o.e.m.j.JvmGcMonitorService] [austorelasticdata02.production.solutetech.net] [gc][18868] overhead, spent [268ms] collecting in the last [1s]
  63. [2019-11-25T13:32:05,026][INFO ][o.e.m.j.JvmGcMonitorService] [austorelasticdata02.production.solutetech.net] [gc][19168] overhead, spent [417ms] collecting in the last [1.3s]
  64. [2019-11-25T14:25:46,700][INFO ][o.e.m.j.JvmGcMonitorService] [austorelasticdata02.production.solutetech.net] [gc][22375] overhead, spent [326ms] collecting in the last [1.1s]
  65. [2019-11-25T14:52:20,760][INFO ][o.e.m.j.JvmGcMonitorService] [austorelasticdata02.production.solutetech.net] [gc][23962] overhead, spent [351ms] collecting in the last [1s]
  66. [2019-11-25T15:31:11,152][INFO ][o.e.m.j.JvmGcMonitorService] [austorelasticdata02.production.solutetech.net] [gc][26282] overhead, spent [302ms] collecting in the last [1s]
  67. [2019-11-25T18:01:46,913][INFO ][o.e.m.j.JvmGcMonitorService] [austorelasticdata02.production.solutetech.net] [gc][35262] overhead, spent [414ms] collecting in the last [1s]
  68. [2019-11-25T20:03:10,580][INFO ][o.e.m.j.JvmGcMonitorService] [austorelasticdata02.production.solutetech.net] [gc][42503] overhead, spent [468ms] collecting in the last [1s]
  69. [2019-11-25T21:34:01,285][INFO ][o.e.m.j.JvmGcMonitorService] [austorelasticdata02.production.solutetech.net] [gc][47929] overhead, spent [565ms] collecting in the last [1.4s]
  70. [2019-11-25T22:48:59,136][INFO ][o.e.m.j.JvmGcMonitorService] [austorelasticdata02.production.solutetech.net] [gc][52406] overhead, spent [527ms] collecting in the last [1s]
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!