Advertisement
Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- [2021-04-22T17:38:45,148][INFO ][o.e.m.j.JvmGcMonitorService] [es03] [gc][178796] overhead, spent [330ms] collecting in the last [1s]
- [2021-04-22T17:38:50,208][WARN ][o.e.t.OutboundHandler ] [es03] sending transport message [MessageSerializer{Response{23789998}{false}{false}{false}{class org.elasticsearch.search.fetch.QueryFetchSearchResult}}] of size [1741433129] on [Netty4TcpChannel{localAddress=/[IP].157:9300, remoteAddress=/[IP].156:60636, profile=default}] took [5404ms] which is above the warn threshold of [5000ms]
- [2021-04-22T17:38:50,208][WARN ][o.e.t.OutboundHandler ] [es03] sending transport message [MessageSerializer{Response{23790257}{false}{false}{false}{class org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicaResponse}}] of size [119] on [Netty4TcpChannel{localAddress=/[IP].157:9300, remoteAddress=/[IP].156:60636, profile=default}] took [5204ms] which is above the warn threshold of [5000ms]
- [2021-04-22T17:41:36,888][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [es03] attempting to trigger G1GC due to high heap usage [10216922360]
- [2021-04-22T17:41:36,908][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [es03] GC did not bring memory usage down, before [10216922360], after [10396944528], allocations [42], duration [19]
- [2021-04-22T17:41:37,718][WARN ][o.e.m.j.JvmGcMonitorService] [es03] [gc][178968] overhead, spent [937ms] collecting in the last [1.5s]
- [2021-04-22T17:41:38,984][WARN ][o.e.m.j.JvmGcMonitorService] [es03] [gc][178969] overhead, spent [1.2s] collecting in the last [1.2s]
- [2021-04-22T17:41:40,021][WARN ][o.e.m.j.JvmGcMonitorService] [es03] [gc][178970] overhead, spent [1s] collecting in the last [1s]
- [2021-04-22T17:41:43,427][WARN ][o.e.m.j.JvmGcMonitorService] [es03] [gc][178971] overhead, spent [3.3s] collecting in the last [3.4s]
- [2021-04-22T17:41:44,438][WARN ][o.e.t.InboundHandler ] [es03] handling inbound transport message [InboundMessage{Header{1765853363}{7.12.0}{3464850}{false}{false}{false}{false}{NO_ACTION_NAME_FOR_RESPONSES}}] took [7673ms] which is above the warn threshold of [5000ms]
- [2021-04-22T17:41:44,443][ERROR][o.e.ExceptionsHelper ] [es03] fatal error
- at org.elasticsearch.ExceptionsHelper.lambda$maybeDieOnAnotherThread$4(ExceptionsHelper.java:283)
- at java.base/java.util.Optional.ifPresent(Optional.java:176)
- at org.elasticsearch.ExceptionsHelper.maybeDieOnAnotherThread(ExceptionsHelper.java:273)
- at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.exceptionCaught(Netty4MessageChannelHandler.java:70)
- at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:302)
- at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:381)
- at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
- at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
- at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271)
- at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
- at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
- at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
- at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
- at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
- at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
- at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
- at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
- at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
- at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615)
- at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578)
- at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
- at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
- at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
- at java.base/java.lang.Thread.run(Thread.java:832)
- [2021-04-22T17:41:44,442][WARN ][o.e.t.InboundHandler ] [es03] handling inbound transport message [InboundMessage{Header{1765853364}{7.12.0}{3464870}{false}{false}{false}{false}{NO_ACTION_NAME_FOR_RESPONSES}}] took [8188ms] which is above the warn threshold of [5000ms]
- [2021-04-22T17:41:44,450][WARN ][o.e.t.OutboundHandler ] [es03] sending transport message [MessageSerializer{Request{internal:coordination/fault_detection/leader_check}{3465028}{false}{false}{false}}] of size [544] on [Netty4TcpChannel{localAddress=/[IP].157:42042, remoteAddress=[IP].131/[IP].131:9300, profile=default}] took [7768ms] which is above the warn threshold of [5000ms]
- [2021-04-22T17:41:44,450][WARN ][o.e.t.OutboundHandler ] [es03] sending transport message [MessageSerializer{Request{indices:data/write/bulk[s]}{3465030}{false}{false}{false}}] of size [2072] on [Netty4TcpChannel{localAddress=/[IP].157:37364, remoteAddress=[IP].156/[IP].156:9300, profile=default}] took [6714ms] which is above the warn threshold of [5000ms]
- [2021-04-22T17:41:44,450][WARN ][o.e.t.OutboundHandler ] [es03] sending transport message [MessageSerializer{Request{indices:data/write/bulk[s]}{3465033}{false}{false}{false}}] of size [1284] on [Netty4TcpChannel{localAddress=/[IP].157:37364, remoteAddress=[IP].156/[IP].156:9300, profile=default}] took [6714ms] which is above the warn threshold of [5000ms]
- [2021-04-22T17:41:44,450][WARN ][o.e.t.OutboundHandler ] [es03] sending transport message [MessageSerializer{Request{indices:data/write/bulk[s]}{3465036}{false}{false}{false}}] of size [1215] on [Netty4TcpChannel{localAddress=/[IP].157:37364, remoteAddress=[IP].156/[IP].156:9300, profile=default}] took [6714ms] which is above the warn threshold of [5000ms]
- [2021-04-22T17:41:44,450][WARN ][o.e.t.OutboundHandler ] [es03] sending transport message [MessageSerializer{Request{indices:data/write/bulk[s]}{3465039}{false}{false}{false}}] of size [1299] on [Netty4TcpChannel{localAddress=/[IP].157:37364, remoteAddress=[IP].156/[IP].156:9300, profile=default}] took [6714ms] which is above the warn threshold of [5000ms]
- [2021-04-22T17:41:44,445][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [es03] fatal error in thread [Thread-11624], exiting
- java.lang.OutOfMemoryError: Java heap space
- at org.elasticsearch.common.io.stream.StreamInput.readBytesReference(StreamInput.java:161) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.common.io.stream.StreamInput.readBytesReference(StreamInput.java:127) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.search.SearchHit.<init>(SearchHit.java:145) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.search.SearchHits.<init>(SearchHits.java:84) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.search.fetch.FetchSearchResult.<init>(FetchSearchResult.java:34) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.search.fetch.QueryFetchSearchResult.<init>(QueryFetchSearchResult.java:28) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.action.search.SearchTransportService$$Lambda$5799/0x0000000801948670.read(Unknown Source) ~[?:?]
- at org.elasticsearch.action.ActionListenerResponseHandler.read(ActionListenerResponseHandler.java:58) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.action.ActionListenerResponseHandler.read(ActionListenerResponseHandler.java:25) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.transport.TransportService$6.read(TransportService.java:755) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.transport.TransportService$6.read(TransportService.java:735) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.read(TransportService.java:1271) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.read(TransportService.java:1258) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.transport.InboundHandler.handleResponse(InboundHandler.java:264) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:128) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:84) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:690) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler$$Lambda$5294/0x000000080176cf38.accept(Unknown Source) ~[?:?]
- at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:131) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:106) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:71) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:63) ~[?:?]
- at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
- at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
- at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
- at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
- at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
- at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
- at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
- at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
- at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
- at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
- [2021-04-22T17:41:44,440][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [es03] fatal error in thread [elasticsearch[es03][refresh][T#1]], exiting
- java.lang.OutOfMemoryError: Java heap space
- at org.apache.lucene.util.compress.LZ4$HighCompressionHashTable.<init>(LZ4.java:309) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.<init>(BlockTreeTermsWriter.java:1028) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:308) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:170) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:120) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:264) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:350) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:476) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:656) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:605) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:290) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:265) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:255) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:55) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:29) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:357) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:339) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:225) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1685) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.index.engine.InternalEngine.maybeRefresh(InternalEngine.java:1664) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.index.shard.IndexShard.scheduledRefresh(IndexShard.java:3459) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.index.IndexService.maybeRefreshEngine(IndexService.java:872) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.index.IndexService.access$200(IndexService.java:101) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:1004) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.common.util.concurrent.AbstractAsyncTask.run(AbstractAsyncTask.java:133) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:673) ~[elasticsearch-7.12.0.jar:7.12.0]
- at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ~[?:?]
- [2021-04-22T17:41:44,455][ERROR][o.e.i.e.Engine ] [es03] [metricbeat-7.12.0-2021.04.04-000001][0] already closed by tragic event on the index writer
- java.lang.OutOfMemoryError: Java heap space
- at org.apache.lucene.util.compress.LZ4$HighCompressionHashTable.<init>(LZ4.java:309) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.<init>(BlockTreeTermsWriter.java:1028) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:308) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:170) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:120) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:264) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:350) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:476) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:656) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:605) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:290) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:265) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:255) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:55) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:29) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:357) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:339) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:225) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1685) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.index.engine.InternalEngine.maybeRefresh(InternalEngine.java:1664) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.index.shard.IndexShard.scheduledRefresh(IndexShard.java:3459) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.index.IndexService.maybeRefreshEngine(IndexService.java:872) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.index.IndexService.access$200(IndexService.java:101) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:1004) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.common.util.concurrent.AbstractAsyncTask.run(AbstractAsyncTask.java:133) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:673) ~[elasticsearch-7.12.0.jar:7.12.0]
- at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ~[?:?]
- [2021-04-22T17:41:44,465][WARN ][o.e.t.TcpTransport ] [es03] exception caught on transport layer [Netty4TcpChannel{localAddress=/[IP].157:37400, remoteAddress=[IP].156/[IP].156:9300, profile=default}], closing connection
- java.lang.Exception: java.lang.OutOfMemoryError: Java heap space
- at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.exceptionCaught(Netty4MessageChannelHandler.java:75) [transport-netty4-client-7.12.0.jar:7.12.0]
- at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:302) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
- at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:381) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
- at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
- at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
- at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
- at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
- at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
- at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
- at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
- at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
- at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
- at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
- at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
- at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
- at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
- at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
- at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
- at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
- at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
- at java.lang.Thread.run(Thread.java:832) [?:?]
- Caused by: java.lang.OutOfMemoryError: Java heap space
- at org.elasticsearch.common.io.stream.StreamInput.readBytesReference(StreamInput.java:161) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.common.io.stream.StreamInput.readBytesReference(StreamInput.java:127) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.search.SearchHit.<init>(SearchHit.java:145) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.search.SearchHits.<init>(SearchHits.java:84) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.search.fetch.FetchSearchResult.<init>(FetchSearchResult.java:34) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.search.fetch.QueryFetchSearchResult.<init>(QueryFetchSearchResult.java:28) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.action.search.SearchTransportService$$Lambda$5799/0x0000000801948670.read(Unknown Source) ~[?:?]
- at org.elasticsearch.action.ActionListenerResponseHandler.read(ActionListenerResponseHandler.java:58) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.action.ActionListenerResponseHandler.read(ActionListenerResponseHandler.java:25) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.transport.TransportService$6.read(TransportService.java:755) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.transport.TransportService$6.read(TransportService.java:735) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.read(TransportService.java:1271) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.read(TransportService.java:1258) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.transport.InboundHandler.handleResponse(InboundHandler.java:264) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:128) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:84) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:690) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler$$Lambda$5294/0x000000080176cf38.accept(Unknown Source) ~[?:?]
- at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:131) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:106) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:71) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:63) ~[?:?]
- at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
- at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
- at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
- at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
- at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
- at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
- at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
- at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
- at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
- at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
- [2021-04-22T17:41:44,473][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [es03] fatal error in thread [elasticsearch[es03][write][T#1]], exiting
- java.lang.OutOfMemoryError: Java heap space
- at org.apache.lucene.util.compress.LZ4$HighCompressionHashTable.<init>(LZ4.java:309) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.<init>(BlockTreeTermsWriter.java:1028) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:308) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:170) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:120) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:264) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:350) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:476) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:656) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:605) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:290) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:265) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:255) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:55) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:29) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:357) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:339) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:225) ~[lucene-core-8.8.0.jar:8.8.0 b10659f0fc18b58b90929cfdadde94544d202c4a - noble - 2021-01-25 19:07:45]
- at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1685) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.index.engine.InternalEngine.maybeRefresh(InternalEngine.java:1664) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.index.shard.IndexShard.scheduledRefresh(IndexShard.java:3459) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.index.IndexService.maybeRefreshEngine(IndexService.java:872) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.index.IndexService.access$200(IndexService.java:101) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:1004) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.common.util.concurrent.AbstractAsyncTask.run(AbstractAsyncTask.java:133) ~[elasticsearch-7.12.0.jar:7.12.0]
- at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:673) ~[elasticsearch-7.12.0.jar:7.12.0]
- at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ~[?:?]
- [2021-04-22T17:41:44,476][WARN ][o.e.t.OutboundHandler ] [es03] sending transport message [MessageSerializer{Request{indices:data/write/bulk[s][r]}{3465031}{false}{false}{false}}] of size [3743] on [Netty4TcpChannel{localAddress=/[IP].157:37396, remoteAddress=[IP].156/[IP].156:9300, profile=default}] took [6714ms] which is above the warn threshold of [5000ms]
- [2021-04-22T17:41:44,477][WARN ][o.e.t.OutboundHandler ] [es03] sending transport message [MessageSerializer{Request{indices:data/write/bulk[s]}{3465034}{false}{false}{false}}] of size [1275] on [Netty4TcpChannel{localAddress=/[IP].157:37396, remoteAddress=[IP].156/[IP].156:9300, profile=default}] took [6714ms] which is above the warn threshold of [5000ms]
- [2021-04-22T17:41:44,477][WARN ][o.e.t.OutboundHandler ] [es03] sending transport message [MessageSerializer{Request{indices:data/write/bulk[s]}{3465037}{false}{false}{false}}] of size [1233] on [Netty4TcpChannel{localAddress=/[IP].157:37396, remoteAddress=[IP].156/[IP].156:9300, profile=default}] took [6714ms] which is above the warn threshold of [5000ms]
- [2021-04-22T17:41:44,477][WARN ][o.e.t.OutboundHandler ] [es03] sending transport message [MessageSerializer{Request{indices:data/write/bulk[s]}{3465040}{false}{false}{false}}] of size [1251] on [Netty4TcpChannel{localAddress=/[IP].157:37396, remoteAddress=[IP].156/[IP].156:9300, profile=default}] took [5447ms] which is above the warn threshold of [5000ms]
- [2021-04-22T17:47:27,464][INFO ][o.e.n.Node ] [es03] version[7.12.0], pid[21201], build[default/deb/78722783c38caa25a70982b5b042074cde5d3b3a/2021-03-18T06:17:15.410153305Z], OS[Linux/4.15.0-136-generic/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]
- [2021-04-22T17:47:27,466][INFO ][o.e.n.Node ] [es03] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
- [2021-04-22T17:47:27,466][INFO ][o.e.n.Node ] [es03] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -Xms10g, -Xmx10g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-2948471995205504418, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:-HeapDumpOnOutOfMemoryError, -XX:MaxDirectMemorySize=5368709120, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb, -Des.bundled_jdk=true]
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement