API tools faq
paste
Advertisement
Guest User

elasticsearch eating memory

a guest
Dec 20th, 2014
264
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 7.03 KB | None | 0 0
raw download clone embed print report
  1. getting this in my logstash.log:
  2.  
  3.  
  4. log4j, [2014-12-20T19:35:56.752] WARN: org.elasticsearch.discovery.zen.ping.unicast: [logstash-lumberjack.domain.com-1456-4170] failed to send ping to [[#zen_unicast_1#][lumberjack.domain.com][inet[localhost/127.0.0.1:9300]]]
  5. org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[localhost/127.0.0.1:9300]][discovery/zen/unicast] request_id [67141] timed out after [4774ms]
  6. at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:356)
  7. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  8. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  9. at java.lang.Thread.run(Thread.java:745)
  10. log4j, [2014-12-20T19:35:56.757] WARN: org.elasticsearch.discovery.zen.ping.unicast: [logstash-lumberjack.domain.com-1456-4172] failed to send ping to [[#zen_unicast_1#][lumberjack.domain.com][inet[localhost/127.0.0.1:9300]]]
  11. org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[localhost/127.0.0.1:9300]][discovery/zen/unicast] request_id [66681] timed out after [4362ms]
  12. at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:356)
  13. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  14. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  15. at java.lang.Thread.run(Thread.java:745)
  16. log4j, [2014-12-20T19:35:56.755] WARN: org.elasticsearch.discovery.zen.ping.unicast: [logstash-lumberjack.domain.com-1456-4168] failed to send ping to [[#zen_unicast_1#][lumberjack.domain.com][inet[localhost/127.0.0.1:9300]]]
  17. org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[localhost/127.0.0.1:9300]][discovery/zen/unicast] request_id [65942] timed out after [4782ms]
  18. at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:356)
  19. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  20. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  21. at java.lang.Thread.run(Thread.java:745)
  22.  
  23. # ps -ef | grep 'logstash\|elasticsearch'
  24. 498 7443 1 99 19:43 ? 00:07:40 /usr/bin/java -Xms512M -Xmx512M -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.1.2.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
  25. logstash 7518 1 42 19:44 pts/4 00:02:11 /usr/bin/java -Djava.io.tmpdir=/var/lib/logstash -Xmx500m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -jar /opt/logstash/vendor/jar/jruby-complete-1.7.11.jar -I/opt/logstash/lib /opt/logstash/lib/logstash/runner.rb agent -f /etc/logstash/conf.d -l /var/log/logstash/logstash.log
  26.  
  27. this is at the top of my elasticsearch.log
  28.  
  29. [2014-12-20 00:01:35,261][WARN ][netty.channel.DefaultChannelPipeline] An exception was thrown by a user handler while handling an exception event ([id: 0xcc04c9a7, /127.0.0.1:50380 :> /127.0.0.1:9300] EXCEPTION: java.lang.OutOfMemoryError: Java heap space)
  30. java.lang.OutOfMemoryError: Java heap space
  31. [2014-12-20 00:06:25,719][WARN ][index.engine.internal ] [Mop Man] [logstash-2014.10.17][3] failed engine
  32. java.lang.OutOfMemoryError: Java heap space
  33. [2014-12-20 00:06:31,109][WARN ][index.engine.internal ] [Mop Man] [logstash-2014.10.17][3] failed to flush after setting shard to inactive
  34. org.elasticsearch.index.engine.FlushFailedEngineException: [logstash-2014.10.17][3] Flush failed
  35. at org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:797)
  36. at org.elasticsearch.index.engine.internal.InternalEngine.updateIndexingBufferSize(InternalEngine.java:223)
  37. at org.elasticsearch.indices.memory.IndexingMemoryController$ShardsIndicesStatusChecker.run(IndexingMemoryController.java:201)
  38. at org.elasticsearch.threadpool.ThreadPool$LoggingRunnable.run(ThreadPool.java:437)
  39. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  40. at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
  41. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
  42. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
  43. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  44. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  45. at java.lang.Thread.run(Thread.java:745)
  46. Caused by: java.lang.OutOfMemoryError: Java heap space
  47.  
  48. rpm -qa | grep 'logstash\|elasticsearch'
  49. logstash-1.4.2-1_2c0f5a1.noarch
  50. elasticsearch-1.1.2-1.noarch
  51.  
  52. cat /proc/meminfo
  53. MemTotal: 2955676 kB
  54. MemFree: 666856 kB
  55. Buffers: 41124 kB
  56. Cached: 915492 kB
  57. SwapCached: 4816 kB
  58. Active: 1309164 kB
  59. Inactive: 770840 kB
  60. Active(anon): 1110988 kB
  61. Inactive(anon): 12716 kB
  62. Active(file): 198176 kB
  63. Inactive(file): 758124 kB
  64. Unevictable: 0 kB
  65. Mlocked: 0 kB
  66. SwapTotal: 1023996 kB
  67. SwapFree: 1008880 kB
  68. Dirty: 1504 kB
  69. Writeback: 0 kB
  70. AnonPages: 1120368 kB
  71. Mapped: 60564 kB
  72. Shmem: 316 kB
  73. Slab: 136556 kB
  74. SReclaimable: 63768 kB
  75. SUnreclaim: 72788 kB
  76. KernelStack: 7288 kB
  77. PageTables: 19020 kB
  78. NFS_Unstable: 0 kB
  79. Bounce: 0 kB
  80. WritebackTmp: 0 kB
  81. CommitLimit: 2501832 kB
  82. Committed_AS: 1982020 kB
  83. VmallocTotal: 34359738367 kB
  84. VmallocUsed: 30260 kB
  85. VmallocChunk: 34359686748 kB
  86. HardwareCorrupted: 0 kB
  87. AnonHugePages: 974848 kB
  88. HugePages_Total: 0
  89. HugePages_Free: 0
  90. HugePages_Rsvd: 0
  91. HugePages_Surp: 0
  92. Hugepagesize: 2048 kB
  93. DirectMap4k: 7552 kB
  94. DirectMap2M: 3137536 kB
  95.  
  96. Tasks: 154 total, 1 running, 153 sleeping, 0 stopped, 0 zombie
  97. Cpu(s): 25.6%us, 1.7%sy, 0.2%ni, 72.3%id, 0.3%wa, 0.0%hi, 0.0%si, 0.0%st
  98. Mem: 2955676k total, 2377272k used, 578404k free, 41180k buffers
  99. Swap: 1023996k total, 15116k used, 1008880k free, 915500k cached
  100.  
  101. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  102. 7443 elastics 20 0 13.6g 707m 48m S 99.9 24.5 26:22.33 java
  103. 2162 root 20 0 113m 5384 1036 S 8.0 0.2 265:48.99 htop
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!