API tools faq
paste
Advertisement
Guest User

mongo oom

a guest
Aug 18th, 2014
258
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 15.70 KB | None | 0 0
raw download clone embed print report
  1. [239261.639648] mongod invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
  2. [239261.639652] mongod cpuset=/ mems_allowed=0-1
  3. [239261.639655] CPU: 6 PID: 107921 Comm: mongod Not tainted 3.13.0-29-generic #53-Ubuntu
  4. [239261.639656] Hardware name: Xen HVM domU, BIOS 4.2.amazon 03/05/2014
  5. [239261.639658] 0000000000000000 ffff883802215a60 ffffffff8171a214 ffff883b3c3247d0
  6. [239261.639662] ffff883802215ae8 ffffffff81714b4f 0000000000000000 0000000000000000
  7. [239261.639665] 0000000000000000 0000000000000000 0000000000000000 0000000000000000
  8. [239261.639669] Call Trace:
  9. [239261.639676] [<ffffffff8171a214>] dump_stack+0x45/0x56
  10. [239261.639679] [<ffffffff81714b4f>] dump_header+0x7f/0x1f1
  11. [239261.639683] [<ffffffff81151a7e>] oom_kill_process+0x1ce/0x330
  12. [239261.639687] [<ffffffff812d35e5>] ? security_capable_noaudit+0x15/0x20
  13. [239261.639689] [<ffffffff811521b4>] out_of_memory+0x414/0x450
  14. [239261.639693] [<ffffffff8115843c>] __alloc_pages_nodemask+0xa5c/0xb80
  15. [239261.639696] [<ffffffff81195fe6>] alloc_page_interleave+0x46/0xa0
  16. [239261.639699] [<ffffffff811987f8>] alloc_pages_vma+0x108/0x140
  17. [239261.639703] [<ffffffff811791eb>] handle_mm_fault+0xb2b/0xf10
  18. [239261.639707] [<ffffffff8109a67a>] ? try_to_wake_up+0x1fa/0x2c0
  19. [239261.639710] [<ffffffff81726164>] __do_page_fault+0x184/0x560
  20. [239261.639714] [<ffffffff8101b7c9>] ? sched_clock+0x9/0x10
  21. [239261.639717] [<ffffffff8109d14d>] ? sched_clock_local+0x1d/0x80
  22. [239261.639721] [<ffffffff8111140c>] ? acct_account_cputime+0x1c/0x20
  23. [239261.639723] [<ffffffff8109d77b>] ? account_user_time+0x8b/0xa0
  24. [239261.639725] [<ffffffff8109dd94>] ? vtime_account_user+0x54/0x60
  25. [239261.639727] [<ffffffff8172655a>] do_page_fault+0x1a/0x70
  26. [239261.639731] [<ffffffff817229c8>] page_fault+0x28/0x30
  27. [239261.639732] Mem-Info:
  28. [239261.639734] Node 0 DMA per-cpu:
  29. [239261.639735] CPU 0: hi: 0, btch: 1 usd: 0
  30. [239261.639736] CPU 1: hi: 0, btch: 1 usd: 0
  31. [239261.639737] CPU 2: hi: 0, btch: 1 usd: 0
  32. [239261.639738] CPU 3: hi: 0, btch: 1 usd: 0
  33. [239261.639739] CPU 4: hi: 0, btch: 1 usd: 0
  34. [239261.639740] CPU 5: hi: 0, btch: 1 usd: 0
  35. [239261.639741] CPU 6: hi: 0, btch: 1 usd: 0
  36. [239261.639742] CPU 7: hi: 0, btch: 1 usd: 0
  37. [239261.639743] CPU 8: hi: 0, btch: 1 usd: 0
  38. [239261.639744] CPU 9: hi: 0, btch: 1 usd: 0
  39. [239261.639745] CPU 10: hi: 0, btch: 1 usd: 0
  40. [239261.639746] CPU 11: hi: 0, btch: 1 usd: 0
  41. [239261.639747] CPU 12: hi: 0, btch: 1 usd: 0
  42. [239261.639748] CPU 13: hi: 0, btch: 1 usd: 0
  43. [239261.639749] CPU 14: hi: 0, btch: 1 usd: 0
  44. [239261.639750] CPU 15: hi: 0, btch: 1 usd: 0
  45. [239261.639750] CPU 16: hi: 0, btch: 1 usd: 0
  46. [239261.639751] CPU 17: hi: 0, btch: 1 usd: 0
  47. [239261.639752] CPU 18: hi: 0, btch: 1 usd: 0
  48. [239261.639753] CPU 19: hi: 0, btch: 1 usd: 0
  49. [239261.639754] CPU 20: hi: 0, btch: 1 usd: 0
  50. [239261.639755] CPU 21: hi: 0, btch: 1 usd: 0
  51. [239261.639756] CPU 22: hi: 0, btch: 1 usd: 0
  52. [239261.639757] CPU 23: hi: 0, btch: 1 usd: 0
  53. [239261.639758] CPU 24: hi: 0, btch: 1 usd: 0
  54. [239261.639759] CPU 25: hi: 0, btch: 1 usd: 0
  55. [239261.639760] CPU 26: hi: 0, btch: 1 usd: 0
  56. [239261.639761] CPU 27: hi: 0, btch: 1 usd: 0
  57. [239261.639762] CPU 28: hi: 0, btch: 1 usd: 0
  58. [239261.639763] CPU 29: hi: 0, btch: 1 usd: 0
  59. [239261.639764] CPU 30: hi: 0, btch: 1 usd: 0
  60. [239261.639764] CPU 31: hi: 0, btch: 1 usd: 0
  61. [239261.639765] Node 0 DMA32 per-cpu:
  62. [239261.639767] CPU 0: hi: 186, btch: 31 usd: 3
  63. [239261.639768] CPU 1: hi: 186, btch: 31 usd: 1
  64. [239261.639769] CPU 2: hi: 186, btch: 31 usd: 33
  65. [239261.639770] CPU 3: hi: 186, btch: 31 usd: 4
  66. [239261.639771] CPU 4: hi: 186, btch: 31 usd: 30
  67. [239261.639772] CPU 5: hi: 186, btch: 31 usd: 58
  68. [239261.639773] CPU 6: hi: 186, btch: 31 usd: 3
  69. [239261.639774] CPU 7: hi: 186, btch: 31 usd: 3
  70. [239261.639774] CPU 8: hi: 186, btch: 31 usd: 3
  71. [239261.639775] CPU 9: hi: 186, btch: 31 usd: 29
  72. [239261.639776] CPU 10: hi: 186, btch: 31 usd: 3
  73. [239261.639777] CPU 11: hi: 186, btch: 31 usd: 4
  74. [239261.639778] CPU 12: hi: 186, btch: 31 usd: 36
  75. [239261.639779] CPU 13: hi: 186, btch: 31 usd: 1
  76. [239261.639780] CPU 14: hi: 186, btch: 31 usd: 1
  77. [239261.639781] CPU 15: hi: 186, btch: 31 usd: 35
  78. [239261.639782] CPU 16: hi: 186, btch: 31 usd: 2
  79. [239261.639783] CPU 17: hi: 186, btch: 31 usd: 1
  80. [239261.639784] CPU 18: hi: 186, btch: 31 usd: 4
  81. [239261.639785] CPU 19: hi: 186, btch: 31 usd: 0
  82. [239261.639786] CPU 20: hi: 186, btch: 31 usd: 4
  83. [239261.639787] CPU 21: hi: 186, btch: 31 usd: 21
  84. [239261.639788] CPU 22: hi: 186, btch: 31 usd: 30
  85. [239261.639789] CPU 23: hi: 186, btch: 31 usd: 185
  86. [239261.639790] CPU 24: hi: 186, btch: 31 usd: 30
  87. [239261.639791] CPU 25: hi: 186, btch: 31 usd: 5
  88. [239261.639792] CPU 26: hi: 186, btch: 31 usd: 9
  89. [239261.639793] CPU 27: hi: 186, btch: 31 usd: 42
  90. [239261.639794] CPU 28: hi: 186, btch: 31 usd: 8
  91. [239261.639795] CPU 29: hi: 186, btch: 31 usd: 32
  92. [239261.639796] CPU 30: hi: 186, btch: 31 usd: 3
  93. [239261.639797] CPU 31: hi: 186, btch: 31 usd: 2
  94. [239261.639798] Node 0 Normal per-cpu:
  95. [239261.639799] CPU 0: hi: 186, btch: 31 usd: 0
  96. [239261.639800] CPU 1: hi: 186, btch: 31 usd: 0
  97. [239261.639801] CPU 2: hi: 186, btch: 31 usd: 23
  98. [239261.639802] CPU 3: hi: 186, btch: 31 usd: 0
  99. [239261.639803] CPU 4: hi: 186, btch: 31 usd: 24
  100. [239261.639804] CPU 5: hi: 186, btch: 31 usd: 0
  101. [239261.639805] CPU 6: hi: 186, btch: 31 usd: 0
  102. [239261.639806] CPU 7: hi: 186, btch: 31 usd: 30
  103. [239261.639807] CPU 8: hi: 186, btch: 31 usd: 0
  104. [239261.639808] CPU 9: hi: 186, btch: 31 usd: 30
  105. [239261.639809] CPU 10: hi: 186, btch: 31 usd: 0
  106. [239261.639810] CPU 11: hi: 186, btch: 31 usd: 1
  107. [239261.639811] CPU 12: hi: 186, btch: 31 usd: 30
  108. [239261.639812] CPU 13: hi: 186, btch: 31 usd: 1
  109. [239261.639813] CPU 14: hi: 186, btch: 31 usd: 0
  110. [239261.639814] CPU 15: hi: 186, btch: 31 usd: 29
  111. [239261.639815] CPU 16: hi: 186, btch: 31 usd: 0
  112. [239261.639816] CPU 17: hi: 186, btch: 31 usd: 0
  113. [239261.639817] CPU 18: hi: 186, btch: 31 usd: 30
  114. [239261.639818] CPU 19: hi: 186, btch: 31 usd: 0
  115. [239261.639819] CPU 20: hi: 186, btch: 31 usd: 0
  116. [239261.639820] CPU 21: hi: 186, btch: 31 usd: 30
  117. [239261.639821] CPU 22: hi: 186, btch: 31 usd: 0
  118. [239261.639822] CPU 23: hi: 186, btch: 31 usd: 49
  119. [239261.639823] CPU 24: hi: 186, btch: 31 usd: 0
  120. [239261.639824] CPU 25: hi: 186, btch: 31 usd: 29
  121. [239261.639825] CPU 26: hi: 186, btch: 31 usd: 31
  122. [239261.639826] CPU 27: hi: 186, btch: 31 usd: 29
  123. [239261.639827] CPU 28: hi: 186, btch: 31 usd: 30
  124. [239261.639828] CPU 29: hi: 186, btch: 31 usd: 0
  125. [239261.639829] CPU 30: hi: 186, btch: 31 usd: 29
  126. [239261.639830] CPU 31: hi: 186, btch: 31 usd: 15
  127. [239261.639831] Node 1 Normal per-cpu:
  128. [239261.639832] CPU 0: hi: 186, btch: 31 usd: 107
  129. [239261.639833] CPU 1: hi: 186, btch: 31 usd: 114
  130. [239261.639834] CPU 2: hi: 186, btch: 31 usd: 78
  131. [239261.639835] CPU 3: hi: 186, btch: 31 usd: 136
  132. [239261.639836] CPU 4: hi: 186, btch: 31 usd: 84
  133. [239261.639837] CPU 5: hi: 186, btch: 31 usd: 159
  134. [239261.639838] CPU 6: hi: 186, btch: 31 usd: 142
  135. [239261.639839] CPU 7: hi: 186, btch: 31 usd: 123
  136. [239261.639840] CPU 8: hi: 186, btch: 31 usd: 148
  137. [239261.639841] CPU 9: hi: 186, btch: 31 usd: 163
  138. [239261.639842] CPU 10: hi: 186, btch: 31 usd: 96
  139. [239261.639843] CPU 11: hi: 186, btch: 31 usd: 51
  140. [239261.639844] CPU 12: hi: 186, btch: 31 usd: 173
  141. [239261.639845] CPU 13: hi: 186, btch: 31 usd: 164
  142. [239261.639846] CPU 14: hi: 186, btch: 31 usd: 154
  143. [239261.639847] CPU 15: hi: 186, btch: 31 usd: 164
  144. [239261.639848] CPU 16: hi: 186, btch: 31 usd: 144
  145. [239261.639849] CPU 17: hi: 186, btch: 31 usd: 182
  146. [239261.639850] CPU 18: hi: 186, btch: 31 usd: 158
  147. [239261.639851] CPU 19: hi: 186, btch: 31 usd: 130
  148. [239261.639852] CPU 20: hi: 186, btch: 31 usd: 169
  149. [239261.639853] CPU 21: hi: 186, btch: 31 usd: 184
  150. [239261.639854] CPU 22: hi: 186, btch: 31 usd: 83
  151. [239261.639855] CPU 23: hi: 186, btch: 31 usd: 1
  152. [239261.639856] CPU 24: hi: 186, btch: 31 usd: 63
  153. [239261.639857] CPU 25: hi: 186, btch: 31 usd: 72
  154. [239261.639859] CPU 26: hi: 186, btch: 31 usd: 164
  155. [239261.639860] CPU 27: hi: 186, btch: 31 usd: 182
  156. [239261.639861] CPU 28: hi: 186, btch: 31 usd: 168
  157. [239261.639862] CPU 29: hi: 186, btch: 31 usd: 140
  158. [239261.639863] CPU 30: hi: 186, btch: 31 usd: 161
  159. [239261.639864] CPU 31: hi: 186, btch: 31 usd: 154
  160. [239261.639867] active_anon:62274127 inactive_anon:81 isolated_anon:0
  161. [239261.639867] active_file:55 inactive_file:103 isolated_file:0
  162. [239261.639867] unevictable:0 dirty:0 writeback:0 unstable:0
  163. [239261.639867] free:147867 slab_reclaimable:12315 slab_unreclaimable:19495
  164. [239261.639867] mapped:0 shmem:101 pagetables:140897 bounce:0
  165. [239261.639867] free_cma:0
  166. [239261.639869] Node 0 DMA free:15904kB min:4kB low:4kB high:4kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15988kB managed:15904kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
  167. [239261.639873] lowmem_reserve[]: 0 3744 122942 122942
  168. [239261.639875] Node 0 DMA32 free:478476kB min:1368kB low:1708kB high:2052kB active_anon:3344328kB inactive_anon:8kB active_file:4kB inactive_file:4kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3915776kB managed:3836820kB mlocked:0kB dirty:0kB writeback:0kB mapped:4kB shmem:8kB slab_reclaimable:1288kB slab_unreclaimable:912kB kernel_stack:24kB pagetables:7728kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:57 all_unreclaimable? yes
  169. [239261.639878] lowmem_reserve[]: 0 0 119198 119198
  170. [239261.639881] Node 0 Normal free:42632kB min:43664kB low:54580kB high:65496kB active_anon:120922124kB inactive_anon:224kB active_file:160kB inactive_file:164kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:124067840kB managed:122059136kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:300kB slab_reclaimable:24148kB slab_unreclaimable:35988kB kernel_stack:5096kB pagetables:274328kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:596 all_unreclaimable? yes
  171. [239261.639884] lowmem_reserve[]: 0 0 0 0
  172. [239261.639886] Node 1 Normal free:54456kB min:45068kB low:56332kB high:67600kB active_anon:124830056kB inactive_anon:92kB active_file:56kB inactive_file:244kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:128000000kB managed:125990096kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:96kB slab_reclaimable:23824kB slab_unreclaimable:41080kB kernel_stack:7624kB pagetables:281532kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:2299 all_unreclaimable? yes
  173. [239261.639889] lowmem_reserve[]: 0 0 0 0
  174. [239261.639891] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15904kB
  175. [239261.639899] Node 0 DMA32: 181*4kB (UE) 245*8kB (UEM) 107*16kB (UEM) 89*32kB (UEM) 51*64kB (UEM) 982*128kB (UEM) 249*256kB (UEM) 18*512kB (EM) 5*1024kB (EM) 3*2048kB (EM) 63*4096kB (MR) = 478476kB
  176. [239261.639909] Node 0 Normal: 1093*4kB (UEM) 639*8kB (UEM) 337*16kB (UEM) 366*32kB (UEM) 152*64kB (UEM) 31*128kB (EM) 7*256kB (EM) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 42588kB
  177. [239261.639917] Node 1 Normal: 2673*4kB (UEM) 908*8kB (UEM) 352*16kB (UEM) 426*32kB (UEM) 147*64kB (UEM) 50*128kB (UEM) 4*256kB (EM) 1*512kB (E) 0*1024kB 0*2048kB 0*4096kB = 54564kB
  178. [239261.639926] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
  179. [239261.639927] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
  180. [239261.639928] 200 total pagecache pages
  181. [239261.639930] 0 pages in swap cache
  182. [239261.639931] Swap cache stats: add 0, delete 0, find 0/0
  183. [239261.639932] Free swap = 0kB
  184. [239261.639932] Total swap = 0kB
  185. [239261.639934] 63999901 pages RAM
  186. [239261.639934] 0 pages HighMem/MovableOnly
  187. [239261.639935] 502476 pages reserved
  188. [239261.639936] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
  189. [239261.639949] [ 965] 0 965 5267 412 14 0 0 upstart-udev-br
  190. [239261.639950] [ 969] 0 969 12425 167 28 0 -1000 systemd-udevd
  191. [239261.639952] [ 1179] 0 1179 3815 64 12 0 0 upstart-socket-
  192. [239261.639954] [ 1185] 0 1185 2556 576 8 0 0 dhclient
  193. [239261.639955] [ 1337] 102 1337 9808 96 23 0 0 dbus-daemon
  194. [239261.639957] [ 1346] 101 1346 63961 283 27 0 0 rsyslogd
  195. [239261.639958] [ 1359] 0 1359 10863 91 26 0 0 systemd-logind
  196. [239261.639960] [ 1384] 0 1384 3819 67 13 0 0 upstart-file-br
  197. [239261.639961] [ 1453] 0 1453 3635 40 12 0 0 getty
  198. [239261.639963] [ 1456] 0 1456 3635 41 12 0 0 getty
  199. [239261.639964] [ 1460] 0 1460 3635 40 12 0 0 getty
  200. [239261.639966] [ 1462] 0 1462 3635 42 12 0 0 getty
  201. [239261.639967] [ 1464] 0 1464 3635 40 12 0 0 getty
  202. [239261.639969] [ 1502] 0 1502 15341 170 33 0 -1000 sshd
  203. [239261.639970] [ 1512] 0 1512 5914 62 18 0 0 cron
  204. [239261.639972] [ 1513] 0 1513 4785 41 13 0 0 atd
  205. [239261.639973] [ 1545] 0 1545 3319 35 11 0 0 mdadm
  206. [239261.639975] [ 1594] 0 1594 4925 175 14 0 0 irqbalance
  207. [239261.639976] [ 1596] 0 1596 1092 35 8 0 0 acpid
  208. [239261.639977] [ 1661] 0 1661 3635 39 12 0 0 getty
  209. [239261.639979] [ 1662] 0 1662 3197 40 12 0 0 getty
  210. [239261.639981] [ 2301] 0 2301 26408 247 54 0 0 sshd
  211. [239261.639982] [ 2355] 1002 2355 26442 260 53 0 0 sshd
  212. [239261.639984] [ 2356] 1002 2356 5297 470 15 0 0 bash
  213. [239261.639986] [92244] 0 92244 26408 247 56 0 0 sshd
  214. [239261.639987] [92297] 1001 92297 26547 372 54 0 0 sshd
  215. [239261.639989] [92298] 1001 92298 5319 502 15 0 0 bash
  216. [239261.639991] [106862] 106 106862 228780278 62254327 140218 0 0 mongod
  217. [239261.639992] [107669] 1002 107669 153193 2319 38 0 0 mongo
  218. [239261.639994] [108631] 1001 108631 154786 1856 40 0 0 mongo
  219. [239261.639996] Out of memory: Kill process 106862 (mongod) score 990 or sacrifice child
  220. [239261.646484] Killed process 106862 (mongod) total-vm:915121112kB, anon-rss:249017308kB, file-rss:0kB
  221. [239282.557046] init: mongodb main process (106862) killed by KILL signal
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!