API tools faq
paste
Advertisement
chrisatk

Untitled

chrisatk
Dec 19th, 2011
474
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 26.15 KB | None | 0 0
raw download clone embed print report
  1. Dec 16 13:12:15 cluster3 kernel: clustat invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0
  2. Dec 16 13:12:15 cluster3 kernel: clustat cpuset=/ mems_allowed=0-1
  3. Dec 16 13:12:15 cluster3 kernel: Pid: 20536, comm: clustat Not tainted 2.6.32-131.0.15.el6.x86_64 #1
  4. Dec 16 13:12:16 cluster3 kernel: Call Trace:
  5. Dec 16 13:12:16 cluster3 kernel: [<ffffffff810c0101>] ? cpuset_print_task_mems_allowed+0x91/0xb0
  6. Dec 16 13:12:16 cluster3 kernel: [<ffffffff811102bb>] ? oom_kill_process+0xcb/0x2e0
  7. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81110880>] ? select_bad_process+0xd0/0x110
  8. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81110918>] ? __out_of_memory+0x58/0xc0
  9. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81110b19>] ? out_of_memory+0x199/0x210
  10. Dec 16 13:12:16 cluster3 kernel: [<ffffffff811202dd>] ? __alloc_pages_nodemask+0x80d/0x8b0
  11. Dec 16 13:12:16 cluster3 kernel: [<ffffffff8115474a>] ? alloc_pages_vma+0x9a/0x150
  12. Dec 16 13:12:16 cluster3 kernel: [<ffffffff811299c5>] ? shmem_alloc_page+0x55/0x60
  13. Dec 16 13:12:16 cluster3 kernel: [<ffffffff812115aa>] ? selinux_vm_enough_memory+0x4a/0x60
  14. Dec 16 13:12:16 cluster3 kernel: [<ffffffff8112caa9>] ? shmem_getpage+0x169/0x900
  15. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81227a67>] ? mls_level_isvalid+0x57/0x150
  16. Dec 16 13:12:16 cluster3 kernel: [<ffffffff8112d2dd>] ? shmem_write_begin+0x2d/0x30
  17. Dec 16 13:12:16 cluster3 kernel: [<ffffffff8110dcce>] ? generic_file_buffered_write+0x10e/0x2a0
  18. Dec 16 13:12:16 cluster3 kernel: [<ffffffff8106dd57>] ? current_fs_time+0x27/0x30
  19. Dec 16 13:12:16 cluster3 kernel: [<ffffffff8112a51d>] ? shmem_xattr_security_get+0x1d/0x30
  20. Dec 16 13:12:16 cluster3 kernel: [<ffffffff8110f620>] ? __generic_file_aio_write+0x250/0x480
  21. Dec 16 13:12:16 cluster3 kernel: [<ffffffff8120e174>] ? inode_has_perm+0x54/0xa0
  22. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81136422>] ? unmap_mapping_range+0x72/0x150
  23. Dec 16 13:12:16 cluster3 kernel: [<ffffffff8110f8bf>] ? generic_file_aio_write+0x6f/0xe0
  24. Dec 16 13:12:16 cluster3 kernel: [<ffffffff8117241a>] ? do_sync_write+0xfa/0x140
  25. Dec 16 13:12:16 cluster3 kernel: [<ffffffff8108e160>] ? autoremove_wake_function+0x0/0x40
  26. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81211d3b>] ? selinux_file_permission+0xfb/0x150
  27. Dec 16 13:12:16 cluster3 kernel: [<ffffffff812051a6>] ? security_file_permission+0x16/0x20
  28. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81172718>] ? vfs_write+0xb8/0x1a0
  29. Dec 16 13:12:16 cluster3 kernel: [<ffffffff810d1b62>] ? audit_syscall_entry+0x272/0x2a0
  30. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81173151>] ? sys_write+0x51/0x90
  31. Dec 16 13:12:16 cluster3 kernel: [<ffffffff8100b172>] ? system_call_fastpath+0x16/0x1b
  32. Dec 16 13:12:16 cluster3 kernel: Mem-Info:
  33. Dec 16 13:12:16 cluster3 kernel: Node 0 DMA per-cpu:
  34. Dec 16 13:12:16 cluster3 kernel: CPU 0: hi: 0, btch: 1 usd: 0
  35. Dec 16 13:12:16 cluster3 kernel: CPU 1: hi: 0, btch: 1 usd: 0
  36. Dec 16 13:12:16 cluster3 kernel: CPU 2: hi: 0, btch: 1 usd: 0
  37. Dec 16 13:12:16 cluster3 kernel: CPU 3: hi: 0, btch: 1 usd: 0
  38. Dec 16 13:12:16 cluster3 kernel: CPU 4: hi: 0, btch: 1 usd: 0
  39. Dec 16 13:12:16 cluster3 kernel: CPU 5: hi: 0, btch: 1 usd: 0
  40. Dec 16 13:12:16 cluster3 kernel: CPU 6: hi: 0, btch: 1 usd: 0
  41. Dec 16 13:12:16 cluster3 kernel: CPU 7: hi: 0, btch: 1 usd: 0
  42. Dec 16 13:12:16 cluster3 kernel: Node 0 DMA32 per-cpu:
  43. Dec 16 13:12:16 cluster3 kernel: CPU 0: hi: 186, btch: 31 usd: 185
  44. Dec 16 13:12:16 cluster3 kernel: CPU 1: hi: 186, btch: 31 usd: 35
  45. Dec 16 13:12:16 cluster3 kernel: CPU 2: hi: 186, btch: 31 usd: 51
  46. Dec 16 13:12:16 cluster3 kernel: CPU 3: hi: 186, btch: 31 usd: 30
  47. Dec 16 13:12:16 cluster3 kernel: CPU 4: hi: 186, btch: 31 usd: 128
  48. Dec 16 13:12:16 cluster3 kernel: CPU 5: hi: 186, btch: 31 usd: 30
  49. Dec 16 13:12:16 cluster3 kernel: CPU 6: hi: 186, btch: 31 usd: 34
  50. Dec 16 13:12:16 cluster3 kernel: CPU 7: hi: 186, btch: 31 usd: 30
  51. Dec 16 13:12:16 cluster3 kernel: Node 1 DMA32 per-cpu:
  52. Dec 16 13:12:16 cluster3 kernel: CPU 0: hi: 186, btch: 31 usd: 70
  53. Dec 16 13:12:16 cluster3 kernel: CPU 1: hi: 186, btch: 31 usd: 52
  54. Dec 16 13:12:16 cluster3 kernel: CPU 2: hi: 186, btch: 31 usd: 92
  55. Dec 16 13:12:16 cluster3 kernel: CPU 3: hi: 186, btch: 31 usd: 0
  56. Dec 16 13:12:16 cluster3 kernel: CPU 4: hi: 186, btch: 31 usd: 51
  57. Dec 16 13:12:16 cluster3 kernel: CPU 5: hi: 186, btch: 31 usd: 167
  58. Dec 16 13:12:16 cluster3 kernel: CPU 6: hi: 186, btch: 31 usd: 172
  59. Dec 16 13:12:16 cluster3 kernel: CPU 7: hi: 186, btch: 31 usd: 0
  60. Dec 16 13:12:16 cluster3 kernel: Node 1 Normal per-cpu:
  61. Dec 16 13:12:16 cluster3 kernel: CPU 0: hi: 186, btch: 31 usd: 0
  62. Dec 16 13:12:16 cluster3 kernel: CPU 1: hi: 186, btch: 31 usd: 0
  63. Dec 16 13:12:16 cluster3 kernel: CPU 2: hi: 186, btch: 31 usd: 0
  64. Dec 16 13:12:16 cluster3 kernel: CPU 3: hi: 186, btch: 31 usd: 0
  65. Dec 16 13:12:16 cluster3 kernel: CPU 4: hi: 186, btch: 31 usd: 61
  66. Dec 16 13:12:16 cluster3 kernel: CPU 5: hi: 186, btch: 31 usd: 0
  67. Dec 16 13:12:16 cluster3 kernel: CPU 6: hi: 186, btch: 31 usd: 0
  68. Dec 16 13:12:16 cluster3 kernel: CPU 7: hi: 186, btch: 31 usd: 0
  69. Dec 16 13:12:16 cluster3 kernel: active_anon:352 inactive_anon:1440 isolated_anon:0
  70. Dec 16 13:12:16 cluster3 kernel: active_file:72 inactive_file:230 isolated_file:0
  71. Dec 16 13:12:16 cluster3 kernel: unevictable:900568 dirty:16 writeback:1303 unstable:0
  72. Dec 16 13:12:16 cluster3 kernel: free:24057 slab_reclaimable:2246 slab_unreclaimable:34152
  73. Dec 16 13:12:16 cluster3 kernel: mapped:10062 shmem:8141 pagetables:3229 bounce:0
  74. Dec 16 13:12:16 cluster3 kernel: Node 0 DMA free:8340kB min:332kB low:412kB high:496kB active_anon:24kB inactive_anon:12kB active_file:0kB inactive_file:12kB unevictable:6984kB isolated(anon):0kB isolated(file):0kB present:15336kB mlocked:852kB dirty:0kB writeback:52kB mapped:52kB shmem:0kB slab_reclaimable:32kB slab_unreclaimable:248kB kernel_stack:0kB pagetables:16kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:11 all_unreclaimable? no
  75. Dec 16 13:12:16 cluster3 kernel: lowmem_reserve[]: 0 2004 2004 2004
  76. Dec 16 13:12:16 cluster3 kernel: Node 0 DMA32 free:44956kB min:44880kB low:56100kB high:67320kB active_anon:1336kB inactive_anon:4780kB active_file:112kB inactive_file:96kB unevictable:1676312kB isolated(anon):0kB isolated(file):0kB present:2052320kB mlocked:456020kB dirty:0kB writeback:5160kB mapped:35864kB shmem:31976kB slab_reclaimable:3896kB slab_unreclaimable:63928kB kernel_stack:2056kB pagetables:5704kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:9568 all_unreclaimable? yes
  77. Dec 16 13:12:16 cluster3 kernel: lowmem_reserve[]: 0 0 0 0
  78. Dec 16 13:12:16 cluster3 kernel: Node 1 DMA32 free:25628kB min:22272kB low:27840kB high:33408kB active_anon:24kB inactive_anon:968kB active_file:160kB inactive_file:788kB unevictable:982624kB isolated(anon):0kB isolated(file):0kB present:1018496kB mlocked:171156kB dirty:48kB writeback:0kB mapped:164kB shmem:588kB slab_reclaimable:40kB slab_unreclaimable:884kB kernel_stack:8kB pagetables:1828kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:96 all_unreclaimable? yes
  79. Dec 16 13:12:16 cluster3 kernel: lowmem_reserve[]: 0 0 1010 1010
  80. Dec 16 13:12:16 cluster3 kernel: Node 1 Normal free:17304kB min:22616kB low:28268kB high:33924kB active_anon:24kB inactive_anon:0kB active_file:16kB inactive_file:24kB unevictable:936352kB isolated(anon):0kB isolated(file):0kB present:1034240kB mlocked:312932kB dirty:16kB writeback:0kB mapped:4168kB shmem:0kB slab_reclaimable:5016kB slab_unreclaimable:71548kB kernel_stack:320kB pagetables:5368kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:108 all_unreclaimable? yes
  81. Dec 16 13:12:16 cluster3 kernel: lowmem_reserve[]: 0 0 0 0
  82. Dec 16 13:12:16 cluster3 kernel: Node 0 DMA: 33*4kB 41*8kB 31*16kB 17*32kB 11*64kB 10*128kB 5*256kB 3*512kB 2*1024kB 0*2048kB 0*4096kB = 8348kB
  83. Dec 16 13:12:16 cluster3 kernel: Node 0 DMA32: 531*4kB 342*8kB 169*16kB 121*32kB 82*64kB 45*128kB 27*256kB 12*512kB 9*1024kB 0*2048kB 0*4096kB = 44716kB
  84. Dec 16 13:12:16 cluster3 kernel: Node 1 DMA32: 36*4kB 190*8kB 138*16kB 58*32kB 27*64kB 11*128kB 9*256kB 4*512kB 12*1024kB 0*2048kB 0*4096kB = 25504kB
  85. Dec 16 13:12:16 cluster3 kernel: Node 1 Normal: 524*4kB 420*8kB 181*16kB 71*32kB 25*64kB 10*128kB 7*256kB 4*512kB 0*1024kB 0*2048kB 0*4096kB = 17344kB
  86. Dec 16 13:12:16 cluster3 kernel: 11956 total pagecache pages
  87. Dec 16 13:12:16 cluster3 kernel: 1469 pages in swap cache
  88. Dec 16 13:12:16 cluster3 kernel: Swap cache stats: add 6894103, delete 6892634, find 3822548/5496695
  89. Dec 16 13:12:16 cluster3 kernel: Free swap = 25553140kB
  90. Dec 16 13:12:16 cluster3 kernel: Total swap = 25624568kB
  91. Dec 16 13:12:16 cluster3 kernel: 1048575 pages RAM
  92. Dec 16 13:12:16 cluster3 kernel: 68025 pages reserved
  93. Dec 16 13:12:16 cluster3 kernel: 22349 pages shared
  94. Dec 16 13:12:16 cluster3 kernel: 924649 pages non-shared
  95. Dec 16 13:12:16 cluster3 kernel: Out of memory: kill process 15944 (pickup) score 19656 or a child
  96. Dec 16 13:12:16 cluster3 kernel: Killed process 15944 (pickup) vsz:78624kB, anon-rss:0kB, file-rss:1116kB
  97. Dec 16 13:12:16 cluster3 kernel: dbus-daemon invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0
  98. Dec 16 13:12:16 cluster3 kernel: dbus-daemon cpuset=/ mems_allowed=0-1
  99. Dec 16 13:12:16 cluster3 kernel: Pid: 1984, comm: dbus-daemon Not tainted 2.6.32-131.0.15.el6.x86_64 #1
  100. Dec 16 13:12:16 cluster3 kernel: Call Trace:
  101. Dec 16 13:12:16 cluster3 kernel: [<ffffffff810c0101>] ? cpuset_print_task_mems_allowed+0x91/0xb0
  102. Dec 16 13:12:16 cluster3 kernel: [<ffffffff811102bb>] ? oom_kill_process+0xcb/0x2e0
  103. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81110880>] ? select_bad_process+0xd0/0x110
  104. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81110918>] ? __out_of_memory+0x58/0xc0
  105. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81110b19>] ? out_of_memory+0x199/0x210
  106. Dec 16 13:12:16 cluster3 kernel: [<ffffffff811202dd>] ? __alloc_pages_nodemask+0x80d/0x8b0
  107. Dec 16 13:12:16 cluster3 kernel: [<ffffffff811875b0>] ? __pollwait+0x0/0xf0
  108. Dec 16 13:12:16 cluster3 kernel: [<ffffffff8115474a>] ? alloc_pages_vma+0x9a/0x150
  109. Dec 16 13:12:16 cluster3 kernel: [<ffffffff811876a0>] ? pollwake+0x0/0x60
  110. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81147102>] ? read_swap_cache_async+0xf2/0x150
  111. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81147b19>] ? valid_swaphandles+0x69/0x160
  112. Dec 16 13:12:16 cluster3 kernel: [<ffffffff811471e7>] ? swapin_readahead+0x87/0xc0
  113. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81137cdb>] ? handle_pte_fault+0x70b/0xb50
  114. Dec 16 13:12:16 cluster3 kernel: [<ffffffff811876a0>] ? pollwake+0x0/0x60
  115. Dec 16 13:12:16 cluster3 kernel: [<ffffffff8140e4b0>] ? sock_aio_write+0x0/0x160
  116. Dec 16 13:12:16 cluster3 kernel: [<ffffffff8100be2e>] ? call_function_single_interrupt+0xe/0x20
  117. Dec 16 13:12:16 cluster3 kernel: [<ffffffff811382f8>] ? handle_mm_fault+0x1d8/0x2a0
  118. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81041529>] ? __do_page_fault+0x139/0x480
  119. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81211cff>] ? selinux_file_permission+0xbf/0x150
  120. Dec 16 13:12:16 cluster3 kernel: [<ffffffff812051a6>] ? security_file_permission+0x16/0x20
  121. Dec 16 13:12:16 cluster3 kernel: [<ffffffff8100bd0e>] ? invalidate_interrupt2+0xe/0x20
  122. Dec 16 13:12:16 cluster3 kernel: [<ffffffff814e0c3e>] ? do_page_fault+0x3e/0xa0
  123. Dec 16 13:12:16 cluster3 kernel: [<ffffffff814ddfe5>] ? page_fault+0x25/0x30
  124. Dec 16 13:12:16 cluster3 kernel: Mem-Info:
  125. Dec 16 13:12:16 cluster3 kernel: Node 0 DMA per-cpu:
  126. Dec 16 13:12:16 cluster3 kernel: CPU 0: hi: 0, btch: 1 usd: 0
  127. Dec 16 13:12:16 cluster3 kernel: CPU 1: hi: 0, btch: 1 usd: 0
  128. Dec 16 13:12:16 cluster3 kernel: CPU 2: hi: 0, btch: 1 usd: 0
  129. Dec 16 13:12:16 cluster3 kernel: CPU 3: hi: 0, btch: 1 usd: 0
  130. Dec 16 13:12:16 cluster3 kernel: CPU 4: hi: 0, btch: 1 usd: 0
  131. Dec 16 13:12:16 cluster3 kernel: CPU 5: hi: 0, btch: 1 usd: 0
  132. Dec 16 13:12:16 cluster3 kernel: CPU 6: hi: 0, btch: 1 usd: 0
  133. Dec 16 13:12:16 cluster3 kernel: CPU 7: hi: 0, btch: 1 usd: 0
  134. Dec 16 13:12:16 cluster3 kernel: Node 0 DMA32 per-cpu:
  135. Dec 16 13:12:16 cluster3 kernel: CPU 0: hi: 186, btch: 31 usd: 176
  136. Dec 16 13:12:16 cluster3 kernel: CPU 1: hi: 186, btch: 31 usd: 35
  137. Dec 16 13:12:16 cluster3 kernel: CPU 2: hi: 186, btch: 31 usd: 31
  138. Dec 16 13:12:16 cluster3 kernel: CPU 3: hi: 186, btch: 31 usd: 48
  139. Dec 16 13:12:16 cluster3 kernel: CPU 4: hi: 186, btch: 31 usd: 123
  140. Dec 16 13:12:16 cluster3 kernel: CPU 5: hi: 186, btch: 31 usd: 23
  141. Dec 16 13:12:16 cluster3 kernel: CPU 6: hi: 186, btch: 31 usd: 33
  142. Dec 16 13:12:16 cluster3 kernel: CPU 7: hi: 186, btch: 31 usd: 30
  143. Dec 16 13:12:16 cluster3 kernel: Node 1 DMA32 per-cpu:
  144. Dec 16 13:12:16 cluster3 kernel: CPU 0: hi: 186, btch: 31 usd: 56
  145. Dec 16 13:12:16 cluster3 kernel: CPU 1: hi: 186, btch: 31 usd: 52
  146. Dec 16 13:12:16 cluster3 kernel: CPU 2: hi: 186, btch: 31 usd: 92
  147. Dec 16 13:12:16 cluster3 kernel: CPU 3: hi: 186, btch: 31 usd: 0
  148. Dec 16 13:12:16 cluster3 kernel: CPU 4: hi: 186, btch: 31 usd: 69
  149. Dec 16 13:12:16 cluster3 kernel: CPU 5: hi: 186, btch: 31 usd: 167
  150. Dec 16 13:12:16 cluster3 kernel: CPU 6: hi: 186, btch: 31 usd: 172
  151. Dec 16 13:12:16 cluster3 kernel: CPU 7: hi: 186, btch: 31 usd: 0
  152. Dec 16 13:12:16 cluster3 kernel: Node 1 Normal per-cpu:
  153. Dec 16 13:12:16 cluster3 kernel: CPU 0: hi: 186, btch: 31 usd: 0
  154. Dec 16 13:12:16 cluster3 kernel: CPU 1: hi: 186, btch: 31 usd: 0
  155. Dec 16 13:12:16 cluster3 kernel: CPU 2: hi: 186, btch: 31 usd: 0
  156. Dec 16 13:12:16 cluster3 kernel: CPU 3: hi: 186, btch: 31 usd: 0
  157. Dec 16 13:12:16 cluster3 kernel: CPU 4: hi: 186, btch: 31 usd: 61
  158. Dec 16 13:12:16 cluster3 kernel: CPU 5: hi: 186, btch: 31 usd: 0
  159. Dec 16 13:12:16 cluster3 kernel: CPU 6: hi: 186, btch: 31 usd: 0
  160. Dec 16 13:12:16 cluster3 kernel: CPU 7: hi: 186, btch: 31 usd: 0
  161. Dec 16 13:12:16 cluster3 kernel: active_anon:433 inactive_anon:1612 isolated_anon:32
  162. Dec 16 13:12:16 cluster3 kernel: active_file:71 inactive_file:321 isolated_file:0
  163. Dec 16 13:12:16 cluster3 kernel: unevictable:900600 dirty:0 writeback:1375 unstable:0
  164. Dec 16 13:12:16 cluster3 kernel: free:24040 slab_reclaimable:2246 slab_unreclaimable:34223
  165. Dec 16 13:12:16 cluster3 kernel: mapped:10066 shmem:8074 pagetables:3272 bounce:0
  166. Dec 16 13:12:16 cluster3 kernel: Node 0 DMA free:8348kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:36kB active_file:0kB inactive_file:44kB unevictable:6984kB isolated(anon):0kB isolated(file):0kB present:15336kB mlocked:852kB dirty:0kB writeback:36kB mapped:44kB shmem:0kB slab_reclaimable:32kB slab_unreclaimable:248kB kernel_stack:0kB pagetables:16kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:69 all_unreclaimable? yes
  167. Dec 16 13:12:16 cluster3 kernel: lowmem_reserve[]: 0 2004 2004 2004
  168. Dec 16 13:12:16 cluster3 kernel: Node 0 DMA32 free:44964kB min:44880kB low:56100kB high:67320kB active_anon:1700kB inactive_anon:5212kB active_file:92kB inactive_file:312kB unevictable:1676440kB isolated(anon):128kB isolated(file):0kB present:2052320kB mlocked:456172kB dirty:0kB writeback:5432kB mapped:35628kB shmem:31640kB slab_reclaimable:3896kB slab_unreclaimable:64084kB kernel_stack:2064kB pagetables:5880kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:224 all_unreclaimable? no
  169. Dec 16 13:12:16 cluster3 kernel: lowmem_reserve[]: 0 0 0 0
  170. Dec 16 13:12:16 cluster3 kernel: Node 1 DMA32 free:25504kB min:22272kB low:27840kB high:33408kB active_anon:32kB inactive_anon:1168kB active_file:168kB inactive_file:912kB unevictable:982624kB isolated(anon):0kB isolated(file):0kB present:1018496kB mlocked:171156kB dirty:0kB writeback:24kB mapped:424kB shmem:656kB slab_reclaimable:40kB slab_unreclaimable:1012kB kernel_stack:8kB pagetables:1820kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:96 all_unreclaimable? yes
  171. Dec 16 13:12:16 cluster3 kernel: lowmem_reserve[]: 0 0 1010 1010
  172. Dec 16 13:12:16 cluster3 kernel: Node 1 Normal free:17344kB min:22616kB low:28268kB high:33924kB active_anon:0kB inactive_anon:32kB active_file:24kB inactive_file:16kB unevictable:936352kB isolated(anon):0kB isolated(file):0kB present:1034240kB mlocked:312932kB dirty:0kB writeback:8kB mapped:4168kB shmem:0kB slab_reclaimable:5016kB slab_unreclaimable:71548kB kernel_stack:328kB pagetables:5372kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:108 all_unreclaimable? yes
  173. Dec 16 13:12:16 cluster3 kernel: lowmem_reserve[]: 0 0 0 0
  174. Dec 16 13:12:16 cluster3 kernel: Node 0 DMA: 33*4kB 41*8kB 31*16kB 17*32kB 11*64kB 10*128kB 5*256kB 3*512kB 2*1024kB 0*2048kB 0*4096kB = 8348kB
  175. Dec 16 13:12:16 cluster3 kernel: Node 0 DMA32: 550*4kB 354*8kB 169*16kB 121*32kB 82*64kB 45*128kB 27*256kB 12*512kB 9*1024kB 0*2048kB 0*4096kB = 44888kB
  176. Dec 16 13:12:16 cluster3 kernel: Node 1 DMA32: 36*4kB 190*8kB 138*16kB 58*32kB 27*64kB 11*128kB 9*256kB 4*512kB 12*1024kB 0*2048kB 0*4096kB = 25504kB
  177. Dec 16 13:12:16 cluster3 kernel: Node 1 Normal: 524*4kB 420*8kB 181*16kB 71*32kB 25*64kB 10*128kB 7*256kB 4*512kB 0*1024kB 0*2048kB 0*4096kB = 17344kB
  178. Dec 16 13:12:16 cluster3 kernel: 12047 total pagecache pages
  179. Dec 16 13:12:16 cluster3 kernel: 1570 pages in swap cache
  180. Dec 16 13:12:16 cluster3 kernel: Swap cache stats: add 6894219, delete 6892649, find 3822577/5496747
  181. Dec 16 13:12:16 cluster3 kernel: Free swap = 25553656kB
  182. Dec 16 13:12:16 cluster3 kernel: Total swap = 25624568kB
  183. Dec 16 13:12:16 cluster3 kernel: 1048575 pages RAM
  184. Dec 16 13:12:16 cluster3 kernel: 68025 pages reserved
  185. Dec 16 13:12:16 cluster3 kernel: 22033 pages shared
  186. Dec 16 13:12:16 cluster3 kernel: 924679 pages non-shared
  187. Dec 16 13:12:16 cluster3 kernel: Out of memory: kill process 2426 (rgmanager) score 5942 or a child
  188. Dec 16 13:12:16 cluster3 kernel: Killed process 2427 (rgmanager) vsz:494812kB, anon-rss:56kB, file-rss:1568kB
  189. Dec 16 13:12:16 cluster3 kernel: snmpd invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0
  190. Dec 16 13:12:16 cluster3 kernel: snmpd cpuset=/ mems_allowed=0-1
  191. Dec 16 13:12:16 cluster3 kernel: Pid: 2158, comm: snmpd Not tainted 2.6.32-131.0.15.el6.x86_64 #1
  192. Dec 16 13:12:16 cluster3 kernel: Call Trace:
  193. Dec 16 13:12:16 cluster3 kernel: [<ffffffff810c0101>] ? cpuset_print_task_mems_allowed+0x91/0xb0
  194. Dec 16 13:12:16 cluster3 kernel: [<ffffffff811102bb>] ? oom_kill_process+0xcb/0x2e0
  195. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81110880>] ? select_bad_process+0xd0/0x110
  196. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81110918>] ? __out_of_memory+0x58/0xc0
  197. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81110b19>] ? out_of_memory+0x199/0x210
  198. Dec 16 13:12:16 cluster3 kernel: [<ffffffff811202dd>] ? __alloc_pages_nodemask+0x80d/0x8b0
  199. Dec 16 13:12:16 cluster3 kernel: [<ffffffff8115474a>] ? alloc_pages_vma+0x9a/0x150
  200. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81147102>] ? read_swap_cache_async+0xf2/0x150
  201. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81147b19>] ? valid_swaphandles+0x69/0x160
  202. Dec 16 13:12:16 cluster3 kernel: [<ffffffff811471e7>] ? swapin_readahead+0x87/0xc0
  203. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81137cdb>] ? handle_pte_fault+0x70b/0xb50
  204. Dec 16 13:12:16 cluster3 kernel: [<ffffffff814db337>] ? thread_return+0x4e/0x777
  205. Dec 16 13:12:16 cluster3 kernel: [<ffffffff8100bc8e>] ? apic_timer_interrupt+0xe/0x20
  206. Dec 16 13:12:16 cluster3 kernel: [<ffffffff811382f8>] ? handle_mm_fault+0x1d8/0x2a0
  207. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81103c56>] ? __perf_event_task_sched_out+0x36/0x50
  208. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81041529>] ? __do_page_fault+0x139/0x480
  209. Dec 16 13:12:16 cluster3 kernel: [<ffffffff810500f0>] ? __dequeue_entity+0x30/0x50
  210. Dec 16 13:12:16 cluster3 kernel: [<ffffffff810097bc>] ? __switch_to+0x1ac/0x320
  211. Dec 16 13:12:16 cluster3 kernel: [<ffffffff814db337>] ? thread_return+0x4e/0x777
  212. Dec 16 13:12:16 cluster3 kernel: [<ffffffff814e0c3e>] ? do_page_fault+0x3e/0xa0
  213. Dec 16 13:12:16 cluster3 kernel: [<ffffffff814ddfe5>] ? page_fault+0x25/0x30
  214. Dec 16 13:12:16 cluster3 kernel: Mem-Info:
  215. Dec 16 13:12:16 cluster3 kernel: Node 0 DMA per-cpu:
  216. Dec 16 13:12:16 cluster3 kernel: CPU 0: hi: 0, btch: 1 usd: 0
  217. Dec 16 13:12:16 cluster3 kernel: CPU 1: hi: 0, btch: 1 usd: 0
  218. Dec 16 13:12:16 cluster3 kernel: CPU 2: hi: 0, btch: 1 usd: 0
  219. Dec 16 13:12:16 cluster3 kernel: CPU 3: hi: 0, btch: 1 usd: 0
  220. Dec 16 13:12:16 cluster3 kernel: CPU 4: hi: 0, btch: 1 usd: 0
  221. Dec 16 13:12:16 cluster3 kernel: CPU 5: hi: 0, btch: 1 usd: 0
  222. Dec 16 13:12:16 cluster3 kernel: CPU 6: hi: 0, btch: 1 usd: 0
  223. Dec 16 13:12:16 cluster3 kernel: CPU 7: hi: 0, btch: 1 usd: 0
  224. Dec 16 13:12:16 cluster3 kernel: Node 0 DMA32 per-cpu:
  225. Dec 16 13:12:16 cluster3 kernel: CPU 0: hi: 186, btch: 31 usd: 159
  226. Dec 16 13:12:16 cluster3 kernel: CPU 1: hi: 186, btch: 31 usd: 35
  227. Dec 16 13:12:16 cluster3 kernel: CPU 2: hi: 186, btch: 31 usd: 133
  228. Dec 16 13:12:16 cluster3 kernel: CPU 3: hi: 186, btch: 31 usd: 48
  229. Dec 16 13:12:16 cluster3 kernel: CPU 4: hi: 186, btch: 31 usd: 43
  230. Dec 16 13:12:16 cluster3 kernel: CPU 5: hi: 186, btch: 31 usd: 10
  231. Dec 16 13:12:16 cluster3 kernel: CPU 6: hi: 186, btch: 31 usd: 30
  232. Dec 16 13:12:16 cluster3 kernel: CPU 7: hi: 186, btch: 31 usd: 44
  233. Dec 16 13:12:16 cluster3 kernel: Node 1 DMA32 per-cpu:
  234. Dec 16 13:12:16 cluster3 kernel: CPU 0: hi: 186, btch: 31 usd: 56
  235. Dec 16 13:12:16 cluster3 kernel: CPU 1: hi: 186, btch: 31 usd: 52
  236. Dec 16 13:12:16 cluster3 kernel: CPU 2: hi: 186, btch: 31 usd: 84
  237. Dec 16 13:12:16 cluster3 kernel: CPU 3: hi: 186, btch: 31 usd: 0
  238. Dec 16 13:12:16 cluster3 kernel: CPU 4: hi: 186, btch: 31 usd: 69
  239. Dec 16 13:12:16 cluster3 kernel: CPU 5: hi: 186, btch: 31 usd: 167
  240. Dec 16 13:12:16 cluster3 kernel: CPU 6: hi: 186, btch: 31 usd: 172
  241. Dec 16 13:12:16 cluster3 kernel: CPU 7: hi: 186, btch: 31 usd: 0
  242. Dec 16 13:12:16 cluster3 kernel: Node 1 Normal per-cpu:
  243. Dec 16 13:12:16 cluster3 kernel: CPU 0: hi: 186, btch: 31 usd: 0
  244. Dec 16 13:12:16 cluster3 kernel: CPU 1: hi: 186, btch: 31 usd: 0
  245. Dec 16 13:12:16 cluster3 kernel: CPU 2: hi: 186, btch: 31 usd: 20
  246. Dec 16 13:12:16 cluster3 kernel: CPU 3: hi: 186, btch: 31 usd: 0
  247. Dec 16 13:12:16 cluster3 kernel: CPU 4: hi: 186, btch: 31 usd: 61
  248. Dec 16 13:12:16 cluster3 kernel: CPU 5: hi: 186, btch: 31 usd: 0
  249. Dec 16 13:12:16 cluster3 kernel: CPU 6: hi: 186, btch: 31 usd: 0
  250. Dec 16 13:12:16 cluster3 kernel: CPU 7: hi: 186, btch: 31 usd: 0
  251. Dec 16 13:12:16 cluster3 kernel: active_anon:433 inactive_anon:1539 isolated_anon:32
  252. Dec 16 13:12:16 cluster3 kernel: active_file:112 inactive_file:236 isolated_file:0
  253. Dec 16 13:12:16 cluster3 kernel: unevictable:900600 dirty:0 writeback:1314 unstable:0
  254. Dec 16 13:12:16 cluster3 kernel: free:24166 slab_reclaimable:2246 slab_unreclaimable:34223
  255. Dec 16 13:12:16 cluster3 kernel: mapped:10066 shmem:8074 pagetables:3272 bounce:0
  256. Dec 16 13:12:16 cluster3 kernel: Node 0 DMA free:8348kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:36kB active_file:0kB inactive_file:44kB unevictable:6984kB isolated(anon):0kB isolated(file):0kB present:15336kB mlocked:852kB dirty:0kB writeback:36kB mapped:44kB shmem:0kB slab_reclaimable:32kB slab_unreclaimable:248kB kernel_stack:0kB pagetables:16kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:69 all_unreclaimable? yes
  257. Dec 16 13:12:16 cluster3 kernel: lowmem_reserve[]: 0 2004 2004 2004
  258. Dec 16 13:12:16 cluster3 kernel: Node 0 DMA32 free:45468kB min:44880kB low:56100kB high:67320kB active_anon:1700kB inactive_anon:4920kB active_file:256kB inactive_file:0kB unevictable:1676440kB isolated(anon):128kB isolated(file):0kB present:2052320kB mlocked:456172kB dirty:0kB writeback:5188kB mapped:35628kB shmem:31640kB slab_reclaimable:3896kB slab_unreclaimable:64084kB kernel_stack:2064kB pagetables:5880kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:199 all_unreclaimable? no
  259. Dec 16 13:12:16 cluster3 kernel: lowmem_reserve[]: 0 0 0 0
  260. Dec 16 13:12:16 cluster3 kernel: Node 1 DMA32 free:25504kB min:22272kB low:27840kB high:33408kB active_anon:32kB inactive_anon:1168kB active_file:168kB inactive_file:912kB unevictable:982624kB isolated(anon):0kB isolated(file):0kB present:1018496kB mlocked:171156kB dirty:0kB writeback:24kB mapped:424kB shmem:656kB slab_reclaimable:40kB slab_unreclaimable:1012kB kernel_stack:8kB pagetables:1820kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:96 all_unreclaimable? yes
  261. Dec 16 13:12:16 cluster3 kernel: lowmem_reserve[]: 0 0 1010 1010
  262. Dec 16 13:12:16 cluster3 kernel: Node 1 Normal free:17344kB min:22616kB low:28268kB high:33924kB active_anon:0kB inactive_anon:32kB active_file:24kB inactive_file:16kB unevictable:936352kB isolated(anon):0kB isolated(file):0kB present:1034240kB mlocked:312932kB dirty:0kB writeback:8kB mapped:4168kB shmem:0kB slab_reclaimable:5016kB slab_unreclaimable:71548kB kernel_stack:328kB pagetables:5372kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:120 all_unreclaimable? yes
  263. Dec 16 13:12:16 cluster3 kernel: lowmem_reserve[]: 0 0 0 0
  264. Dec 16 13:12:16 cluster3 kernel: Node 0 DMA: 33*4kB 41*8kB 31*16kB 17*32kB 11*64kB 10*128kB 5*256kB 3*512kB 2*1024kB 0*2048kB 0*4096kB = 8348kB
  265. Dec 16 13:12:16 cluster3 kernel: Node 0 DMA32: 577*4kB 377*8kB 180*16kB 125*32kB 83*64kB 45*128kB 27*256kB 12*512kB 9*1024kB 0*2048kB 0*4096kB = 45548kB
  266. Dec 16 13:12:16 cluster3 kernel: Node 1 DMA32: 36*4kB 190*8kB 138*16kB 58*32kB 27*64kB 11*128kB 9*256kB 4*512kB 12*1024kB 0*2048kB 0*4096kB = 25504kB
  267. Dec 16 13:12:16 cluster3 kernel: Node 1 Normal: 524*4kB 421*8kB 181*16kB 71*32kB 25*64kB 10*128kB 7*256kB 4*512kB 0*1024kB 0*2048kB 0*4096kB = 17352kB
  268. Dec 16 13:12:16 cluster3 kernel: 11925 total pagecache pages
  269. Dec 16 13:12:16 cluster3 kernel: 1486 pages in swap cache
  270. Dec 16 13:12:16 cluster3 kernel: Swap cache stats: add 6894351, delete 6892865, find 3822613/5496811
  271. Dec 16 13:12:16 cluster3 kernel: Free swap = 25554632kB
  272. Dec 16 13:12:16 cluster3 kernel: Total swap = 25624568kB
  273. Dec 16 13:12:16 cluster3 kernel: 1048575 pages RAM
  274. Dec 16 13:12:16 cluster3 kernel: 68025 pages reserved
  275. Dec 16 13:12:16 cluster3 kernel: 21648 pages shared
  276. Dec 16 13:12:16 cluster3 kernel: 924249 pages non-shared
  277. Dec 16 13:12:16 cluster3 kernel: Out of memory: kill process 2280 (qmgr) score 4924 or a child
  278. Dec 16 13:12:16 cluster3 kernel: Killed process 2280 (qmgr) vsz:78796kB, anon-rss:0kB, file-rss:1164kB
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!