Pastebin launched a little side project called VERYVIRAL.com, check it out ;-) Want more features on Pastebin? Sign Up, it's FREE!
Guest

Untitled

By: chrisatk on Dec 19th, 2011  |  syntax: None  |  size: 26.15 KB  |  views: 57  |  expires: Never
download  |  raw  |  embed  |  report abuse  |  print
Text below is selected. Please press Ctrl+C to copy to your clipboard. (⌘+C on Mac)
  1. Dec 16 13:12:15 cluster3 kernel: clustat invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0
  2. Dec 16 13:12:15 cluster3 kernel: clustat cpuset=/ mems_allowed=0-1
  3. Dec 16 13:12:15 cluster3 kernel: Pid: 20536, comm: clustat Not tainted 2.6.32-131.0.15.el6.x86_64 #1
  4. Dec 16 13:12:16 cluster3 kernel: Call Trace:
  5. Dec 16 13:12:16 cluster3 kernel: [<ffffffff810c0101>] ? cpuset_print_task_mems_allowed+0x91/0xb0
  6. Dec 16 13:12:16 cluster3 kernel: [<ffffffff811102bb>] ? oom_kill_process+0xcb/0x2e0
  7. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81110880>] ? select_bad_process+0xd0/0x110
  8. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81110918>] ? __out_of_memory+0x58/0xc0
  9. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81110b19>] ? out_of_memory+0x199/0x210
  10. Dec 16 13:12:16 cluster3 kernel: [<ffffffff811202dd>] ? __alloc_pages_nodemask+0x80d/0x8b0
  11. Dec 16 13:12:16 cluster3 kernel: [<ffffffff8115474a>] ? alloc_pages_vma+0x9a/0x150
  12. Dec 16 13:12:16 cluster3 kernel: [<ffffffff811299c5>] ? shmem_alloc_page+0x55/0x60
  13. Dec 16 13:12:16 cluster3 kernel: [<ffffffff812115aa>] ? selinux_vm_enough_memory+0x4a/0x60
  14. Dec 16 13:12:16 cluster3 kernel: [<ffffffff8112caa9>] ? shmem_getpage+0x169/0x900
  15. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81227a67>] ? mls_level_isvalid+0x57/0x150
  16. Dec 16 13:12:16 cluster3 kernel: [<ffffffff8112d2dd>] ? shmem_write_begin+0x2d/0x30
  17. Dec 16 13:12:16 cluster3 kernel: [<ffffffff8110dcce>] ? generic_file_buffered_write+0x10e/0x2a0
  18. Dec 16 13:12:16 cluster3 kernel: [<ffffffff8106dd57>] ? current_fs_time+0x27/0x30
  19. Dec 16 13:12:16 cluster3 kernel: [<ffffffff8112a51d>] ? shmem_xattr_security_get+0x1d/0x30
  20. Dec 16 13:12:16 cluster3 kernel: [<ffffffff8110f620>] ? __generic_file_aio_write+0x250/0x480
  21. Dec 16 13:12:16 cluster3 kernel: [<ffffffff8120e174>] ? inode_has_perm+0x54/0xa0
  22. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81136422>] ? unmap_mapping_range+0x72/0x150
  23. Dec 16 13:12:16 cluster3 kernel: [<ffffffff8110f8bf>] ? generic_file_aio_write+0x6f/0xe0
  24. Dec 16 13:12:16 cluster3 kernel: [<ffffffff8117241a>] ? do_sync_write+0xfa/0x140
  25. Dec 16 13:12:16 cluster3 kernel: [<ffffffff8108e160>] ? autoremove_wake_function+0x0/0x40
  26. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81211d3b>] ? selinux_file_permission+0xfb/0x150
  27. Dec 16 13:12:16 cluster3 kernel: [<ffffffff812051a6>] ? security_file_permission+0x16/0x20
  28. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81172718>] ? vfs_write+0xb8/0x1a0
  29. Dec 16 13:12:16 cluster3 kernel: [<ffffffff810d1b62>] ? audit_syscall_entry+0x272/0x2a0
  30. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81173151>] ? sys_write+0x51/0x90
  31. Dec 16 13:12:16 cluster3 kernel: [<ffffffff8100b172>] ? system_call_fastpath+0x16/0x1b
  32. Dec 16 13:12:16 cluster3 kernel: Mem-Info:
  33. Dec 16 13:12:16 cluster3 kernel: Node 0 DMA per-cpu:
  34. Dec 16 13:12:16 cluster3 kernel: CPU    0: hi:    0, btch:   1 usd:   0
  35. Dec 16 13:12:16 cluster3 kernel: CPU    1: hi:    0, btch:   1 usd:   0
  36. Dec 16 13:12:16 cluster3 kernel: CPU    2: hi:    0, btch:   1 usd:   0
  37. Dec 16 13:12:16 cluster3 kernel: CPU    3: hi:    0, btch:   1 usd:   0
  38. Dec 16 13:12:16 cluster3 kernel: CPU    4: hi:    0, btch:   1 usd:   0
  39. Dec 16 13:12:16 cluster3 kernel: CPU    5: hi:    0, btch:   1 usd:   0
  40. Dec 16 13:12:16 cluster3 kernel: CPU    6: hi:    0, btch:   1 usd:   0
  41. Dec 16 13:12:16 cluster3 kernel: CPU    7: hi:    0, btch:   1 usd:   0
  42. Dec 16 13:12:16 cluster3 kernel: Node 0 DMA32 per-cpu:
  43. Dec 16 13:12:16 cluster3 kernel: CPU    0: hi:  186, btch:  31 usd: 185
  44. Dec 16 13:12:16 cluster3 kernel: CPU    1: hi:  186, btch:  31 usd:  35
  45. Dec 16 13:12:16 cluster3 kernel: CPU    2: hi:  186, btch:  31 usd:  51
  46. Dec 16 13:12:16 cluster3 kernel: CPU    3: hi:  186, btch:  31 usd:  30
  47. Dec 16 13:12:16 cluster3 kernel: CPU    4: hi:  186, btch:  31 usd: 128
  48. Dec 16 13:12:16 cluster3 kernel: CPU    5: hi:  186, btch:  31 usd:  30
  49. Dec 16 13:12:16 cluster3 kernel: CPU    6: hi:  186, btch:  31 usd:  34
  50. Dec 16 13:12:16 cluster3 kernel: CPU    7: hi:  186, btch:  31 usd:  30
  51. Dec 16 13:12:16 cluster3 kernel: Node 1 DMA32 per-cpu:
  52. Dec 16 13:12:16 cluster3 kernel: CPU    0: hi:  186, btch:  31 usd:  70
  53. Dec 16 13:12:16 cluster3 kernel: CPU    1: hi:  186, btch:  31 usd:  52
  54. Dec 16 13:12:16 cluster3 kernel: CPU    2: hi:  186, btch:  31 usd:  92
  55. Dec 16 13:12:16 cluster3 kernel: CPU    3: hi:  186, btch:  31 usd:   0
  56. Dec 16 13:12:16 cluster3 kernel: CPU    4: hi:  186, btch:  31 usd:  51
  57. Dec 16 13:12:16 cluster3 kernel: CPU    5: hi:  186, btch:  31 usd: 167
  58. Dec 16 13:12:16 cluster3 kernel: CPU    6: hi:  186, btch:  31 usd: 172
  59. Dec 16 13:12:16 cluster3 kernel: CPU    7: hi:  186, btch:  31 usd:   0
  60. Dec 16 13:12:16 cluster3 kernel: Node 1 Normal per-cpu:
  61. Dec 16 13:12:16 cluster3 kernel: CPU    0: hi:  186, btch:  31 usd:   0
  62. Dec 16 13:12:16 cluster3 kernel: CPU    1: hi:  186, btch:  31 usd:   0
  63. Dec 16 13:12:16 cluster3 kernel: CPU    2: hi:  186, btch:  31 usd:   0
  64. Dec 16 13:12:16 cluster3 kernel: CPU    3: hi:  186, btch:  31 usd:   0
  65. Dec 16 13:12:16 cluster3 kernel: CPU    4: hi:  186, btch:  31 usd:  61
  66. Dec 16 13:12:16 cluster3 kernel: CPU    5: hi:  186, btch:  31 usd:   0
  67. Dec 16 13:12:16 cluster3 kernel: CPU    6: hi:  186, btch:  31 usd:   0
  68. Dec 16 13:12:16 cluster3 kernel: CPU    7: hi:  186, btch:  31 usd:   0
  69. Dec 16 13:12:16 cluster3 kernel: active_anon:352 inactive_anon:1440 isolated_anon:0
  70. Dec 16 13:12:16 cluster3 kernel: active_file:72 inactive_file:230 isolated_file:0
  71. Dec 16 13:12:16 cluster3 kernel: unevictable:900568 dirty:16 writeback:1303 unstable:0
  72. Dec 16 13:12:16 cluster3 kernel: free:24057 slab_reclaimable:2246 slab_unreclaimable:34152
  73. Dec 16 13:12:16 cluster3 kernel: mapped:10062 shmem:8141 pagetables:3229 bounce:0
  74. Dec 16 13:12:16 cluster3 kernel: Node 0 DMA free:8340kB min:332kB low:412kB high:496kB active_anon:24kB inactive_anon:12kB active_file:0kB inactive_file:12kB unevictable:6984kB isolated(anon):0kB isolated(file):0kB present:15336kB mlocked:852kB dirty:0kB writeback:52kB mapped:52kB shmem:0kB slab_reclaimable:32kB slab_unreclaimable:248kB kernel_stack:0kB pagetables:16kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:11 all_unreclaimable? no
  75. Dec 16 13:12:16 cluster3 kernel: lowmem_reserve[]: 0 2004 2004 2004
  76. Dec 16 13:12:16 cluster3 kernel: Node 0 DMA32 free:44956kB min:44880kB low:56100kB high:67320kB active_anon:1336kB inactive_anon:4780kB active_file:112kB inactive_file:96kB unevictable:1676312kB isolated(anon):0kB isolated(file):0kB present:2052320kB mlocked:456020kB dirty:0kB writeback:5160kB mapped:35864kB shmem:31976kB slab_reclaimable:3896kB slab_unreclaimable:63928kB kernel_stack:2056kB pagetables:5704kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:9568 all_unreclaimable? yes
  77. Dec 16 13:12:16 cluster3 kernel: lowmem_reserve[]: 0 0 0 0
  78. Dec 16 13:12:16 cluster3 kernel: Node 1 DMA32 free:25628kB min:22272kB low:27840kB high:33408kB active_anon:24kB inactive_anon:968kB active_file:160kB inactive_file:788kB unevictable:982624kB isolated(anon):0kB isolated(file):0kB present:1018496kB mlocked:171156kB dirty:48kB writeback:0kB mapped:164kB shmem:588kB slab_reclaimable:40kB slab_unreclaimable:884kB kernel_stack:8kB pagetables:1828kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:96 all_unreclaimable? yes
  79. Dec 16 13:12:16 cluster3 kernel: lowmem_reserve[]: 0 0 1010 1010
  80. Dec 16 13:12:16 cluster3 kernel: Node 1 Normal free:17304kB min:22616kB low:28268kB high:33924kB active_anon:24kB inactive_anon:0kB active_file:16kB inactive_file:24kB unevictable:936352kB isolated(anon):0kB isolated(file):0kB present:1034240kB mlocked:312932kB dirty:16kB writeback:0kB mapped:4168kB shmem:0kB slab_reclaimable:5016kB slab_unreclaimable:71548kB kernel_stack:320kB pagetables:5368kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:108 all_unreclaimable? yes
  81. Dec 16 13:12:16 cluster3 kernel: lowmem_reserve[]: 0 0 0 0
  82. Dec 16 13:12:16 cluster3 kernel: Node 0 DMA: 33*4kB 41*8kB 31*16kB 17*32kB 11*64kB 10*128kB 5*256kB 3*512kB 2*1024kB 0*2048kB 0*4096kB = 8348kB
  83. Dec 16 13:12:16 cluster3 kernel: Node 0 DMA32: 531*4kB 342*8kB 169*16kB 121*32kB 82*64kB 45*128kB 27*256kB 12*512kB 9*1024kB 0*2048kB 0*4096kB = 44716kB
  84. Dec 16 13:12:16 cluster3 kernel: Node 1 DMA32: 36*4kB 190*8kB 138*16kB 58*32kB 27*64kB 11*128kB 9*256kB 4*512kB 12*1024kB 0*2048kB 0*4096kB = 25504kB
  85. Dec 16 13:12:16 cluster3 kernel: Node 1 Normal: 524*4kB 420*8kB 181*16kB 71*32kB 25*64kB 10*128kB 7*256kB 4*512kB 0*1024kB 0*2048kB 0*4096kB = 17344kB
  86. Dec 16 13:12:16 cluster3 kernel: 11956 total pagecache pages
  87. Dec 16 13:12:16 cluster3 kernel: 1469 pages in swap cache
  88. Dec 16 13:12:16 cluster3 kernel: Swap cache stats: add 6894103, delete 6892634, find 3822548/5496695
  89. Dec 16 13:12:16 cluster3 kernel: Free swap  = 25553140kB
  90. Dec 16 13:12:16 cluster3 kernel: Total swap = 25624568kB
  91. Dec 16 13:12:16 cluster3 kernel: 1048575 pages RAM
  92. Dec 16 13:12:16 cluster3 kernel: 68025 pages reserved
  93. Dec 16 13:12:16 cluster3 kernel: 22349 pages shared
  94. Dec 16 13:12:16 cluster3 kernel: 924649 pages non-shared
  95. Dec 16 13:12:16 cluster3 kernel: Out of memory: kill process 15944 (pickup) score 19656 or a child
  96. Dec 16 13:12:16 cluster3 kernel: Killed process 15944 (pickup) vsz:78624kB, anon-rss:0kB, file-rss:1116kB
  97. Dec 16 13:12:16 cluster3 kernel: dbus-daemon invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0
  98. Dec 16 13:12:16 cluster3 kernel: dbus-daemon cpuset=/ mems_allowed=0-1
  99. Dec 16 13:12:16 cluster3 kernel: Pid: 1984, comm: dbus-daemon Not tainted 2.6.32-131.0.15.el6.x86_64 #1
  100. Dec 16 13:12:16 cluster3 kernel: Call Trace:
  101. Dec 16 13:12:16 cluster3 kernel: [<ffffffff810c0101>] ? cpuset_print_task_mems_allowed+0x91/0xb0
  102. Dec 16 13:12:16 cluster3 kernel: [<ffffffff811102bb>] ? oom_kill_process+0xcb/0x2e0
  103. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81110880>] ? select_bad_process+0xd0/0x110
  104. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81110918>] ? __out_of_memory+0x58/0xc0
  105. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81110b19>] ? out_of_memory+0x199/0x210
  106. Dec 16 13:12:16 cluster3 kernel: [<ffffffff811202dd>] ? __alloc_pages_nodemask+0x80d/0x8b0
  107. Dec 16 13:12:16 cluster3 kernel: [<ffffffff811875b0>] ? __pollwait+0x0/0xf0
  108. Dec 16 13:12:16 cluster3 kernel: [<ffffffff8115474a>] ? alloc_pages_vma+0x9a/0x150
  109. Dec 16 13:12:16 cluster3 kernel: [<ffffffff811876a0>] ? pollwake+0x0/0x60
  110. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81147102>] ? read_swap_cache_async+0xf2/0x150
  111. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81147b19>] ? valid_swaphandles+0x69/0x160
  112. Dec 16 13:12:16 cluster3 kernel: [<ffffffff811471e7>] ? swapin_readahead+0x87/0xc0
  113. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81137cdb>] ? handle_pte_fault+0x70b/0xb50
  114. Dec 16 13:12:16 cluster3 kernel: [<ffffffff811876a0>] ? pollwake+0x0/0x60
  115. Dec 16 13:12:16 cluster3 kernel: [<ffffffff8140e4b0>] ? sock_aio_write+0x0/0x160
  116. Dec 16 13:12:16 cluster3 kernel: [<ffffffff8100be2e>] ? call_function_single_interrupt+0xe/0x20
  117. Dec 16 13:12:16 cluster3 kernel: [<ffffffff811382f8>] ? handle_mm_fault+0x1d8/0x2a0
  118. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81041529>] ? __do_page_fault+0x139/0x480
  119. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81211cff>] ? selinux_file_permission+0xbf/0x150
  120. Dec 16 13:12:16 cluster3 kernel: [<ffffffff812051a6>] ? security_file_permission+0x16/0x20
  121. Dec 16 13:12:16 cluster3 kernel: [<ffffffff8100bd0e>] ? invalidate_interrupt2+0xe/0x20
  122. Dec 16 13:12:16 cluster3 kernel: [<ffffffff814e0c3e>] ? do_page_fault+0x3e/0xa0
  123. Dec 16 13:12:16 cluster3 kernel: [<ffffffff814ddfe5>] ? page_fault+0x25/0x30
  124. Dec 16 13:12:16 cluster3 kernel: Mem-Info:
  125. Dec 16 13:12:16 cluster3 kernel: Node 0 DMA per-cpu:
  126. Dec 16 13:12:16 cluster3 kernel: CPU    0: hi:    0, btch:   1 usd:   0
  127. Dec 16 13:12:16 cluster3 kernel: CPU    1: hi:    0, btch:   1 usd:   0
  128. Dec 16 13:12:16 cluster3 kernel: CPU    2: hi:    0, btch:   1 usd:   0
  129. Dec 16 13:12:16 cluster3 kernel: CPU    3: hi:    0, btch:   1 usd:   0
  130. Dec 16 13:12:16 cluster3 kernel: CPU    4: hi:    0, btch:   1 usd:   0
  131. Dec 16 13:12:16 cluster3 kernel: CPU    5: hi:    0, btch:   1 usd:   0
  132. Dec 16 13:12:16 cluster3 kernel: CPU    6: hi:    0, btch:   1 usd:   0
  133. Dec 16 13:12:16 cluster3 kernel: CPU    7: hi:    0, btch:   1 usd:   0
  134. Dec 16 13:12:16 cluster3 kernel: Node 0 DMA32 per-cpu:
  135. Dec 16 13:12:16 cluster3 kernel: CPU    0: hi:  186, btch:  31 usd: 176
  136. Dec 16 13:12:16 cluster3 kernel: CPU    1: hi:  186, btch:  31 usd:  35
  137. Dec 16 13:12:16 cluster3 kernel: CPU    2: hi:  186, btch:  31 usd:  31
  138. Dec 16 13:12:16 cluster3 kernel: CPU    3: hi:  186, btch:  31 usd:  48
  139. Dec 16 13:12:16 cluster3 kernel: CPU    4: hi:  186, btch:  31 usd: 123
  140. Dec 16 13:12:16 cluster3 kernel: CPU    5: hi:  186, btch:  31 usd:  23
  141. Dec 16 13:12:16 cluster3 kernel: CPU    6: hi:  186, btch:  31 usd:  33
  142. Dec 16 13:12:16 cluster3 kernel: CPU    7: hi:  186, btch:  31 usd:  30
  143. Dec 16 13:12:16 cluster3 kernel: Node 1 DMA32 per-cpu:
  144. Dec 16 13:12:16 cluster3 kernel: CPU    0: hi:  186, btch:  31 usd:  56
  145. Dec 16 13:12:16 cluster3 kernel: CPU    1: hi:  186, btch:  31 usd:  52
  146. Dec 16 13:12:16 cluster3 kernel: CPU    2: hi:  186, btch:  31 usd:  92
  147. Dec 16 13:12:16 cluster3 kernel: CPU    3: hi:  186, btch:  31 usd:   0
  148. Dec 16 13:12:16 cluster3 kernel: CPU    4: hi:  186, btch:  31 usd:  69
  149. Dec 16 13:12:16 cluster3 kernel: CPU    5: hi:  186, btch:  31 usd: 167
  150. Dec 16 13:12:16 cluster3 kernel: CPU    6: hi:  186, btch:  31 usd: 172
  151. Dec 16 13:12:16 cluster3 kernel: CPU    7: hi:  186, btch:  31 usd:   0
  152. Dec 16 13:12:16 cluster3 kernel: Node 1 Normal per-cpu:
  153. Dec 16 13:12:16 cluster3 kernel: CPU    0: hi:  186, btch:  31 usd:   0
  154. Dec 16 13:12:16 cluster3 kernel: CPU    1: hi:  186, btch:  31 usd:   0
  155. Dec 16 13:12:16 cluster3 kernel: CPU    2: hi:  186, btch:  31 usd:   0
  156. Dec 16 13:12:16 cluster3 kernel: CPU    3: hi:  186, btch:  31 usd:   0
  157. Dec 16 13:12:16 cluster3 kernel: CPU    4: hi:  186, btch:  31 usd:  61
  158. Dec 16 13:12:16 cluster3 kernel: CPU    5: hi:  186, btch:  31 usd:   0
  159. Dec 16 13:12:16 cluster3 kernel: CPU    6: hi:  186, btch:  31 usd:   0
  160. Dec 16 13:12:16 cluster3 kernel: CPU    7: hi:  186, btch:  31 usd:   0
  161. Dec 16 13:12:16 cluster3 kernel: active_anon:433 inactive_anon:1612 isolated_anon:32
  162. Dec 16 13:12:16 cluster3 kernel: active_file:71 inactive_file:321 isolated_file:0
  163. Dec 16 13:12:16 cluster3 kernel: unevictable:900600 dirty:0 writeback:1375 unstable:0
  164. Dec 16 13:12:16 cluster3 kernel: free:24040 slab_reclaimable:2246 slab_unreclaimable:34223
  165. Dec 16 13:12:16 cluster3 kernel: mapped:10066 shmem:8074 pagetables:3272 bounce:0
  166. Dec 16 13:12:16 cluster3 kernel: Node 0 DMA free:8348kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:36kB active_file:0kB inactive_file:44kB unevictable:6984kB isolated(anon):0kB isolated(file):0kB present:15336kB mlocked:852kB dirty:0kB writeback:36kB mapped:44kB shmem:0kB slab_reclaimable:32kB slab_unreclaimable:248kB kernel_stack:0kB pagetables:16kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:69 all_unreclaimable? yes
  167. Dec 16 13:12:16 cluster3 kernel: lowmem_reserve[]: 0 2004 2004 2004
  168. Dec 16 13:12:16 cluster3 kernel: Node 0 DMA32 free:44964kB min:44880kB low:56100kB high:67320kB active_anon:1700kB inactive_anon:5212kB active_file:92kB inactive_file:312kB unevictable:1676440kB isolated(anon):128kB isolated(file):0kB present:2052320kB mlocked:456172kB dirty:0kB writeback:5432kB mapped:35628kB shmem:31640kB slab_reclaimable:3896kB slab_unreclaimable:64084kB kernel_stack:2064kB pagetables:5880kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:224 all_unreclaimable? no
  169. Dec 16 13:12:16 cluster3 kernel: lowmem_reserve[]: 0 0 0 0
  170. Dec 16 13:12:16 cluster3 kernel: Node 1 DMA32 free:25504kB min:22272kB low:27840kB high:33408kB active_anon:32kB inactive_anon:1168kB active_file:168kB inactive_file:912kB unevictable:982624kB isolated(anon):0kB isolated(file):0kB present:1018496kB mlocked:171156kB dirty:0kB writeback:24kB mapped:424kB shmem:656kB slab_reclaimable:40kB slab_unreclaimable:1012kB kernel_stack:8kB pagetables:1820kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:96 all_unreclaimable? yes
  171. Dec 16 13:12:16 cluster3 kernel: lowmem_reserve[]: 0 0 1010 1010
  172. Dec 16 13:12:16 cluster3 kernel: Node 1 Normal free:17344kB min:22616kB low:28268kB high:33924kB active_anon:0kB inactive_anon:32kB active_file:24kB inactive_file:16kB unevictable:936352kB isolated(anon):0kB isolated(file):0kB present:1034240kB mlocked:312932kB dirty:0kB writeback:8kB mapped:4168kB shmem:0kB slab_reclaimable:5016kB slab_unreclaimable:71548kB kernel_stack:328kB pagetables:5372kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:108 all_unreclaimable? yes
  173. Dec 16 13:12:16 cluster3 kernel: lowmem_reserve[]: 0 0 0 0
  174. Dec 16 13:12:16 cluster3 kernel: Node 0 DMA: 33*4kB 41*8kB 31*16kB 17*32kB 11*64kB 10*128kB 5*256kB 3*512kB 2*1024kB 0*2048kB 0*4096kB = 8348kB
  175. Dec 16 13:12:16 cluster3 kernel: Node 0 DMA32: 550*4kB 354*8kB 169*16kB 121*32kB 82*64kB 45*128kB 27*256kB 12*512kB 9*1024kB 0*2048kB 0*4096kB = 44888kB
  176. Dec 16 13:12:16 cluster3 kernel: Node 1 DMA32: 36*4kB 190*8kB 138*16kB 58*32kB 27*64kB 11*128kB 9*256kB 4*512kB 12*1024kB 0*2048kB 0*4096kB = 25504kB
  177. Dec 16 13:12:16 cluster3 kernel: Node 1 Normal: 524*4kB 420*8kB 181*16kB 71*32kB 25*64kB 10*128kB 7*256kB 4*512kB 0*1024kB 0*2048kB 0*4096kB = 17344kB
  178. Dec 16 13:12:16 cluster3 kernel: 12047 total pagecache pages
  179. Dec 16 13:12:16 cluster3 kernel: 1570 pages in swap cache
  180. Dec 16 13:12:16 cluster3 kernel: Swap cache stats: add 6894219, delete 6892649, find 3822577/5496747
  181. Dec 16 13:12:16 cluster3 kernel: Free swap  = 25553656kB
  182. Dec 16 13:12:16 cluster3 kernel: Total swap = 25624568kB
  183. Dec 16 13:12:16 cluster3 kernel: 1048575 pages RAM
  184. Dec 16 13:12:16 cluster3 kernel: 68025 pages reserved
  185. Dec 16 13:12:16 cluster3 kernel: 22033 pages shared
  186. Dec 16 13:12:16 cluster3 kernel: 924679 pages non-shared
  187. Dec 16 13:12:16 cluster3 kernel: Out of memory: kill process 2426 (rgmanager) score 5942 or a child
  188. Dec 16 13:12:16 cluster3 kernel: Killed process 2427 (rgmanager) vsz:494812kB, anon-rss:56kB, file-rss:1568kB
  189. Dec 16 13:12:16 cluster3 kernel: snmpd invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0
  190. Dec 16 13:12:16 cluster3 kernel: snmpd cpuset=/ mems_allowed=0-1
  191. Dec 16 13:12:16 cluster3 kernel: Pid: 2158, comm: snmpd Not tainted 2.6.32-131.0.15.el6.x86_64 #1
  192. Dec 16 13:12:16 cluster3 kernel: Call Trace:
  193. Dec 16 13:12:16 cluster3 kernel: [<ffffffff810c0101>] ? cpuset_print_task_mems_allowed+0x91/0xb0
  194. Dec 16 13:12:16 cluster3 kernel: [<ffffffff811102bb>] ? oom_kill_process+0xcb/0x2e0
  195. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81110880>] ? select_bad_process+0xd0/0x110
  196. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81110918>] ? __out_of_memory+0x58/0xc0
  197. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81110b19>] ? out_of_memory+0x199/0x210
  198. Dec 16 13:12:16 cluster3 kernel: [<ffffffff811202dd>] ? __alloc_pages_nodemask+0x80d/0x8b0
  199. Dec 16 13:12:16 cluster3 kernel: [<ffffffff8115474a>] ? alloc_pages_vma+0x9a/0x150
  200. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81147102>] ? read_swap_cache_async+0xf2/0x150
  201. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81147b19>] ? valid_swaphandles+0x69/0x160
  202. Dec 16 13:12:16 cluster3 kernel: [<ffffffff811471e7>] ? swapin_readahead+0x87/0xc0
  203. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81137cdb>] ? handle_pte_fault+0x70b/0xb50
  204. Dec 16 13:12:16 cluster3 kernel: [<ffffffff814db337>] ? thread_return+0x4e/0x777
  205. Dec 16 13:12:16 cluster3 kernel: [<ffffffff8100bc8e>] ? apic_timer_interrupt+0xe/0x20
  206. Dec 16 13:12:16 cluster3 kernel: [<ffffffff811382f8>] ? handle_mm_fault+0x1d8/0x2a0
  207. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81103c56>] ? __perf_event_task_sched_out+0x36/0x50
  208. Dec 16 13:12:16 cluster3 kernel: [<ffffffff81041529>] ? __do_page_fault+0x139/0x480
  209. Dec 16 13:12:16 cluster3 kernel: [<ffffffff810500f0>] ? __dequeue_entity+0x30/0x50
  210. Dec 16 13:12:16 cluster3 kernel: [<ffffffff810097bc>] ? __switch_to+0x1ac/0x320
  211. Dec 16 13:12:16 cluster3 kernel: [<ffffffff814db337>] ? thread_return+0x4e/0x777
  212. Dec 16 13:12:16 cluster3 kernel: [<ffffffff814e0c3e>] ? do_page_fault+0x3e/0xa0
  213. Dec 16 13:12:16 cluster3 kernel: [<ffffffff814ddfe5>] ? page_fault+0x25/0x30
  214. Dec 16 13:12:16 cluster3 kernel: Mem-Info:
  215. Dec 16 13:12:16 cluster3 kernel: Node 0 DMA per-cpu:
  216. Dec 16 13:12:16 cluster3 kernel: CPU    0: hi:    0, btch:   1 usd:   0
  217. Dec 16 13:12:16 cluster3 kernel: CPU    1: hi:    0, btch:   1 usd:   0
  218. Dec 16 13:12:16 cluster3 kernel: CPU    2: hi:    0, btch:   1 usd:   0
  219. Dec 16 13:12:16 cluster3 kernel: CPU    3: hi:    0, btch:   1 usd:   0
  220. Dec 16 13:12:16 cluster3 kernel: CPU    4: hi:    0, btch:   1 usd:   0
  221. Dec 16 13:12:16 cluster3 kernel: CPU    5: hi:    0, btch:   1 usd:   0
  222. Dec 16 13:12:16 cluster3 kernel: CPU    6: hi:    0, btch:   1 usd:   0
  223. Dec 16 13:12:16 cluster3 kernel: CPU    7: hi:    0, btch:   1 usd:   0
  224. Dec 16 13:12:16 cluster3 kernel: Node 0 DMA32 per-cpu:
  225. Dec 16 13:12:16 cluster3 kernel: CPU    0: hi:  186, btch:  31 usd: 159
  226. Dec 16 13:12:16 cluster3 kernel: CPU    1: hi:  186, btch:  31 usd:  35
  227. Dec 16 13:12:16 cluster3 kernel: CPU    2: hi:  186, btch:  31 usd: 133
  228. Dec 16 13:12:16 cluster3 kernel: CPU    3: hi:  186, btch:  31 usd:  48
  229. Dec 16 13:12:16 cluster3 kernel: CPU    4: hi:  186, btch:  31 usd:  43
  230. Dec 16 13:12:16 cluster3 kernel: CPU    5: hi:  186, btch:  31 usd:  10
  231. Dec 16 13:12:16 cluster3 kernel: CPU    6: hi:  186, btch:  31 usd:  30
  232. Dec 16 13:12:16 cluster3 kernel: CPU    7: hi:  186, btch:  31 usd:  44
  233. Dec 16 13:12:16 cluster3 kernel: Node 1 DMA32 per-cpu:
  234. Dec 16 13:12:16 cluster3 kernel: CPU    0: hi:  186, btch:  31 usd:  56
  235. Dec 16 13:12:16 cluster3 kernel: CPU    1: hi:  186, btch:  31 usd:  52
  236. Dec 16 13:12:16 cluster3 kernel: CPU    2: hi:  186, btch:  31 usd:  84
  237. Dec 16 13:12:16 cluster3 kernel: CPU    3: hi:  186, btch:  31 usd:   0
  238. Dec 16 13:12:16 cluster3 kernel: CPU    4: hi:  186, btch:  31 usd:  69
  239. Dec 16 13:12:16 cluster3 kernel: CPU    5: hi:  186, btch:  31 usd: 167
  240. Dec 16 13:12:16 cluster3 kernel: CPU    6: hi:  186, btch:  31 usd: 172
  241. Dec 16 13:12:16 cluster3 kernel: CPU    7: hi:  186, btch:  31 usd:   0
  242. Dec 16 13:12:16 cluster3 kernel: Node 1 Normal per-cpu:
  243. Dec 16 13:12:16 cluster3 kernel: CPU    0: hi:  186, btch:  31 usd:   0
  244. Dec 16 13:12:16 cluster3 kernel: CPU    1: hi:  186, btch:  31 usd:   0
  245. Dec 16 13:12:16 cluster3 kernel: CPU    2: hi:  186, btch:  31 usd:  20
  246. Dec 16 13:12:16 cluster3 kernel: CPU    3: hi:  186, btch:  31 usd:   0
  247. Dec 16 13:12:16 cluster3 kernel: CPU    4: hi:  186, btch:  31 usd:  61
  248. Dec 16 13:12:16 cluster3 kernel: CPU    5: hi:  186, btch:  31 usd:   0
  249. Dec 16 13:12:16 cluster3 kernel: CPU    6: hi:  186, btch:  31 usd:   0
  250. Dec 16 13:12:16 cluster3 kernel: CPU    7: hi:  186, btch:  31 usd:   0
  251. Dec 16 13:12:16 cluster3 kernel: active_anon:433 inactive_anon:1539 isolated_anon:32
  252. Dec 16 13:12:16 cluster3 kernel: active_file:112 inactive_file:236 isolated_file:0
  253. Dec 16 13:12:16 cluster3 kernel: unevictable:900600 dirty:0 writeback:1314 unstable:0
  254. Dec 16 13:12:16 cluster3 kernel: free:24166 slab_reclaimable:2246 slab_unreclaimable:34223
  255. Dec 16 13:12:16 cluster3 kernel: mapped:10066 shmem:8074 pagetables:3272 bounce:0
  256. Dec 16 13:12:16 cluster3 kernel: Node 0 DMA free:8348kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:36kB active_file:0kB inactive_file:44kB unevictable:6984kB isolated(anon):0kB isolated(file):0kB present:15336kB mlocked:852kB dirty:0kB writeback:36kB mapped:44kB shmem:0kB slab_reclaimable:32kB slab_unreclaimable:248kB kernel_stack:0kB pagetables:16kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:69 all_unreclaimable? yes
  257. Dec 16 13:12:16 cluster3 kernel: lowmem_reserve[]: 0 2004 2004 2004
  258. Dec 16 13:12:16 cluster3 kernel: Node 0 DMA32 free:45468kB min:44880kB low:56100kB high:67320kB active_anon:1700kB inactive_anon:4920kB active_file:256kB inactive_file:0kB unevictable:1676440kB isolated(anon):128kB isolated(file):0kB present:2052320kB mlocked:456172kB dirty:0kB writeback:5188kB mapped:35628kB shmem:31640kB slab_reclaimable:3896kB slab_unreclaimable:64084kB kernel_stack:2064kB pagetables:5880kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:199 all_unreclaimable? no
  259. Dec 16 13:12:16 cluster3 kernel: lowmem_reserve[]: 0 0 0 0
  260. Dec 16 13:12:16 cluster3 kernel: Node 1 DMA32 free:25504kB min:22272kB low:27840kB high:33408kB active_anon:32kB inactive_anon:1168kB active_file:168kB inactive_file:912kB unevictable:982624kB isolated(anon):0kB isolated(file):0kB present:1018496kB mlocked:171156kB dirty:0kB writeback:24kB mapped:424kB shmem:656kB slab_reclaimable:40kB slab_unreclaimable:1012kB kernel_stack:8kB pagetables:1820kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:96 all_unreclaimable? yes
  261. Dec 16 13:12:16 cluster3 kernel: lowmem_reserve[]: 0 0 1010 1010
  262. Dec 16 13:12:16 cluster3 kernel: Node 1 Normal free:17344kB min:22616kB low:28268kB high:33924kB active_anon:0kB inactive_anon:32kB active_file:24kB inactive_file:16kB unevictable:936352kB isolated(anon):0kB isolated(file):0kB present:1034240kB mlocked:312932kB dirty:0kB writeback:8kB mapped:4168kB shmem:0kB slab_reclaimable:5016kB slab_unreclaimable:71548kB kernel_stack:328kB pagetables:5372kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:120 all_unreclaimable? yes
  263. Dec 16 13:12:16 cluster3 kernel: lowmem_reserve[]: 0 0 0 0
  264. Dec 16 13:12:16 cluster3 kernel: Node 0 DMA: 33*4kB 41*8kB 31*16kB 17*32kB 11*64kB 10*128kB 5*256kB 3*512kB 2*1024kB 0*2048kB 0*4096kB = 8348kB
  265. Dec 16 13:12:16 cluster3 kernel: Node 0 DMA32: 577*4kB 377*8kB 180*16kB 125*32kB 83*64kB 45*128kB 27*256kB 12*512kB 9*1024kB 0*2048kB 0*4096kB = 45548kB
  266. Dec 16 13:12:16 cluster3 kernel: Node 1 DMA32: 36*4kB 190*8kB 138*16kB 58*32kB 27*64kB 11*128kB 9*256kB 4*512kB 12*1024kB 0*2048kB 0*4096kB = 25504kB
  267. Dec 16 13:12:16 cluster3 kernel: Node 1 Normal: 524*4kB 421*8kB 181*16kB 71*32kB 25*64kB 10*128kB 7*256kB 4*512kB 0*1024kB 0*2048kB 0*4096kB = 17352kB
  268. Dec 16 13:12:16 cluster3 kernel: 11925 total pagecache pages
  269. Dec 16 13:12:16 cluster3 kernel: 1486 pages in swap cache
  270. Dec 16 13:12:16 cluster3 kernel: Swap cache stats: add 6894351, delete 6892865, find 3822613/5496811
  271. Dec 16 13:12:16 cluster3 kernel: Free swap  = 25554632kB
  272. Dec 16 13:12:16 cluster3 kernel: Total swap = 25624568kB
  273. Dec 16 13:12:16 cluster3 kernel: 1048575 pages RAM
  274. Dec 16 13:12:16 cluster3 kernel: 68025 pages reserved
  275. Dec 16 13:12:16 cluster3 kernel: 21648 pages shared
  276. Dec 16 13:12:16 cluster3 kernel: 924249 pages non-shared
  277. Dec 16 13:12:16 cluster3 kernel: Out of memory: kill process 2280 (qmgr) score 4924 or a child
  278. Dec 16 13:12:16 cluster3 kernel: Killed process 2280 (qmgr) vsz:78796kB, anon-rss:0kB, file-rss:1164kB