Dec 16 13:12:15 cluster3 kernel: clustat invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0
Dec 16 13:12:15 cluster3 kernel: clustat cpuset=/ mems_allowed=0-1
Dec 16 13:12:15 cluster3 kernel: Pid: 20536, comm: clustat Not tainted 2.6.32-131.0.15.el6.x86_64 #1
Dec 16 13:12:16 cluster3 kernel: Call Trace:
Dec 16 13:12:16 cluster3 kernel: [<ffffffff810c0101>] ? cpuset_print_task_mems_allowed+0x91/0xb0
Dec 16 13:12:16 cluster3 kernel: [<ffffffff811102bb>] ? oom_kill_process+0xcb/0x2e0
Dec 16 13:12:16 cluster3 kernel: [<ffffffff81110880>] ? select_bad_process+0xd0/0x110
Dec 16 13:12:16 cluster3 kernel: [<ffffffff81110918>] ? __out_of_memory+0x58/0xc0
Dec 16 13:12:16 cluster3 kernel: [<ffffffff81110b19>] ? out_of_memory+0x199/0x210
Dec 16 13:12:16 cluster3 kernel: [<ffffffff811202dd>] ? __alloc_pages_nodemask+0x80d/0x8b0
Dec 16 13:12:16 cluster3 kernel: [<ffffffff8115474a>] ? alloc_pages_vma+0x9a/0x150
Dec 16 13:12:16 cluster3 kernel: [<ffffffff811299c5>] ? shmem_alloc_page+0x55/0x60
Dec 16 13:12:16 cluster3 kernel: [<ffffffff812115aa>] ? selinux_vm_enough_memory+0x4a/0x60
Dec 16 13:12:16 cluster3 kernel: [<ffffffff8112caa9>] ? shmem_getpage+0x169/0x900
Dec 16 13:12:16 cluster3 kernel: [<ffffffff81227a67>] ? mls_level_isvalid+0x57/0x150
Dec 16 13:12:16 cluster3 kernel: [<ffffffff8112d2dd>] ? shmem_write_begin+0x2d/0x30
Dec 16 13:12:16 cluster3 kernel: [<ffffffff8110dcce>] ? generic_file_buffered_write+0x10e/0x2a0
Dec 16 13:12:16 cluster3 kernel: [<ffffffff8106dd57>] ? current_fs_time+0x27/0x30
Dec 16 13:12:16 cluster3 kernel: [<ffffffff8112a51d>] ? shmem_xattr_security_get+0x1d/0x30
Dec 16 13:12:16 cluster3 kernel: [<ffffffff8110f620>] ? __generic_file_aio_write+0x250/0x480
Dec 16 13:12:16 cluster3 kernel: [<ffffffff8120e174>] ? inode_has_perm+0x54/0xa0
Dec 16 13:12:16 cluster3 kernel: [<ffffffff81136422>] ? unmap_mapping_range+0x72/0x150
Dec 16 13:12:16 cluster3 kernel: [<ffffffff8110f8bf>] ? generic_file_aio_write+0x6f/0xe0
Dec 16 13:12:16 cluster3 kernel: [<ffffffff8117241a>] ? do_sync_write+0xfa/0x140
Dec 16 13:12:16 cluster3 kernel: [<ffffffff8108e160>] ? autoremove_wake_function+0x0/0x40
Dec 16 13:12:16 cluster3 kernel: [<ffffffff81211d3b>] ? selinux_file_permission+0xfb/0x150
Dec 16 13:12:16 cluster3 kernel: [<ffffffff812051a6>] ? security_file_permission+0x16/0x20
Dec 16 13:12:16 cluster3 kernel: [<ffffffff81172718>] ? vfs_write+0xb8/0x1a0
Dec 16 13:12:16 cluster3 kernel: [<ffffffff810d1b62>] ? audit_syscall_entry+0x272/0x2a0
Dec 16 13:12:16 cluster3 kernel: [<ffffffff81173151>] ? sys_write+0x51/0x90
Dec 16 13:12:16 cluster3 kernel: [<ffffffff8100b172>] ? system_call_fastpath+0x16/0x1b
Dec 16 13:12:16 cluster3 kernel: Mem-Info:
Dec 16 13:12:16 cluster3 kernel: Node 0 DMA per-cpu:
Dec 16 13:12:16 cluster3 kernel: CPU    0: hi:    0, btch:   1 usd:   0
Dec 16 13:12:16 cluster3 kernel: CPU    1: hi:    0, btch:   1 usd:   0
Dec 16 13:12:16 cluster3 kernel: CPU    2: hi:    0, btch:   1 usd:   0
Dec 16 13:12:16 cluster3 kernel: CPU    3: hi:    0, btch:   1 usd:   0
Dec 16 13:12:16 cluster3 kernel: CPU    4: hi:    0, btch:   1 usd:   0
Dec 16 13:12:16 cluster3 kernel: CPU    5: hi:    0, btch:   1 usd:   0
Dec 16 13:12:16 cluster3 kernel: CPU    6: hi:    0, btch:   1 usd:   0
Dec 16 13:12:16 cluster3 kernel: CPU    7: hi:    0, btch:   1 usd:   0
Dec 16 13:12:16 cluster3 kernel: Node 0 DMA32 per-cpu:
Dec 16 13:12:16 cluster3 kernel: CPU    0: hi:  186, btch:  31 usd: 185
Dec 16 13:12:16 cluster3 kernel: CPU    1: hi:  186, btch:  31 usd:  35
Dec 16 13:12:16 cluster3 kernel: CPU    2: hi:  186, btch:  31 usd:  51
Dec 16 13:12:16 cluster3 kernel: CPU    3: hi:  186, btch:  31 usd:  30
Dec 16 13:12:16 cluster3 kernel: CPU    4: hi:  186, btch:  31 usd: 128
Dec 16 13:12:16 cluster3 kernel: CPU    5: hi:  186, btch:  31 usd:  30
Dec 16 13:12:16 cluster3 kernel: CPU    6: hi:  186, btch:  31 usd:  34
Dec 16 13:12:16 cluster3 kernel: CPU    7: hi:  186, btch:  31 usd:  30
Dec 16 13:12:16 cluster3 kernel: Node 1 DMA32 per-cpu:
Dec 16 13:12:16 cluster3 kernel: CPU    0: hi:  186, btch:  31 usd:  70
Dec 16 13:12:16 cluster3 kernel: CPU    1: hi:  186, btch:  31 usd:  52
Dec 16 13:12:16 cluster3 kernel: CPU    2: hi:  186, btch:  31 usd:  92
Dec 16 13:12:16 cluster3 kernel: CPU    3: hi:  186, btch:  31 usd:   0
Dec 16 13:12:16 cluster3 kernel: CPU    4: hi:  186, btch:  31 usd:  51
Dec 16 13:12:16 cluster3 kernel: CPU    5: hi:  186, btch:  31 usd: 167
Dec 16 13:12:16 cluster3 kernel: CPU    6: hi:  186, btch:  31 usd: 172
Dec 16 13:12:16 cluster3 kernel: CPU    7: hi:  186, btch:  31 usd:   0
Dec 16 13:12:16 cluster3 kernel: Node 1 Normal per-cpu:
Dec 16 13:12:16 cluster3 kernel: CPU    0: hi:  186, btch:  31 usd:   0
Dec 16 13:12:16 cluster3 kernel: CPU    1: hi:  186, btch:  31 usd:   0
Dec 16 13:12:16 cluster3 kernel: CPU    2: hi:  186, btch:  31 usd:   0
Dec 16 13:12:16 cluster3 kernel: CPU    3: hi:  186, btch:  31 usd:   0
Dec 16 13:12:16 cluster3 kernel: CPU    4: hi:  186, btch:  31 usd:  61
Dec 16 13:12:16 cluster3 kernel: CPU    5: hi:  186, btch:  31 usd:   0
Dec 16 13:12:16 cluster3 kernel: CPU    6: hi:  186, btch:  31 usd:   0
Dec 16 13:12:16 cluster3 kernel: CPU    7: hi:  186, btch:  31 usd:   0
Dec 16 13:12:16 cluster3 kernel: active_anon:352 inactive_anon:1440 isolated_anon:0
Dec 16 13:12:16 cluster3 kernel: active_file:72 inactive_file:230 isolated_file:0
Dec 16 13:12:16 cluster3 kernel: unevictable:900568 dirty:16 writeback:1303 unstable:0
Dec 16 13:12:16 cluster3 kernel: free:24057 slab_reclaimable:2246 slab_unreclaimable:34152
Dec 16 13:12:16 cluster3 kernel: mapped:10062 shmem:8141 pagetables:3229 bounce:0
Dec 16 13:12:16 cluster3 kernel: Node 0 DMA free:8340kB min:332kB low:412kB high:496kB active_anon:24kB inactive_anon:12kB active_file:0kB inactive_file:12kB unevictable:6984kB isolated(anon):0kB isolated(file):0kB present:15336kB mlocked:852kB dirty:0kB writeback:52kB mapped:52kB shmem:0kB slab_reclaimable:32kB slab_unreclaimable:248kB kernel_stack:0kB pagetables:16kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:11 all_unreclaimable? no
Dec 16 13:12:16 cluster3 kernel: lowmem_reserve[]: 0 2004 2004 2004
Dec 16 13:12:16 cluster3 kernel: Node 0 DMA32 free:44956kB min:44880kB low:56100kB high:67320kB active_anon:1336kB inactive_anon:4780kB active_file:112kB inactive_file:96kB unevictable:1676312kB isolated(anon):0kB isolated(file):0kB present:2052320kB mlocked:456020kB dirty:0kB writeback:5160kB mapped:35864kB shmem:31976kB slab_reclaimable:3896kB slab_unreclaimable:63928kB kernel_stack:2056kB pagetables:5704kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:9568 all_unreclaimable? yes
Dec 16 13:12:16 cluster3 kernel: lowmem_reserve[]: 0 0 0 0
Dec 16 13:12:16 cluster3 kernel: Node 1 DMA32 free:25628kB min:22272kB low:27840kB high:33408kB active_anon:24kB inactive_anon:968kB active_file:160kB inactive_file:788kB unevictable:982624kB isolated(anon):0kB isolated(file):0kB present:1018496kB mlocked:171156kB dirty:48kB writeback:0kB mapped:164kB shmem:588kB slab_reclaimable:40kB slab_unreclaimable:884kB kernel_stack:8kB pagetables:1828kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:96 all_unreclaimable? yes
Dec 16 13:12:16 cluster3 kernel: lowmem_reserve[]: 0 0 1010 1010
Dec 16 13:12:16 cluster3 kernel: Node 1 Normal free:17304kB min:22616kB low:28268kB high:33924kB active_anon:24kB inactive_anon:0kB active_file:16kB inactive_file:24kB unevictable:936352kB isolated(anon):0kB isolated(file):0kB present:1034240kB mlocked:312932kB dirty:16kB writeback:0kB mapped:4168kB shmem:0kB slab_reclaimable:5016kB slab_unreclaimable:71548kB kernel_stack:320kB pagetables:5368kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:108 all_unreclaimable? yes
Dec 16 13:12:16 cluster3 kernel: lowmem_reserve[]: 0 0 0 0
Dec 16 13:12:16 cluster3 kernel: Node 0 DMA: 33*4kB 41*8kB 31*16kB 17*32kB 11*64kB 10*128kB 5*256kB 3*512kB 2*1024kB 0*2048kB 0*4096kB = 8348kB
Dec 16 13:12:16 cluster3 kernel: Node 0 DMA32: 531*4kB 342*8kB 169*16kB 121*32kB 82*64kB 45*128kB 27*256kB 12*512kB 9*1024kB 0*2048kB 0*4096kB = 44716kB
Dec 16 13:12:16 cluster3 kernel: Node 1 DMA32: 36*4kB 190*8kB 138*16kB 58*32kB 27*64kB 11*128kB 9*256kB 4*512kB 12*1024kB 0*2048kB 0*4096kB = 25504kB
Dec 16 13:12:16 cluster3 kernel: Node 1 Normal: 524*4kB 420*8kB 181*16kB 71*32kB 25*64kB 10*128kB 7*256kB 4*512kB 0*1024kB 0*2048kB 0*4096kB = 17344kB
Dec 16 13:12:16 cluster3 kernel: 11956 total pagecache pages
Dec 16 13:12:16 cluster3 kernel: 1469 pages in swap cache
Dec 16 13:12:16 cluster3 kernel: Swap cache stats: add 6894103, delete 6892634, find 3822548/5496695
Dec 16 13:12:16 cluster3 kernel: Free swap  = 25553140kB
Dec 16 13:12:16 cluster3 kernel: Total swap = 25624568kB
Dec 16 13:12:16 cluster3 kernel: 1048575 pages RAM
Dec 16 13:12:16 cluster3 kernel: 68025 pages reserved
Dec 16 13:12:16 cluster3 kernel: 22349 pages shared
Dec 16 13:12:16 cluster3 kernel: 924649 pages non-shared
Dec 16 13:12:16 cluster3 kernel: Out of memory: kill process 15944 (pickup) score 19656 or a child
Dec 16 13:12:16 cluster3 kernel: Killed process 15944 (pickup) vsz:78624kB, anon-rss:0kB, file-rss:1116kB
Dec 16 13:12:16 cluster3 kernel: dbus-daemon invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0
Dec 16 13:12:16 cluster3 kernel: dbus-daemon cpuset=/ mems_allowed=0-1
Dec 16 13:12:16 cluster3 kernel: Pid: 1984, comm: dbus-daemon Not tainted 2.6.32-131.0.15.el6.x86_64 #1
Dec 16 13:12:16 cluster3 kernel: Call Trace:
Dec 16 13:12:16 cluster3 kernel: [<ffffffff810c0101>] ? cpuset_print_task_mems_allowed+0x91/0xb0
Dec 16 13:12:16 cluster3 kernel: [<ffffffff811102bb>] ? oom_kill_process+0xcb/0x2e0
Dec 16 13:12:16 cluster3 kernel: [<ffffffff81110880>] ? select_bad_process+0xd0/0x110
Dec 16 13:12:16 cluster3 kernel: [<ffffffff81110918>] ? __out_of_memory+0x58/0xc0
Dec 16 13:12:16 cluster3 kernel: [<ffffffff81110b19>] ? out_of_memory+0x199/0x210
Dec 16 13:12:16 cluster3 kernel: [<ffffffff811202dd>] ? __alloc_pages_nodemask+0x80d/0x8b0
Dec 16 13:12:16 cluster3 kernel: [<ffffffff811875b0>] ? __pollwait+0x0/0xf0
Dec 16 13:12:16 cluster3 kernel: [<ffffffff8115474a>] ? alloc_pages_vma+0x9a/0x150
Dec 16 13:12:16 cluster3 kernel: [<ffffffff811876a0>] ? pollwake+0x0/0x60
Dec 16 13:12:16 cluster3 kernel: [<ffffffff81147102>] ? read_swap_cache_async+0xf2/0x150
Dec 16 13:12:16 cluster3 kernel: [<ffffffff81147b19>] ? valid_swaphandles+0x69/0x160
Dec 16 13:12:16 cluster3 kernel: [<ffffffff811471e7>] ? swapin_readahead+0x87/0xc0
Dec 16 13:12:16 cluster3 kernel: [<ffffffff81137cdb>] ? handle_pte_fault+0x70b/0xb50
Dec 16 13:12:16 cluster3 kernel: [<ffffffff811876a0>] ? pollwake+0x0/0x60
Dec 16 13:12:16 cluster3 kernel: [<ffffffff8140e4b0>] ? sock_aio_write+0x0/0x160
Dec 16 13:12:16 cluster3 kernel: [<ffffffff8100be2e>] ? call_function_single_interrupt+0xe/0x20
Dec 16 13:12:16 cluster3 kernel: [<ffffffff811382f8>] ? handle_mm_fault+0x1d8/0x2a0
Dec 16 13:12:16 cluster3 kernel: [<ffffffff81041529>] ? __do_page_fault+0x139/0x480
Dec 16 13:12:16 cluster3 kernel: [<ffffffff81211cff>] ? selinux_file_permission+0xbf/0x150
Dec 16 13:12:16 cluster3 kernel: [<ffffffff812051a6>] ? security_file_permission+0x16/0x20
Dec 16 13:12:16 cluster3 kernel: [<ffffffff8100bd0e>] ? invalidate_interrupt2+0xe/0x20
Dec 16 13:12:16 cluster3 kernel: [<ffffffff814e0c3e>] ? do_page_fault+0x3e/0xa0
Dec 16 13:12:16 cluster3 kernel: [<ffffffff814ddfe5>] ? page_fault+0x25/0x30
Dec 16 13:12:16 cluster3 kernel: Mem-Info:
Dec 16 13:12:16 cluster3 kernel: Node 0 DMA per-cpu:
Dec 16 13:12:16 cluster3 kernel: CPU    0: hi:    0, btch:   1 usd:   0
Dec 16 13:12:16 cluster3 kernel: CPU    1: hi:    0, btch:   1 usd:   0
Dec 16 13:12:16 cluster3 kernel: CPU    2: hi:    0, btch:   1 usd:   0
Dec 16 13:12:16 cluster3 kernel: CPU    3: hi:    0, btch:   1 usd:   0
Dec 16 13:12:16 cluster3 kernel: CPU    4: hi:    0, btch:   1 usd:   0
Dec 16 13:12:16 cluster3 kernel: CPU    5: hi:    0, btch:   1 usd:   0
Dec 16 13:12:16 cluster3 kernel: CPU    6: hi:    0, btch:   1 usd:   0
Dec 16 13:12:16 cluster3 kernel: CPU    7: hi:    0, btch:   1 usd:   0
Dec 16 13:12:16 cluster3 kernel: Node 0 DMA32 per-cpu:
Dec 16 13:12:16 cluster3 kernel: CPU    0: hi:  186, btch:  31 usd: 176
Dec 16 13:12:16 cluster3 kernel: CPU    1: hi:  186, btch:  31 usd:  35
Dec 16 13:12:16 cluster3 kernel: CPU    2: hi:  186, btch:  31 usd:  31
Dec 16 13:12:16 cluster3 kernel: CPU    3: hi:  186, btch:  31 usd:  48
Dec 16 13:12:16 cluster3 kernel: CPU    4: hi:  186, btch:  31 usd: 123
Dec 16 13:12:16 cluster3 kernel: CPU    5: hi:  186, btch:  31 usd:  23
Dec 16 13:12:16 cluster3 kernel: CPU    6: hi:  186, btch:  31 usd:  33
Dec 16 13:12:16 cluster3 kernel: CPU    7: hi:  186, btch:  31 usd:  30
Dec 16 13:12:16 cluster3 kernel: Node 1 DMA32 per-cpu:
Dec 16 13:12:16 cluster3 kernel: CPU    0: hi:  186, btch:  31 usd:  56
Dec 16 13:12:16 cluster3 kernel: CPU    1: hi:  186, btch:  31 usd:  52
Dec 16 13:12:16 cluster3 kernel: CPU    2: hi:  186, btch:  31 usd:  92
Dec 16 13:12:16 cluster3 kernel: CPU    3: hi:  186, btch:  31 usd:   0
Dec 16 13:12:16 cluster3 kernel: CPU    4: hi:  186, btch:  31 usd:  69
Dec 16 13:12:16 cluster3 kernel: CPU    5: hi:  186, btch:  31 usd: 167
Dec 16 13:12:16 cluster3 kernel: CPU    6: hi:  186, btch:  31 usd: 172
Dec 16 13:12:16 cluster3 kernel: CPU    7: hi:  186, btch:  31 usd:   0
Dec 16 13:12:16 cluster3 kernel: Node 1 Normal per-cpu:
Dec 16 13:12:16 cluster3 kernel: CPU    0: hi:  186, btch:  31 usd:   0
Dec 16 13:12:16 cluster3 kernel: CPU    1: hi:  186, btch:  31 usd:   0
Dec 16 13:12:16 cluster3 kernel: CPU    2: hi:  186, btch:  31 usd:   0
Dec 16 13:12:16 cluster3 kernel: CPU    3: hi:  186, btch:  31 usd:   0
Dec 16 13:12:16 cluster3 kernel: CPU    4: hi:  186, btch:  31 usd:  61
Dec 16 13:12:16 cluster3 kernel: CPU    5: hi:  186, btch:  31 usd:   0
Dec 16 13:12:16 cluster3 kernel: CPU    6: hi:  186, btch:  31 usd:   0
Dec 16 13:12:16 cluster3 kernel: CPU    7: hi:  186, btch:  31 usd:   0
Dec 16 13:12:16 cluster3 kernel: active_anon:433 inactive_anon:1612 isolated_anon:32
Dec 16 13:12:16 cluster3 kernel: active_file:71 inactive_file:321 isolated_file:0
Dec 16 13:12:16 cluster3 kernel: unevictable:900600 dirty:0 writeback:1375 unstable:0
Dec 16 13:12:16 cluster3 kernel: free:24040 slab_reclaimable:2246 slab_unreclaimable:34223
Dec 16 13:12:16 cluster3 kernel: mapped:10066 shmem:8074 pagetables:3272 bounce:0
Dec 16 13:12:16 cluster3 kernel: Node 0 DMA free:8348kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:36kB active_file:0kB inactive_file:44kB unevictable:6984kB isolated(anon):0kB isolated(file):0kB present:15336kB mlocked:852kB dirty:0kB writeback:36kB mapped:44kB shmem:0kB slab_reclaimable:32kB slab_unreclaimable:248kB kernel_stack:0kB pagetables:16kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:69 all_unreclaimable? yes
Dec 16 13:12:16 cluster3 kernel: lowmem_reserve[]: 0 2004 2004 2004
Dec 16 13:12:16 cluster3 kernel: Node 0 DMA32 free:44964kB min:44880kB low:56100kB high:67320kB active_anon:1700kB inactive_anon:5212kB active_file:92kB inactive_file:312kB unevictable:1676440kB isolated(anon):128kB isolated(file):0kB present:2052320kB mlocked:456172kB dirty:0kB writeback:5432kB mapped:35628kB shmem:31640kB slab_reclaimable:3896kB slab_unreclaimable:64084kB kernel_stack:2064kB pagetables:5880kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:224 all_unreclaimable? no
Dec 16 13:12:16 cluster3 kernel: lowmem_reserve[]: 0 0 0 0
Dec 16 13:12:16 cluster3 kernel: Node 1 DMA32 free:25504kB min:22272kB low:27840kB high:33408kB active_anon:32kB inactive_anon:1168kB active_file:168kB inactive_file:912kB unevictable:982624kB isolated(anon):0kB isolated(file):0kB present:1018496kB mlocked:171156kB dirty:0kB writeback:24kB mapped:424kB shmem:656kB slab_reclaimable:40kB slab_unreclaimable:1012kB kernel_stack:8kB pagetables:1820kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:96 all_unreclaimable? yes
Dec 16 13:12:16 cluster3 kernel: lowmem_reserve[]: 0 0 1010 1010
Dec 16 13:12:16 cluster3 kernel: Node 1 Normal free:17344kB min:22616kB low:28268kB high:33924kB active_anon:0kB inactive_anon:32kB active_file:24kB inactive_file:16kB unevictable:936352kB isolated(anon):0kB isolated(file):0kB present:1034240kB mlocked:312932kB dirty:0kB writeback:8kB mapped:4168kB shmem:0kB slab_reclaimable:5016kB slab_unreclaimable:71548kB kernel_stack:328kB pagetables:5372kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:108 all_unreclaimable? yes
Dec 16 13:12:16 cluster3 kernel: lowmem_reserve[]: 0 0 0 0
Dec 16 13:12:16 cluster3 kernel: Node 0 DMA: 33*4kB 41*8kB 31*16kB 17*32kB 11*64kB 10*128kB 5*256kB 3*512kB 2*1024kB 0*2048kB 0*4096kB = 8348kB
Dec 16 13:12:16 cluster3 kernel: Node 0 DMA32: 550*4kB 354*8kB 169*16kB 121*32kB 82*64kB 45*128kB 27*256kB 12*512kB 9*1024kB 0*2048kB 0*4096kB = 44888kB
Dec 16 13:12:16 cluster3 kernel: Node 1 DMA32: 36*4kB 190*8kB 138*16kB 58*32kB 27*64kB 11*128kB 9*256kB 4*512kB 12*1024kB 0*2048kB 0*4096kB = 25504kB
Dec 16 13:12:16 cluster3 kernel: Node 1 Normal: 524*4kB 420*8kB 181*16kB 71*32kB 25*64kB 10*128kB 7*256kB 4*512kB 0*1024kB 0*2048kB 0*4096kB = 17344kB
Dec 16 13:12:16 cluster3 kernel: 12047 total pagecache pages
Dec 16 13:12:16 cluster3 kernel: 1570 pages in swap cache
Dec 16 13:12:16 cluster3 kernel: Swap cache stats: add 6894219, delete 6892649, find 3822577/5496747
Dec 16 13:12:16 cluster3 kernel: Free swap  = 25553656kB
Dec 16 13:12:16 cluster3 kernel: Total swap = 25624568kB
Dec 16 13:12:16 cluster3 kernel: 1048575 pages RAM
Dec 16 13:12:16 cluster3 kernel: 68025 pages reserved
Dec 16 13:12:16 cluster3 kernel: 22033 pages shared
Dec 16 13:12:16 cluster3 kernel: 924679 pages non-shared
Dec 16 13:12:16 cluster3 kernel: Out of memory: kill process 2426 (rgmanager) score 5942 or a child
Dec 16 13:12:16 cluster3 kernel: Killed process 2427 (rgmanager) vsz:494812kB, anon-rss:56kB, file-rss:1568kB
Dec 16 13:12:16 cluster3 kernel: snmpd invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0
Dec 16 13:12:16 cluster3 kernel: snmpd cpuset=/ mems_allowed=0-1
Dec 16 13:12:16 cluster3 kernel: Pid: 2158, comm: snmpd Not tainted 2.6.32-131.0.15.el6.x86_64 #1
Dec 16 13:12:16 cluster3 kernel: Call Trace:
Dec 16 13:12:16 cluster3 kernel: [<ffffffff810c0101>] ? cpuset_print_task_mems_allowed+0x91/0xb0
Dec 16 13:12:16 cluster3 kernel: [<ffffffff811102bb>] ? oom_kill_process+0xcb/0x2e0
Dec 16 13:12:16 cluster3 kernel: [<ffffffff81110880>] ? select_bad_process+0xd0/0x110
Dec 16 13:12:16 cluster3 kernel: [<ffffffff81110918>] ? __out_of_memory+0x58/0xc0
Dec 16 13:12:16 cluster3 kernel: [<ffffffff81110b19>] ? out_of_memory+0x199/0x210
Dec 16 13:12:16 cluster3 kernel: [<ffffffff811202dd>] ? __alloc_pages_nodemask+0x80d/0x8b0
Dec 16 13:12:16 cluster3 kernel: [<ffffffff8115474a>] ? alloc_pages_vma+0x9a/0x150
Dec 16 13:12:16 cluster3 kernel: [<ffffffff81147102>] ? read_swap_cache_async+0xf2/0x150
Dec 16 13:12:16 cluster3 kernel: [<ffffffff81147b19>] ? valid_swaphandles+0x69/0x160
Dec 16 13:12:16 cluster3 kernel: [<ffffffff811471e7>] ? swapin_readahead+0x87/0xc0
Dec 16 13:12:16 cluster3 kernel: [<ffffffff81137cdb>] ? handle_pte_fault+0x70b/0xb50
Dec 16 13:12:16 cluster3 kernel: [<ffffffff814db337>] ? thread_return+0x4e/0x777
Dec 16 13:12:16 cluster3 kernel: [<ffffffff8100bc8e>] ? apic_timer_interrupt+0xe/0x20
Dec 16 13:12:16 cluster3 kernel: [<ffffffff811382f8>] ? handle_mm_fault+0x1d8/0x2a0
Dec 16 13:12:16 cluster3 kernel: [<ffffffff81103c56>] ? __perf_event_task_sched_out+0x36/0x50
Dec 16 13:12:16 cluster3 kernel: [<ffffffff81041529>] ? __do_page_fault+0x139/0x480
Dec 16 13:12:16 cluster3 kernel: [<ffffffff810500f0>] ? __dequeue_entity+0x30/0x50
Dec 16 13:12:16 cluster3 kernel: [<ffffffff810097bc>] ? __switch_to+0x1ac/0x320
Dec 16 13:12:16 cluster3 kernel: [<ffffffff814db337>] ? thread_return+0x4e/0x777
Dec 16 13:12:16 cluster3 kernel: [<ffffffff814e0c3e>] ? do_page_fault+0x3e/0xa0
Dec 16 13:12:16 cluster3 kernel: [<ffffffff814ddfe5>] ? page_fault+0x25/0x30
Dec 16 13:12:16 cluster3 kernel: Mem-Info:
Dec 16 13:12:16 cluster3 kernel: Node 0 DMA per-cpu:
Dec 16 13:12:16 cluster3 kernel: CPU    0: hi:    0, btch:   1 usd:   0
Dec 16 13:12:16 cluster3 kernel: CPU    1: hi:    0, btch:   1 usd:   0
Dec 16 13:12:16 cluster3 kernel: CPU    2: hi:    0, btch:   1 usd:   0
Dec 16 13:12:16 cluster3 kernel: CPU    3: hi:    0, btch:   1 usd:   0
Dec 16 13:12:16 cluster3 kernel: CPU    4: hi:    0, btch:   1 usd:   0
Dec 16 13:12:16 cluster3 kernel: CPU    5: hi:    0, btch:   1 usd:   0
Dec 16 13:12:16 cluster3 kernel: CPU    6: hi:    0, btch:   1 usd:   0
Dec 16 13:12:16 cluster3 kernel: CPU    7: hi:    0, btch:   1 usd:   0
Dec 16 13:12:16 cluster3 kernel: Node 0 DMA32 per-cpu:
Dec 16 13:12:16 cluster3 kernel: CPU    0: hi:  186, btch:  31 usd: 159
Dec 16 13:12:16 cluster3 kernel: CPU    1: hi:  186, btch:  31 usd:  35
Dec 16 13:12:16 cluster3 kernel: CPU    2: hi:  186, btch:  31 usd: 133
Dec 16 13:12:16 cluster3 kernel: CPU    3: hi:  186, btch:  31 usd:  48
Dec 16 13:12:16 cluster3 kernel: CPU    4: hi:  186, btch:  31 usd:  43
Dec 16 13:12:16 cluster3 kernel: CPU    5: hi:  186, btch:  31 usd:  10
Dec 16 13:12:16 cluster3 kernel: CPU    6: hi:  186, btch:  31 usd:  30
Dec 16 13:12:16 cluster3 kernel: CPU    7: hi:  186, btch:  31 usd:  44
Dec 16 13:12:16 cluster3 kernel: Node 1 DMA32 per-cpu:
Dec 16 13:12:16 cluster3 kernel: CPU    0: hi:  186, btch:  31 usd:  56
Dec 16 13:12:16 cluster3 kernel: CPU    1: hi:  186, btch:  31 usd:  52
Dec 16 13:12:16 cluster3 kernel: CPU    2: hi:  186, btch:  31 usd:  84
Dec 16 13:12:16 cluster3 kernel: CPU    3: hi:  186, btch:  31 usd:   0
Dec 16 13:12:16 cluster3 kernel: CPU    4: hi:  186, btch:  31 usd:  69
Dec 16 13:12:16 cluster3 kernel: CPU    5: hi:  186, btch:  31 usd: 167
Dec 16 13:12:16 cluster3 kernel: CPU    6: hi:  186, btch:  31 usd: 172
Dec 16 13:12:16 cluster3 kernel: CPU    7: hi:  186, btch:  31 usd:   0
Dec 16 13:12:16 cluster3 kernel: Node 1 Normal per-cpu:
Dec 16 13:12:16 cluster3 kernel: CPU    0: hi:  186, btch:  31 usd:   0
Dec 16 13:12:16 cluster3 kernel: CPU    1: hi:  186, btch:  31 usd:   0
Dec 16 13:12:16 cluster3 kernel: CPU    2: hi:  186, btch:  31 usd:  20
Dec 16 13:12:16 cluster3 kernel: CPU    3: hi:  186, btch:  31 usd:   0
Dec 16 13:12:16 cluster3 kernel: CPU    4: hi:  186, btch:  31 usd:  61
Dec 16 13:12:16 cluster3 kernel: CPU    5: hi:  186, btch:  31 usd:   0
Dec 16 13:12:16 cluster3 kernel: CPU    6: hi:  186, btch:  31 usd:   0
Dec 16 13:12:16 cluster3 kernel: CPU    7: hi:  186, btch:  31 usd:   0
Dec 16 13:12:16 cluster3 kernel: active_anon:433 inactive_anon:1539 isolated_anon:32
Dec 16 13:12:16 cluster3 kernel: active_file:112 inactive_file:236 isolated_file:0
Dec 16 13:12:16 cluster3 kernel: unevictable:900600 dirty:0 writeback:1314 unstable:0
Dec 16 13:12:16 cluster3 kernel: free:24166 slab_reclaimable:2246 slab_unreclaimable:34223
Dec 16 13:12:16 cluster3 kernel: mapped:10066 shmem:8074 pagetables:3272 bounce:0
Dec 16 13:12:16 cluster3 kernel: Node 0 DMA free:8348kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:36kB active_file:0kB inactive_file:44kB unevictable:6984kB isolated(anon):0kB isolated(file):0kB present:15336kB mlocked:852kB dirty:0kB writeback:36kB mapped:44kB shmem:0kB slab_reclaimable:32kB slab_unreclaimable:248kB kernel_stack:0kB pagetables:16kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:69 all_unreclaimable? yes
Dec 16 13:12:16 cluster3 kernel: lowmem_reserve[]: 0 2004 2004 2004
Dec 16 13:12:16 cluster3 kernel: Node 0 DMA32 free:45468kB min:44880kB low:56100kB high:67320kB active_anon:1700kB inactive_anon:4920kB active_file:256kB inactive_file:0kB unevictable:1676440kB isolated(anon):128kB isolated(file):0kB present:2052320kB mlocked:456172kB dirty:0kB writeback:5188kB mapped:35628kB shmem:31640kB slab_reclaimable:3896kB slab_unreclaimable:64084kB kernel_stack:2064kB pagetables:5880kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:199 all_unreclaimable? no
Dec 16 13:12:16 cluster3 kernel: lowmem_reserve[]: 0 0 0 0
Dec 16 13:12:16 cluster3 kernel: Node 1 DMA32 free:25504kB min:22272kB low:27840kB high:33408kB active_anon:32kB inactive_anon:1168kB active_file:168kB inactive_file:912kB unevictable:982624kB isolated(anon):0kB isolated(file):0kB present:1018496kB mlocked:171156kB dirty:0kB writeback:24kB mapped:424kB shmem:656kB slab_reclaimable:40kB slab_unreclaimable:1012kB kernel_stack:8kB pagetables:1820kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:96 all_unreclaimable? yes
Dec 16 13:12:16 cluster3 kernel: lowmem_reserve[]: 0 0 1010 1010
Dec 16 13:12:16 cluster3 kernel: Node 1 Normal free:17344kB min:22616kB low:28268kB high:33924kB active_anon:0kB inactive_anon:32kB active_file:24kB inactive_file:16kB unevictable:936352kB isolated(anon):0kB isolated(file):0kB present:1034240kB mlocked:312932kB dirty:0kB writeback:8kB mapped:4168kB shmem:0kB slab_reclaimable:5016kB slab_unreclaimable:71548kB kernel_stack:328kB pagetables:5372kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:120 all_unreclaimable? yes
Dec 16 13:12:16 cluster3 kernel: lowmem_reserve[]: 0 0 0 0
Dec 16 13:12:16 cluster3 kernel: Node 0 DMA: 33*4kB 41*8kB 31*16kB 17*32kB 11*64kB 10*128kB 5*256kB 3*512kB 2*1024kB 0*2048kB 0*4096kB = 8348kB
Dec 16 13:12:16 cluster3 kernel: Node 0 DMA32: 577*4kB 377*8kB 180*16kB 125*32kB 83*64kB 45*128kB 27*256kB 12*512kB 9*1024kB 0*2048kB 0*4096kB = 45548kB
Dec 16 13:12:16 cluster3 kernel: Node 1 DMA32: 36*4kB 190*8kB 138*16kB 58*32kB 27*64kB 11*128kB 9*256kB 4*512kB 12*1024kB 0*2048kB 0*4096kB = 25504kB
Dec 16 13:12:16 cluster3 kernel: Node 1 Normal: 524*4kB 421*8kB 181*16kB 71*32kB 25*64kB 10*128kB 7*256kB 4*512kB 0*1024kB 0*2048kB 0*4096kB = 17352kB
Dec 16 13:12:16 cluster3 kernel: 11925 total pagecache pages
Dec 16 13:12:16 cluster3 kernel: 1486 pages in swap cache
Dec 16 13:12:16 cluster3 kernel: Swap cache stats: add 6894351, delete 6892865, find 3822613/5496811
Dec 16 13:12:16 cluster3 kernel: Free swap  = 25554632kB
Dec 16 13:12:16 cluster3 kernel: Total swap = 25624568kB
Dec 16 13:12:16 cluster3 kernel: 1048575 pages RAM
Dec 16 13:12:16 cluster3 kernel: 68025 pages reserved
Dec 16 13:12:16 cluster3 kernel: 21648 pages shared
Dec 16 13:12:16 cluster3 kernel: 924249 pages non-shared
Dec 16 13:12:16 cluster3 kernel: Out of memory: kill process 2280 (qmgr) score 4924 or a child
Dec 16 13:12:16 cluster3 kernel: Killed process 2280 (qmgr) vsz:78796kB, anon-rss:0kB, file-rss:1164kB