Jul 30 07:08:08 dbserver kernel: 0kB isolated(file):0kB present:7514400kB mlocked:0kB dirty:84kB writeback:0kB mapped:724kB shmem:8kB slab_reclaimable:8652kB slab_unreclaimable:10988kB kernel_stack:1256kB pagetables:11880kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:416 all_unreclaimable? yes Jul 30 07:08:08 dbserver kernel: [40888.797019] lowmem_reserve[]: 0 0 0 0 Jul 30 07:08:08 dbserver kernel: [40888.797021] Node 0 DMA: 2*4kB 1*8kB 2*16kB 1*32kB 2*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15824kB Jul 30 07:08:08 dbserver kernel: [40888.797026] Node 0 DMA32: 15*4kB 24*8kB 7*16kB 4*32kB 0*64kB 2*128kB 0*256kB 1*512kB 3*1024kB 2*2048kB 6*4096kB = 33004kB Jul 30 07:08:08 dbserver kernel: [40888.797031] Node 0 Normal: 176*4kB 24*8kB 89*16kB 29*32kB 8*64kB 7*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 9008kB Jul 30 07:08:08 dbserver kernel: [40888.797035] 85036 total pagecache pages Jul 30 07:08:08 dbserver kernel: [40888.797036] 83905 pages in swap cache Jul 30 07:08:08 dbserver kernel: [40888.797038] Swap cache stats: add 166633, delete 82728, find 3836/5944 Jul 30 07:08:08 dbserver kernel: [40888.797038] Free swap = 0kB Jul 30 07:08:08 dbserver kernel: [40888.797039] Total swap = 577532kB Jul 30 07:08:08 dbserver kernel: [40888.812903] 2691056 pages RAM Jul 30 07:08:08 dbserver kernel: [40888.812905] 56870 pages reserved Jul 30 07:08:08 dbserver kernel: [40888.812905] 293 pages shared Jul 30 07:08:08 dbserver kernel: [40888.812906] 2618608 pages non-shared Jul 30 07:08:08 dbserver kernel: [40888.812908] Out of memory: kill process 7602 (mysqld) score 110721 or a child Jul 30 07:08:08 dbserver kernel: [40888.812973] Killed process 7602 (mysqld) vsz:6200404kB, anon-rss:4732512kB, file-rss:0kB Jul 30 07:08:08 dbserver kernel: [40888.817148] mysqld invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0 Jul 30 07:08:08 dbserver kernel: [40888.817151] mysqld cpuset=/ mems_allowed=0 Jul 30 07:08:08 dbserver kernel: [40888.817152] Pid: 7604, comm: mysqld Not tainted 2.6.35-22-generic #35-Ubuntu Jul 30 07:08:08 dbserver kernel: [40888.817154] Call Trace: Jul 30 07:08:08 dbserver kernel: [40888.817159] [] ? cpuset_print_task_mems_allowed+0x9d/0xb0 Jul 30 07:08:08 dbserver kernel: [40888.817164] [] dump_header+0x81/0xc0 Jul 30 07:08:08 dbserver kernel: [40888.817166] [] oom_kill_process+0x81/0x180 Jul 30 07:08:08 dbserver kernel: [40888.817168] [] __out_of_memory+0x58/0xd0 Jul 30 07:08:08 dbserver kernel: [40888.817169] [] out_of_memory+0x86/0x1c0 Jul 30 07:08:08 dbserver kernel: [40888.817172] [] __alloc_pages_slowpath+0x526/0x590 Jul 30 07:08:08 dbserver kernel: [40888.817174] [] __alloc_pages_nodemask+0x19a/0x1f0 Jul 30 07:08:08 dbserver kernel: [40888.817176] [] alloc_page_vma+0x89/0x140 Jul 30 07:08:08 dbserver kernel: [40888.817179] [] read_swap_cache_async+0xde/0x130 Jul 30 07:08:08 dbserver kernel: [40888.817181] [] ? valid_swaphandles+0x141/0x170 Jul 30 07:08:08 dbserver kernel: [40888.817183] [] swapin_readahead+0x87/0xc0 Jul 30 07:08:08 dbserver kernel: [40888.817185] [] do_swap_page+0x115/0x5a0 Jul 30 07:08:08 dbserver kernel: [40888.817188] [] ? __mem_cgroup_try_charge+0x71/0x450 Jul 30 07:08:08 dbserver kernel: [40888.817191] [] handle_mm_fault+0x2db/0x440 Jul 30 07:08:08 dbserver kernel: [40888.817194] [] ? find_busiest_group+0x757/0xa00 Jul 30 07:08:08 dbserver kernel: [40888.817197] [] do_page_fault+0x12a/0x350 Jul 30 07:08:08 dbserver kernel: [40888.817199] [] page_fault+0x25/0x30 Jul 30 07:08:08 dbserver kernel: [40888.817202] [] ? __get_user_8+0x1c/0x23 Jul 30 07:08:08 dbserver kernel: [40888.817205] [] ? exit_robust_list+0x5d/0x160 Jul 30 07:08:08 dbserver kernel: [40888.817208] [] ? default_spin_lock_flags+0x9/0x10 Jul 30 07:08:08 dbserver kernel: [40888.817211] [] mm_release+0x113/0x130 Jul 30 07:08:08 dbserver kernel: [40888.817213] [] exit_mm+0x26/0x140 Jul 30 07:08:08 dbserver kernel: [40888.817215] [] ? taskstats_exit+0x19d/0x260 Jul 30 07:08:08 dbserver kernel: [40888.817217] [] ? acct_collect+0x160/0x1b0 Jul 30 07:08:08 dbserver kernel: [40888.817219] [] do_exit+0x141/0x3f0 Jul 30 07:08:08 dbserver kernel: [40888.817222] [] ? lock_timer_base+0x3c/0x70 Jul 30 07:08:08 dbserver kernel: [40888.817224] [] do_group_exit+0x55/0xd0 Jul 30 07:08:08 dbserver kernel: [40888.817226] [] get_signal_to_deliver+0x201/0x440 Jul 30 07:08:08 dbserver kernel: [40888.817229] [] do_signal+0x69/0x1a0 Jul 30 07:08:08 dbserver kernel: [40888.817232] [] ? timeout_func+0x0/0x20 Jul 30 07:08:08 dbserver kernel: [40888.817234] [] do_notify_resume+0x65/0x90 Jul 30 07:08:08 dbserver kernel: [40888.817236] [] int_signal+0x12/0x17 Jul 30 07:08:08 dbserver kernel: [40888.817237] Mem-Info: Jul 30 07:08:08 dbserver kernel: [40888.817238] Node 0 DMA per-cpu: Jul 30 07:08:08 dbserver kernel: [40888.817240] CPU 0: hi: 0, btch: 1 usd: 0 Jul 30 07:08:08 dbserver kernel: [40888.817241] CPU 1: hi: 0, btch: 1 usd: 0 Jul 30 07:08:08 dbserver kernel: [40888.817242] CPU 2: hi: 0, btch: 1 usd: 0 Jul 30 07:08:08 dbserver kernel: [40888.817243] CPU 3: hi: 0, btch: 1 usd: 0 Jul 30 07:08:08 dbserver kernel: [40888.817244] Node 0 DMA32 per-cpu: Jul 30 07:08:08 dbserver kernel: [40888.817245] CPU 0: hi: 186, btch: 31 usd: 30 Jul 30 07:08:08 dbserver kernel: [40888.817246] CPU 1: hi: 186, btch: 31 usd: 91 Jul 30 07:08:08 dbserver kernel: [40888.817247] CPU 2: hi: 186, btch: 31 usd: 77 Jul 30 07:08:08 dbserver kernel: [40888.817249] CPU 3: hi: 186, btch: 31 usd: 178 Jul 30 07:08:08 dbserver kernel: [40888.817249] Node 0 Normal per-cpu: Jul 30 07:08:08 dbserver kernel: [40888.817251] CPU 0: hi: 186, btch: 31 usd: 36 Jul 30 07:08:08 dbserver kernel: [40888.817252] CPU 1: hi: 186, btch: 31 usd: 8 Jul 30 07:08:08 dbserver kernel: [40888.817253] CPU 2: hi: 186, btch: 31 usd: 78 Jul 30 07:08:08 dbserver kernel: [40888.817254] CPU 3: hi: 186, btch: 31 usd: 165 Jul 30 07:08:08 dbserver kernel: [40888.817256] active_anon:1118901 inactive_anon:148659 isolated_anon:0 Jul 30 07:08:08 dbserver kernel: [40888.817257] active_file:43 inactive_file:1138 isolated_file:0 Jul 30 07:08:08 dbserver kernel: [40888.817257] unevictable:0 dirty:21 writeback:0 unstable:0 Jul 30 07:08:08 dbserver kernel: [40888.817258] free:14490 slab_reclaimable:2163 slab_unreclaimable:2752 Jul 30 07:08:08 dbserver kernel: [40888.817258] mapped:211 shmem:2 pagetables:3138 bounce:0 Jul 30 07:08:08 dbserver kernel: [40888.817260] Node 0 DMA free:15824kB min:16kB low:20kB high:24kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15708kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jul 30 07:08:08 dbserver kernel: [40888.817265] lowmem_reserve[]: 0 3000 10338 10338 Jul 30 07:08:08 dbserver kernel: [40888.817267] Node 0 DMA32 free:33004kB min:3772kB low:4712kB high:5656kB active_anon:255152kB inactive_anon:67028kB active_file:88kB inactive_file:16kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3072096kB mlocked:0kB dirty:0kB writeback:0kB mapped:120kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:12kB kernel_stack:0kB pagetables:672kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:156 all_unreclaimable? yes Jul 30 07:08:08 dbserver kernel: [40888.817272] lowmem_reserve[]: 0 0 7338 7338 Jul 30 07:08:08 dbserver kernel: [40888.817274] Node 0 Normal free:9132kB min:9228kB low:11532kB high:13840kB active_anon:4220452kB inactive_anon:527608kB active_file:84kB inactive_file:4536kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:7514400kB mlocked:0kB dirty:84kB writeback:0kB mapped:724kB shmem:8kB slab_reclaimable:8652kB slab_unreclaimable:10988kB kernel_stack:1256kB pagetables:11880kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:576 all_unreclaimable? yes Jul 30 07:08:08 dbserver kernel: [40888.817279] lowmem_reserve[]: 0 0 0 0 Jul 30 07:08:08 dbserver kernel: [40888.817281] Node 0 DMA: 2*4kB 1*8kB 2*16kB 1*32kB 2*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15824kB Jul 30 07:08:08 dbserver kernel: [40888.817286] Node 0 DMA32: 15*4kB 24*8kB 7*16kB 4*32kB 0*64kB 2*128kB 0*256kB 1*512kB 3*1024kB 2*2048kB 6*4096kB = 33004kB Jul 30 07:08:08 dbserver kernel: [40888.817294] Node 0 Normal: 176*4kB 24*8kB 89*16kB 29*32kB 8*64kB 7*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 9008kB Jul 30 07:08:08 dbserver kernel: [40888.817301] 85036 total pagecache pages Jul 30 07:08:08 dbserver kernel: [40888.817302] 83899 pages in swap cache Jul 30 07:08:08 dbserver kernel: [40888.817304] Swap cache stats: add 166635, delete 82736, find 3838/5952 Jul 30 07:08:08 dbserver kernel: [40888.817306] Free swap = 0kB Jul 30 07:08:08 dbserver kernel: [40888.817307] Total swap = 577532kB Jul 30 07:08:08 dbserver kernel: [40888.831310] 2691056 pages RAM Jul 30 07:08:08 dbserver kernel: [40888.831313] 56870 pages reserved Jul 30 07:08:08 dbserver kernel: [40888.831314] 293 pages shared Jul 30 07:08:08 dbserver kernel: [40888.831315] 2618602 pages non-shared Jul 30 07:08:08 dbserver kernel: [40888.831318] Out of memory: kill process 749 (rsyslogd) score 15750 or a child Jul 30 07:08:08 dbserver kernel: [40888.831375] Killed process 749 (rsyslogd) vsz:126004kB, anon-rss:72kB, file-rss:0kB Jul 30 07:08:08 dbserver kernel: [40888.836877] mysqld invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0 Jul 30 07:08:08 dbserver kernel: [40888.836879] mysqld cpuset=/ mems_allowed=0 Jul 30 07:08:08 dbserver kernel: [40888.836881] Pid: 7604, comm: mysqld Not tainted 2.6.35-22-generic #35-Ubuntu Jul 30 07:08:08 dbserver kernel: [40888.836882] Call Trace: Jul 30 07:08:08 dbserver kernel: [40888.836888] [] ? cpuset_print_task_mems_allowed+0x9d/0xb0 Jul 30 07:08:08 dbserver kernel: [40888.836891] [] dump_header+0x81/0xc0 Jul 30 07:08:08 dbserver kernel: [40888.836893] [] oom_kill_process+0x81/0x180 Jul 30 07:08:08 dbserver kernel: [40888.836894] [] __out_of_memory+0x58/0xd0 Jul 30 07:08:08 dbserver kernel: [40888.836896] [] out_of_memory+0x86/0x1c0 Jul 30 07:08:08 dbserver kernel: [40888.836899] [] __alloc_pages_slowpath+0x526/0x590 Jul 30 07:08:08 dbserver kernel: [40888.836902] [] __alloc_pages_nodemask+0x19a/0x1f0 Jul 30 07:08:08 dbserver kernel: [40888.836906] [] alloc_page_vma+0x89/0x140 Jul 30 07:08:08 dbserver kernel: [40888.836910] [] read_swap_cache_async+0xde/0x130 Jul 30 07:08:08 dbserver kernel: [40888.836913] [] ? valid_swaphandles+0x141/0x170 Jul 30 07:08:08 dbserver kernel: [40888.836916] [] swapin_readahead+0x87/0xc0 Jul 30 07:08:08 dbserver kernel: [40888.836920] [] do_swap_page+0x115/0x5a0 Jul 30 07:08:08 dbserver kernel: [40888.836924] [] ? __mem_cgroup_try_charge+0x71/0x450 Jul 30 07:08:08 dbserver kernel: [40888.836927] [] handle_mm_fault+0x2db/0x440 Jul 30 07:08:08 dbserver kernel: [40888.836930] [] ? find_busiest_group+0x757/0xa00 Jul 30 07:08:08 dbserver kernel: [40888.836934] [] do_page_fault+0x12a/0x350 Jul 30 07:08:08 dbserver kernel: [40888.836936] [] page_fault+0x25/0x30 Jul 30 07:08:08 dbserver kernel: [40888.836939] [] ? __get_user_8+0x1c/0x23 Jul 30 07:08:08 dbserver kernel: [40888.836942] [] ? exit_robust_list+0x5d/0x160 Jul 30 07:08:08 dbserver kernel: [40888.836945] [] ? default_spin_lock_flags+0x9/0x10 Jul 30 07:08:08 dbserver kernel: [40888.836947] [] mm_release+0x113/0x130 Jul 30 07:08:08 dbserver kernel: [40888.836949] [] exit_mm+0x26/0x140 Jul 30 07:08:08 dbserver kernel: [40888.836952] [] ? taskstats_exit+0x19d/0x260 Jul 30 07:08:08 dbserver kernel: [40888.836953] [] ? acct_collect+0x160/0x1b0 Jul 30 07:08:08 dbserver kernel: [40888.836955] [] do_exit+0x141/0x3f0 Jul 30 07:08:08 dbserver kernel: [40888.836958] [] ? lock_timer_base+0x3c/0x70 Jul 30 07:08:08 dbserver kernel: [40888.836961] [] do_group_exit+0x55/0xd0 Jul 30 07:08:08 dbserver kernel: [40888.836962] [] get_signal_to_deliver+0x201/0x440 Jul 30 07:08:08 dbserver kernel: [40888.836965] [] do_signal+0x69/0x1a0 Jul 30 07:08:08 dbserver kernel: [40888.836969] [] ? timeout_func+0x0/0x20 Jul 30 07:08:08 dbserver kernel: [40888.836971] [] do_notify_resume+0x65/0x90 Jul 30 07:08:08 dbserver kernel: [40888.836973] [] int_signal+0x12/0x17 Jul 30 07:08:08 dbserver kernel: [40888.836974] Mem-Info: Jul 30 07:08:08 dbserver kernel: [40888.836975] Node 0 DMA per-cpu: Jul 30 07:08:08 dbserver kernel: [40888.836977] CPU 0: hi: 0, btch: 1 usd: 0 Jul 30 07:08:08 dbserver kernel: [40888.836978] CPU 1: hi: 0, btch: 1 usd: 0 Jul 30 07:08:08 dbserver kernel: [40888.836979] CPU 2: hi: 0, btch: 1 usd: 0 Jul 30 07:08:08 dbserver kernel: [40888.836980] CPU 3: hi: 0, btch: 1 usd: 0 Jul 30 07:08:08 dbserver kernel: [40888.836981] Node 0 DMA32 per-cpu: Jul 30 07:08:08 dbserver kernel: [40888.836982] CPU 0: hi: 186, btch: 31 usd: 30 Jul 30 07:08:08 dbserver kernel: [40888.836983] CPU 1: hi: 186, btch: 31 usd: 91 Jul 30 07:08:08 dbserver kernel: [40888.836984] CPU 2: hi: 186, btch: 31 usd: 77 Jul 30 07:08:08 dbserver kernel: [40888.836985] CPU 3: hi: 186, btch: 31 usd: 178 Jul 30 07:08:08 dbserver kernel: [40888.836986] Node 0 Normal per-cpu: Jul 30 07:08:08 dbserver kernel: [40888.836987] CPU 0: hi: 186, btch: 31 usd: 36 Jul 30 07:08:08 dbserver kernel: [40888.836988] CPU 1: hi: 186, btch: 31 usd: 8 Jul 30 07:08:08 dbserver kernel: [40888.836989] CPU 2: hi: 186, btch: 31 usd: 78 Jul 30 07:08:08 dbserver kernel: [40888.836990] CPU 3: hi: 186, btch: 31 usd: 165 Jul 30 07:08:08 dbserver kernel: [40888.836993] active_anon:1118901 inactive_anon:148659 isolated_anon:0 Jul 30 07:08:08 dbserver kernel: [40888.836994] active_file:43 inactive_file:1138 isolated_file:0 Jul 30 07:08:08 dbserver kernel: [40888.836994] unevictable:0 dirty:21 writeback:0 unstable:0 Jul 30 07:08:08 dbserver kernel: [40888.836995] free:14490 slab_reclaimable:2163 slab_unreclaimable:2752 Jul 30 07:08:08 dbserver kernel: [40888.836995] mapped:211 shmem:2 pagetables:3138 bounce:0 Jul 30 07:08:08 dbserver kernel: [40888.836996] Node 0 DMA free:15824kB min:16kB low:20kB high:24kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15708kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jul 30 07:08:08 dbserver kernel: [40888.837002] lowmem_reserve[]: 0 3000 10338 10338 Jul 30 07:08:08 dbserver kernel: [40888.837004] Node 0 DMA32 free:33004kB min:3772kB low:4712kB high:5656kB active_anon:255152kB inactive_anon:67028kB active_file:88kB inactive_file:16kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3072096kB mlocked:0kB dirty:0kB writeback:0kB mapped:120kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:12kB kernel_stack:0kB pagetables:672kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:156 all_unreclaimable? yes Jul 30 07:08:08 dbserver kernel: [40888.837009] lowmem_reserve[]: 0 0 7338 7338 Jul 30 07:08:08 dbserver kernel: [40888.837011] Node 0 Normal free:9132kB min:9228kB low:11532kB high:13840kB active_anon:4220452kB inactive_anon:527608kB active_file:84kB inactive_file:4536kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:7514400kB mlocked:0kB dirty:84kB writeback:0kB mapped:724kB shmem:8kB slab_reclaimable:8652kB slab_unreclaimable:10988kB kernel_stack:1256kB pagetables:11880kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:576 all_unreclaimable? yes Jul 30 07:08:08 dbserver kernel: [40888.837016] lowmem_reserve[]: 0 0 0 0 Jul 30 07:08:08 dbserver kernel: [40888.837018] Node 0 DMA: 2*4kB 1*8kB 2*16kB 1*32kB 2*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15824kB Jul 30 07:08:08 dbserver kernel: [40888.837023] Node 0 DMA32: 15*4kB 24*8kB 7*16kB 4*32kB 0*64kB 2*128kB 0*256kB 1*512kB 3*1024kB 2*2048kB 6*4096kB = 33004kB Jul 30 07:08:08 dbserver kernel: [40888.837027] Node 0 Normal: 176*4kB 24*8kB 89*16kB 29*32kB 8*64kB 7*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 9008kB Jul 30 07:08:08 dbserver kernel: [40888.837032] 85036 total pagecache pages Jul 30 07:08:08 dbserver kernel: [40888.837033] 83899 pages in swap cache Jul 30 07:08:08 dbserver kernel: [40888.837034] Swap cache stats: add 166635, delete 82736, find 3838/5952 Jul 30 07:08:08 dbserver kernel: [40888.837035] Free swap = 0kB Jul 30 07:08:08 dbserver kernel: [40888.837036] Total swap = 577532kB Jul 30 07:08:08 dbserver kernel: [40888.851031] 2691056 pages RAM Jul 30 07:08:08 dbserver kernel: [40888.851033] 56870 pages reserved Jul 30 07:08:08 dbserver kernel: [40888.851034] 296 pages shared Jul 30 07:08:08 dbserver kernel: [40888.851035] 2618439 pages non-shared Jul 30 07:08:08 dbserver kernel: [40888.851037] Out of memory: kill process 799 (atd) score 2366 or a child Jul 30 07:08:08 dbserver kernel: [40888.851093] Killed process 799 (atd) vsz:18932kB, anon-rss:0kB, file-rss:0kB