Mar 9 14:47:39 IP-FAIL-OVER kernel: [97761.714523] Mem-Info: Mar 9 14:47:39 IP-FAIL-OVER kernel: [97761.714525] Node 0 DMA per-cpu: Mar 9 14:47:39 IP-FAIL-OVER kernel: [97761.714529] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 14:47:39 IP-FAIL-OVER kernel: [97761.714532] Node 0 DMA32 per-cpu: Mar 9 14:47:39 IP-FAIL-OVER kernel: [97761.714536] CPU 0: hi: 186, btch: 31 usd: 30 Mar 9 14:47:39 IP-FAIL-OVER kernel: [97761.714542] active_anon:111761 inactive_anon:111809 isolated_anon:4416 Mar 9 14:47:39 IP-FAIL-OVER kernel: [97761.714543] active_file:166 inactive_file:219 isolated_file:0 Mar 9 14:47:39 IP-FAIL-OVER kernel: [97761.714545] unevictable:0 dirty:0 writeback:287 unstable:0 Mar 9 14:47:39 IP-FAIL-OVER kernel: [97761.714546] free:1956 slab_reclaimable:1369 slab_unreclaimable:2966 Mar 9 14:47:39 IP-FAIL-OVER kernel: [97761.714547] mapped:164 shmem:11 pagetables:11325 bounce:0 Mar 9 14:47:39 IP-FAIL-OVER kernel: [97761.714551] Node 0 DMA free:3928kB min:52kB low:64kB high:76kB active_anon:4832kB inactive_anon:4944kB active_file:20kB inactive_file:64kB unevictable:0kB isolated(anon):128kB isolated(file):0kB present:13884kB mlocked:0kB dirty:0kB writeback:8kB mapped:16kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:32kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 14:47:39 IP-FAIL-OVER kernel: [97761.714564] lowmem_reserve[]: 0 970 970 970 Mar 9 14:47:39 IP-FAIL-OVER kernel: [97761.714572] Node 0 DMA32 free:3896kB min:3956kB low:4944kB high:5932kB active_anon:442212kB inactive_anon:442292kB active_file:644kB inactive_file:812kB unevictable:0kB isolated(anon):17536kB isolated(file):0kB present:993840kB mlocked:0kB dirty:0kB writeback:1140kB mapped:640kB shmem:44kB slab_reclaimable:5476kB slab_unreclaimable:11864kB kernel_stack:1624kB pagetables:45268kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:480 all_unreclaimable? no Mar 9 14:47:39 IP-FAIL-OVER kernel: [97761.714586] lowmem_reserve[]: 0 0 0 0 Mar 9 14:47:39 IP-FAIL-OVER kernel: [97761.714593] Node 0 DMA: 3*4kB 0*8kB 1*16kB 0*32kB 1*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3932kB Mar 9 14:47:39 IP-FAIL-OVER kernel: [97761.714610] Node 0 DMA32: 438*4kB 8*8kB 4*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 3896kB Mar 9 14:47:39 IP-FAIL-OVER kernel: [97761.714627] 35001 total pagecache pages Mar 9 14:47:39 IP-FAIL-OVER kernel: [97761.714629] 34600 pages in swap cache Mar 9 14:47:39 IP-FAIL-OVER kernel: [97761.714632] Swap cache stats: add 1913526, delete 1878926, find 141163/292198 Mar 9 14:47:39 IP-FAIL-OVER kernel: [97761.714635] Free swap = 0kB Mar 9 14:47:39 IP-FAIL-OVER kernel: [97761.714637] Total swap = 1947640kB Mar 9 14:47:39 IP-FAIL-OVER kernel: [97761.716057] 256000 pages RAM Mar 9 14:47:39 IP-FAIL-OVER kernel: [97761.716057] 6399 pages reserved Mar 9 14:47:39 IP-FAIL-OVER kernel: [97761.716057] 58079 pages shared Mar 9 14:47:39 IP-FAIL-OVER kernel: [97761.716057] 242478 pages non-shared Mar 9 14:47:39 IP-FAIL-OVER kernel: [97761.716057] Out of memory: kill process 559 (apache2) score 60692 or a child Mar 9 14:47:39 IP-FAIL-OVER kernel: [97761.716057] Killed process 9198 (apache2) Mar 9 14:47:39 IP-FAIL-OVER kernel: [97774.993718] IPv6 addrconf: prefix with wrong length 56 Mar 9 14:47:39 IP-FAIL-OVER kernel: [97829.168225] apache2 invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0 Mar 9 14:47:39 IP-FAIL-OVER kernel: [97829.168229] apache2 cpuset=/ mems_allowed=0 Mar 9 14:47:39 IP-FAIL-OVER kernel: [97829.168232] Pid: 9487, comm: apache2 Not tainted 2.6.32-5-amd64 #1 Mar 9 14:47:39 IP-FAIL-OVER kernel: [97829.168234] Call Trace: Mar 9 14:47:39 IP-FAIL-OVER kernel: [97829.168240] [] ? oom_kill_process+0x7f/0x23f Mar 9 14:47:39 IP-FAIL-OVER kernel: [97829.168243] [] ? __out_of_memory+0x12a/0x141 Mar 9 14:47:39 IP-FAIL-OVER kernel: [97829.168246] [] ? out_of_memory+0x140/0x172 Mar 9 14:47:39 IP-FAIL-OVER kernel: [97829.168249] [] ? __alloc_pages_nodemask+0x4ec/0x5fc Mar 9 14:47:39 IP-FAIL-OVER kernel: [97829.168253] [] ? __do_page_cache_readahead+0x9b/0x1b4 Mar 9 14:47:39 IP-FAIL-OVER kernel: [97829.168257] [] ? check_events+0x12/0x20 Mar 9 14:47:39 IP-FAIL-OVER kernel: [97829.168260] [] ? check_events+0x12/0x20 Mar 9 14:47:39 IP-FAIL-OVER kernel: [97829.168262] [] ? ra_submit+0x1c/0x20 Mar 9 14:47:39 IP-FAIL-OVER kernel: [97829.168266] [] ? filemap_fault+0x17d/0x2f6 Mar 9 14:47:39 IP-FAIL-OVER kernel: [97829.168270] [] ? __do_fault+0x54/0x3c3 Mar 9 14:47:39 IP-FAIL-OVER kernel: [97829.168274] [] ? __wait_on_bit_lock+0x76/0x84 Mar 9 14:47:39 IP-FAIL-OVER kernel: [97829.168277] [] ? __raw_callee_save_xen_pud_val+0x11/0x1e Mar 9 14:47:39 IP-FAIL-OVER kernel: [97829.168281] [] ? handle_mm_fault+0x3b8/0x80f Mar 9 14:47:39 IP-FAIL-OVER kernel: [97829.168285] [] ? cap_cred_commit+0x0/0x1 Mar 9 14:47:39 IP-FAIL-OVER kernel: [97829.168289] [] ? do_page_fault+0x2e0/0x2fc Mar 9 14:47:39 IP-FAIL-OVER kernel: [97829.168292] [] ? page_fault+0x25/0x30 Mar 9 14:47:39 IP-FAIL-OVER kernel: [97829.168294] Mem-Info: Mar 9 14:47:39 IP-FAIL-OVER kernel: [97829.168295] Node 0 DMA per-cpu: Mar 9 14:47:39 IP-FAIL-OVER kernel: [97829.168298] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 14:47:39 IP-FAIL-OVER kernel: [97829.168299] Node 0 DMA32 per-cpu: Mar 9 14:47:39 IP-FAIL-OVER kernel: [97829.168302] CPU 0: hi: 186, btch: 31 usd: 19 Mar 9 14:47:39 IP-FAIL-OVER kernel: [97829.168305] active_anon:111812 inactive_anon:111870 isolated_anon:4448 Mar 9 14:47:39 IP-FAIL-OVER kernel: [97829.168306] active_file:58 inactive_file:97 isolated_file:0 Mar 9 14:47:39 IP-FAIL-OVER kernel: [97829.168307] unevictable:0 dirty:0 writeback:274 unstable:0 Mar 9 14:47:39 IP-FAIL-OVER kernel: [97829.168308] free:1999 slab_reclaimable:1370 slab_unreclaimable:2982 Mar 9 14:47:39 IP-FAIL-OVER kernel: [97829.168308] mapped:57 shmem:11 pagetables:11386 bounce:0 Mar 9 14:47:39 IP-FAIL-OVER kernel: [97829.168310] Node 0 DMA free:3924kB min:52kB low:64kB high:76kB active_anon:4316kB inactive_anon:4484kB active_file:12kB inactive_file:0kB unevictable:0kB isolated(anon):1152kB isolated(file):0kB present:13884kB mlocked:0kB dirty:0kB writeback:52kB mapped:16kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:40kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 14:47:39 IP-FAIL-OVER kernel: [97829.168318] lowmem_reserve[]: 0 970 970 970 Mar 9 14:47:39 IP-FAIL-OVER kernel: [97829.168322] Node 0 DMA32 free:4072kB min:3956kB low:4944kB high:5932kB active_anon:442932kB inactive_anon:442996kB active_file:220kB inactive_file:388kB unevictable:0kB isolated(anon):16640kB isolated(file):0kB present:993840kB mlocked:0kB dirty:0kB writeback:1044kB mapped:212kB shmem:44kB slab_reclaimable:5480kB slab_unreclaimable:11928kB kernel_stack:1664kB pagetables:45504kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:32 all_unreclaimable? no Mar 9 14:47:39 IP-FAIL-OVER kernel: [97829.168331] lowmem_reserve[]: 0 0 0 0 Mar 9 14:47:39 IP-FAIL-OVER kernel: [97829.168334] Node 0 DMA: 1*4kB 0*8kB 1*16kB 0*32kB 1*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3924kB Mar 9 14:47:39 IP-FAIL-OVER kernel: [97829.168344] Node 0 DMA32: 510*4kB 2*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 4072kB Mar 9 14:47:39 IP-FAIL-OVER kernel: [97829.168354] 39588 total pagecache pages Mar 9 14:47:39 IP-FAIL-OVER kernel: [97829.168356] 39411 pages in swap cache Mar 9 14:47:39 IP-FAIL-OVER kernel: [97829.168357] Swap cache stats: add 1957334, delete 1917923, find 143598/299180 Mar 9 14:47:46 IP-FAIL-OVER kernel: [97829.168359] Free swap = 0kB Mar 9 14:47:46 IP-FAIL-OVER kernel: [97829.168360] Total swap = 1947640kB Mar 9 14:47:46 IP-FAIL-OVER kernel: [97829.170510] 256000 pages RAM Mar 9 14:47:46 IP-FAIL-OVER kernel: [97829.170512] 6399 pages reserved Mar 9 14:47:46 IP-FAIL-OVER kernel: [97829.170513] 52764 pages shared Mar 9 14:47:46 IP-FAIL-OVER kernel: [97829.170514] 242444 pages non-shared Mar 9 14:47:46 IP-FAIL-OVER kernel: [97829.170517] Out of memory: kill process 559 (apache2) score 60637 or a child Mar 9 14:47:46 IP-FAIL-OVER kernel: [97829.170527] Killed process 9199 (apache2) Mar 9 14:47:58 IP-FAIL-OVER kernel: [97867.432320] apache2 invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0 Mar 9 14:47:59 IP-FAIL-OVER kernel: [97867.432323] apache2 cpuset=/ mems_allowed=0 Mar 9 14:47:59 IP-FAIL-OVER kernel: [97867.432326] Pid: 9203, comm: apache2 Not tainted 2.6.32-5-amd64 #1 Mar 9 14:47:59 IP-FAIL-OVER kernel: [97867.432328] Call Trace: Mar 9 14:47:59 IP-FAIL-OVER kernel: [97867.432334] [] ? oom_kill_process+0x7f/0x23f Mar 9 14:47:59 IP-FAIL-OVER kernel: [97867.432337] [] ? __out_of_memory+0x12a/0x141 Mar 9 14:47:59 IP-FAIL-OVER kernel: [97867.432340] [] ? out_of_memory+0x140/0x172 Mar 9 14:47:59 IP-FAIL-OVER kernel: [97867.432343] [] ? __alloc_pages_nodemask+0x4ec/0x5fc Mar 9 14:47:59 IP-FAIL-OVER kernel: [97867.432347] [] ? __do_page_cache_readahead+0x9b/0x1b4 Mar 9 14:47:59 IP-FAIL-OVER kernel: [97867.432349] [] ? ra_submit+0x1c/0x20 Mar 9 14:47:59 IP-FAIL-OVER kernel: [97867.432352] [] ? filemap_fault+0x17d/0x2f6 Mar 9 14:47:59 IP-FAIL-OVER kernel: [97867.432356] [] ? __do_fault+0x54/0x3c3 Mar 9 14:47:59 IP-FAIL-OVER kernel: [97867.432360] [] ? __wait_on_bit_lock+0x76/0x84 Mar 9 14:47:59 IP-FAIL-OVER kernel: [97867.432365] [] ? __raw_callee_save_xen_pud_val+0x11/0x1e Mar 9 14:47:59 IP-FAIL-OVER kernel: [97867.432368] [] ? handle_mm_fault+0x3b8/0x80f Mar 9 14:47:59 IP-FAIL-OVER kernel: [97867.432371] [] ? _atomic_dec_and_lock+0x33/0x50 Mar 9 14:47:59 IP-FAIL-OVER kernel: [97867.432374] [] ? do_page_fault+0x2e0/0x2fc Mar 9 14:47:59 IP-FAIL-OVER kernel: [97867.432377] [] ? page_fault+0x25/0x30 Mar 9 14:47:59 IP-FAIL-OVER kernel: [97867.432379] Mem-Info: Mar 9 14:47:59 IP-FAIL-OVER kernel: [97867.432380] Node 0 DMA per-cpu: Mar 9 14:47:59 IP-FAIL-OVER kernel: [97867.432383] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 14:47:59 IP-FAIL-OVER kernel: [97867.432384] Node 0 DMA32 per-cpu: Mar 9 14:47:59 IP-FAIL-OVER kernel: [97867.432386] CPU 0: hi: 186, btch: 31 usd: 78 Mar 9 14:47:59 IP-FAIL-OVER kernel: [97867.432390] active_anon:112244 inactive_anon:112482 isolated_anon:2848 Mar 9 14:47:59 IP-FAIL-OVER kernel: [97867.432391] active_file:27 inactive_file:427 isolated_file:0 Mar 9 14:47:59 IP-FAIL-OVER kernel: [97867.432391] unevictable:0 dirty:0 writeback:218 unstable:0 Mar 9 14:47:59 IP-FAIL-OVER kernel: [97867.432392] free:2190 slab_reclaimable:1366 slab_unreclaimable:2985 Mar 9 14:47:59 IP-FAIL-OVER kernel: [97867.432393] mapped:25 shmem:11 pagetables:11375 bounce:0 Mar 9 14:47:59 IP-FAIL-OVER kernel: [97867.432395] Node 0 DMA free:3936kB min:52kB low:64kB high:76kB active_anon:4384kB inactive_anon:5328kB active_file:56kB inactive_file:248kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13884kB mlocked:0kB dirty:0kB writeback:8kB mapped:64kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:40kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:3766 all_unreclaimable? no Mar 9 14:47:59 IP-FAIL-OVER kernel: [97867.432402] lowmem_reserve[]: 0 970 970 970 Mar 9 14:47:59 IP-FAIL-OVER kernel: [97867.432406] Node 0 DMA32 free:4824kB min:3956kB low:4944kB high:5932kB active_anon:444592kB inactive_anon:444600kB active_file:52kB inactive_file:1460kB unevictable:0kB isolated(anon):11392kB isolated(file):0kB present:993840kB mlocked:0kB dirty:0kB writeback:864kB mapped:36kB shmem:44kB slab_reclaimable:5464kB slab_unreclaimable:11932kB kernel_stack:1672kB pagetables:45460kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:24305 all_unreclaimable? no Mar 9 14:47:59 IP-FAIL-OVER kernel: [97867.432414] lowmem_reserve[]: 0 0 0 0 Mar 9 14:47:59 IP-FAIL-OVER kernel: [97867.432418] Node 0 DMA: 2*4kB 1*8kB 1*16kB 0*32kB 1*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3936kB Mar 9 14:47:59 IP-FAIL-OVER kernel: [97867.432427] Node 0 DMA32: 624*4kB 35*8kB 2*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 4824kB Mar 9 14:47:59 IP-FAIL-OVER kernel: [97867.432437] 43570 total pagecache pages Mar 9 14:47:59 IP-FAIL-OVER kernel: [97867.432438] 43102 pages in swap cache Mar 9 14:47:59 IP-FAIL-OVER kernel: [97867.432440] Swap cache stats: add 1984029, delete 1940927, find 145160/303304 Mar 9 14:47:59 IP-FAIL-OVER kernel: [97867.432441] Free swap = 0kB Mar 9 14:47:59 IP-FAIL-OVER kernel: [97867.432442] Total swap = 1947640kB Mar 9 14:47:59 IP-FAIL-OVER kernel: [97867.434608] 256000 pages RAM Mar 9 14:47:59 IP-FAIL-OVER kernel: [97867.434610] 6399 pages reserved Mar 9 14:47:59 IP-FAIL-OVER kernel: [97867.434611] 51668 pages shared Mar 9 14:47:59 IP-FAIL-OVER kernel: [97867.434613] 243672 pages non-shared Mar 9 14:47:59 IP-FAIL-OVER kernel: [97867.434616] Out of memory: kill process 559 (apache2) score 60593 or a child Mar 9 14:47:59 IP-FAIL-OVER kernel: [97867.434628] Killed process 9200 (apache2) Mar 9 14:50:03 IP-FAIL-OVER kernel: [97975.505748] IPv6 addrconf: prefix with wrong length 56 Mar 9 14:53:05 IP-FAIL-OVER kernel: [98176.200479] IPv6 addrconf: prefix with wrong length 56 Mar 9 14:53:39 IP-FAIL-OVER kernel: [98210.524660] apache2 invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0 Mar 9 14:53:40 IP-FAIL-OVER kernel: [98210.524667] apache2 cpuset=/ mems_allowed=0 Mar 9 14:53:40 IP-FAIL-OVER kernel: [98210.524671] Pid: 9487, comm: apache2 Not tainted 2.6.32-5-amd64 #1 Mar 9 14:53:40 IP-FAIL-OVER kernel: [98210.524674] Call Trace: Mar 9 14:53:40 IP-FAIL-OVER kernel: [98210.524684] [] ? oom_kill_process+0x7f/0x23f Mar 9 14:53:40 IP-FAIL-OVER kernel: [98210.524689] [] ? __out_of_memory+0x12a/0x141 Mar 9 14:53:40 IP-FAIL-OVER kernel: [98210.524694] [] ? out_of_memory+0x140/0x172 Mar 9 14:53:40 IP-FAIL-OVER kernel: [98210.524699] [] ? __alloc_pages_nodemask+0x4ec/0x5fc Mar 9 14:53:40 IP-FAIL-OVER kernel: [98210.524706] [] ? io_schedule+0x93/0xb7 Mar 9 14:53:40 IP-FAIL-OVER kernel: [98210.524712] [] ? __do_page_cache_readahead+0x9b/0x1b4 Mar 9 14:53:40 IP-FAIL-OVER kernel: [98210.524718] [] ? wake_bit_function+0x0/0x23 Mar 9 14:53:40 IP-FAIL-OVER kernel: [98210.524723] [] ? ra_submit+0x1c/0x20 Mar 9 14:53:40 IP-FAIL-OVER kernel: [98210.524729] [] ? filemap_fault+0x17d/0x2f6 Mar 9 14:53:40 IP-FAIL-OVER kernel: [98210.524735] [] ? __do_fault+0x54/0x3c3 Mar 9 14:53:40 IP-FAIL-OVER kernel: [98210.524740] [] ? __wait_on_bit_lock+0x76/0x84 Mar 9 14:53:40 IP-FAIL-OVER kernel: [98210.524746] [] ? __raw_callee_save_xen_pud_val+0x11/0x1e Mar 9 14:53:40 IP-FAIL-OVER kernel: [98210.524752] [] ? handle_mm_fault+0x3b8/0x80f Mar 9 14:53:40 IP-FAIL-OVER kernel: [98210.524758] [] ? do_page_fault+0x2e0/0x2fc Mar 9 14:53:40 IP-FAIL-OVER kernel: [98210.524763] [] ? page_fault+0x25/0x30 Mar 9 14:53:40 IP-FAIL-OVER kernel: [98210.524766] Mem-Info: Mar 9 14:53:40 IP-FAIL-OVER kernel: [98210.524768] Node 0 DMA per-cpu: Mar 9 14:53:40 IP-FAIL-OVER kernel: [98210.524772] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 14:53:40 IP-FAIL-OVER kernel: [98210.524775] Node 0 DMA32 per-cpu: Mar 9 14:53:40 IP-FAIL-OVER kernel: [98210.524779] CPU 0: hi: 186, btch: 31 usd: 125 Mar 9 14:53:40 IP-FAIL-OVER kernel: [98210.524785] active_anon:112248 inactive_anon:112261 isolated_anon:3014 Mar 9 14:53:40 IP-FAIL-OVER kernel: [98210.524787] active_file:174 inactive_file:443 isolated_file:0 Mar 9 14:53:40 IP-FAIL-OVER kernel: [98210.524788] unevictable:0 dirty:0 writeback:237 unstable:0 Mar 9 14:53:40 IP-FAIL-OVER kernel: [98210.524789] free:2080 slab_reclaimable:1385 slab_unreclaimable:2974 Mar 9 14:53:40 IP-FAIL-OVER kernel: [98210.524791] mapped:167 shmem:11 pagetables:11340 bounce:0 Mar 9 14:53:40 IP-FAIL-OVER kernel: [98210.524794] Node 0 DMA free:3936kB min:52kB low:64kB high:76kB active_anon:4804kB inactive_anon:4776kB active_file:28kB inactive_file:32kB unevictable:0kB isolated(anon):384kB isolated(file):0kB present:13884kB mlocked:0kB dirty:0kB writeback:28kB mapped:28kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:36kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:284 all_unreclaimable? no Mar 9 14:53:40 IP-FAIL-OVER kernel: [98210.524808] lowmem_reserve[]: 0 970 970 970 Mar 9 14:53:40 IP-FAIL-OVER kernel: [98210.524815] Node 0 DMA32 free:4384kB min:3956kB low:4944kB high:5932kB active_anon:444188kB inactive_anon:444268kB active_file:668kB inactive_file:1740kB unevictable:0kB isolated(anon):11672kB isolated(file):0kB present:993840kB mlocked:0kB dirty:0kB writeback:920kB mapped:640kB shmem:44kB slab_reclaimable:5540kB slab_unreclaimable:11892kB kernel_stack:1664kB pagetables:45324kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:11456 all_unreclaimable? no Mar 9 14:53:40 IP-FAIL-OVER kernel: [98210.524829] lowmem_reserve[]: 0 0 0 0 Mar 9 14:53:40 IP-FAIL-OVER kernel: [98210.524836] Node 0 DMA: 4*4kB 0*8kB 1*16kB 0*32kB 1*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3936kB Mar 9 14:53:40 IP-FAIL-OVER kernel: [98210.524853] Node 0 DMA32: 570*4kB 13*8kB 1*16kB 2*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 4384kB Mar 9 14:53:40 IP-FAIL-OVER kernel: [98210.524870] 40221 total pagecache pages Mar 9 14:53:40 IP-FAIL-OVER kernel: [98210.524873] 39577 pages in swap cache Mar 9 14:53:40 IP-FAIL-OVER kernel: [98210.524876] Swap cache stats: add 2225969, delete 2186392, find 161142/346492 Mar 9 14:53:40 IP-FAIL-OVER kernel: [98210.524879] Free swap = 0kB Mar 9 14:53:40 IP-FAIL-OVER kernel: [98210.524881] Total swap = 1947640kB Mar 9 14:53:40 IP-FAIL-OVER kernel: [98210.528054] 256000 pages RAM Mar 9 14:53:40 IP-FAIL-OVER kernel: [98210.528054] 6399 pages reserved Mar 9 14:53:40 IP-FAIL-OVER kernel: [98210.528054] 63665 pages shared Mar 9 14:53:40 IP-FAIL-OVER kernel: [98210.528054] 243450 pages non-shared Mar 9 14:53:40 IP-FAIL-OVER kernel: [98210.528054] Out of memory: kill process 559 (apache2) score 60275 or a child Mar 9 14:53:40 IP-FAIL-OVER kernel: [98210.528054] Killed process 9201 (apache2) Mar 9 14:56:25 IP-FAIL-OVER kernel: [98376.768962] IPv6 addrconf: prefix with wrong length 56 Mar 9 14:57:35 IP-FAIL-OVER kernel: [98438.076388] apache2 invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0 Mar 9 14:57:50 IP-FAIL-OVER kernel: [98438.076394] apache2 cpuset=/ mems_allowed=0 Mar 9 14:57:50 IP-FAIL-OVER kernel: [98438.076399] Pid: 9217, comm: apache2 Not tainted 2.6.32-5-amd64 #1 Mar 9 14:57:50 IP-FAIL-OVER kernel: [98438.076402] Call Trace: Mar 9 14:57:50 IP-FAIL-OVER kernel: [98438.076411] [] ? oom_kill_process+0x7f/0x23f Mar 9 14:57:50 IP-FAIL-OVER kernel: [98438.076417] [] ? __out_of_memory+0x12a/0x141 Mar 9 14:57:50 IP-FAIL-OVER kernel: [98438.076421] [] ? out_of_memory+0x140/0x172 Mar 9 14:57:50 IP-FAIL-OVER kernel: [98438.076427] [] ? __alloc_pages_nodemask+0x4ec/0x5fc Mar 9 14:57:50 IP-FAIL-OVER kernel: [98438.076433] [] ? __do_page_cache_readahead+0x9b/0x1b4 Mar 9 14:57:50 IP-FAIL-OVER kernel: [98438.076439] [] ? ra_submit+0x1c/0x20 Mar 9 14:57:50 IP-FAIL-OVER kernel: [98438.076444] [] ? filemap_fault+0x17d/0x2f6 Mar 9 14:57:50 IP-FAIL-OVER kernel: [98438.076451] [] ? __do_fault+0x54/0x3c3 Mar 9 14:57:50 IP-FAIL-OVER kernel: [98438.076458] [] ? __wait_on_bit_lock+0x76/0x84 Mar 9 14:57:50 IP-FAIL-OVER kernel: [98438.076464] [] ? __raw_callee_save_xen_pud_val+0x11/0x1e Mar 9 14:57:50 IP-FAIL-OVER kernel: [98438.076470] [] ? handle_mm_fault+0x3b8/0x80f Mar 9 14:57:50 IP-FAIL-OVER kernel: [98438.076476] [] ? get_unmapped_area+0xd7/0x139 Mar 9 14:57:50 IP-FAIL-OVER kernel: [98438.076481] [] ? do_page_fault+0x2e0/0x2fc Mar 9 14:57:50 IP-FAIL-OVER kernel: [98438.076487] [] ? page_fault+0x25/0x30 Mar 9 14:57:50 IP-FAIL-OVER kernel: [98438.076490] Mem-Info: Mar 9 14:57:50 IP-FAIL-OVER kernel: [98438.076492] Node 0 DMA per-cpu: Mar 9 14:57:50 IP-FAIL-OVER kernel: [98438.076496] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 14:57:50 IP-FAIL-OVER kernel: [98438.076499] Node 0 DMA32 per-cpu: Mar 9 14:57:50 IP-FAIL-OVER kernel: [98438.076502] CPU 0: hi: 186, btch: 31 usd: 27 Mar 9 14:57:50 IP-FAIL-OVER kernel: [98438.076509] active_anon:111998 inactive_anon:111965 isolated_anon:3905 Mar 9 14:57:50 IP-FAIL-OVER kernel: [98438.076510] active_file:150 inactive_file:252 isolated_file:0 Mar 9 14:57:50 IP-FAIL-OVER kernel: [98438.076512] unevictable:0 dirty:0 writeback:260 unstable:0 Mar 9 14:57:50 IP-FAIL-OVER kernel: [98438.076513] free:2088 slab_reclaimable:1454 slab_unreclaimable:2946 Mar 9 14:57:50 IP-FAIL-OVER kernel: [98438.076514] mapped:138 shmem:11 pagetables:11339 bounce:0 Mar 9 14:57:50 IP-FAIL-OVER kernel: [98438.076518] Node 0 DMA free:3944kB min:52kB low:64kB high:76kB active_anon:4820kB inactive_anon:5052kB active_file:16kB inactive_file:16kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13884kB mlocked:0kB dirty:0kB writeback:0kB mapped:20kB shmem:0kB slab_reclaimable:8kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:124kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:57 all_unreclaimable? no Mar 9 14:57:50 IP-FAIL-OVER kernel: [98438.076532] lowmem_reserve[]: 0 970 970 970 Mar 9 14:57:50 IP-FAIL-OVER kernel: [98438.076539] Node 0 DMA32 free:4408kB min:3956kB low:4944kB high:5932kB active_anon:443172kB inactive_anon:442808kB active_file:584kB inactive_file:992kB unevictable:0kB isolated(anon):15620kB isolated(file):0kB present:993840kB mlocked:0kB dirty:0kB writeback:1040kB mapped:532kB shmem:44kB slab_reclaimable:5808kB slab_unreclaimable:11784kB kernel_stack:1664kB pagetables:45232kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:5474 all_unreclaimable? no Mar 9 14:57:50 IP-FAIL-OVER kernel: [98438.076553] lowmem_reserve[]: 0 0 0 0 Mar 9 14:57:50 IP-FAIL-OVER kernel: [98438.076560] Node 0 DMA: 6*4kB 0*8kB 1*16kB 0*32kB 1*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3944kB Mar 9 14:57:50 IP-FAIL-OVER kernel: [98438.076577] Node 0 DMA32: 598*4kB 4*8kB 2*16kB 1*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 4408kB Mar 9 14:57:50 IP-FAIL-OVER kernel: [98438.076594] 52632 total pagecache pages Mar 9 14:57:50 IP-FAIL-OVER kernel: [98438.076596] 52219 pages in swap cache Mar 9 14:57:50 IP-FAIL-OVER kernel: [98438.076599] Swap cache stats: add 2374865, delete 2322646, find 170161/371859 Mar 9 14:57:50 IP-FAIL-OVER kernel: [98438.076602] Free swap = 0kB Mar 9 14:57:50 IP-FAIL-OVER kernel: [98438.076604] Total swap = 1947640kB Mar 9 14:57:50 IP-FAIL-OVER kernel: [98438.079807] 256000 pages RAM Mar 9 14:57:50 IP-FAIL-OVER kernel: [98438.079811] 6399 pages reserved Mar 9 14:57:50 IP-FAIL-OVER kernel: [98438.079813] 65667 pages shared Mar 9 14:57:50 IP-FAIL-OVER kernel: [98438.079815] 242558 pages non-shared Mar 9 14:57:50 IP-FAIL-OVER kernel: [98438.079819] Out of memory: kill process 559 (apache2) score 60054 or a child Mar 9 14:57:50 IP-FAIL-OVER kernel: [98438.079837] Killed process 9202 (apache2) Mar 9 14:59:45 IP-FAIL-OVER kernel: [98577.276882] IPv6 addrconf: prefix with wrong length 56 Mar 9 15:03:15 IP-FAIL-OVER kernel: [98777.926666] IPv6 addrconf: prefix with wrong length 56 Mar 9 15:05:42 IP-FAIL-OVER mysqld: 120309 15:05:41 [Note] Plugin 'FEDERATED' is disabled. Mar 9 15:05:42 IP-FAIL-OVER mysqld: 120309 15:05:42 InnoDB: The InnoDB memory heap is disabled Mar 9 15:05:42 IP-FAIL-OVER mysqld: 120309 15:05:42 InnoDB: Mutexes and rw_locks use GCC atomic builtins Mar 9 15:05:42 IP-FAIL-OVER mysqld: 120309 15:05:42 InnoDB: Compressed tables use zlib 1.2.3.4 Mar 9 15:05:42 IP-FAIL-OVER mysqld: 120309 15:05:42 InnoDB: Using Linux native AIO Mar 9 15:05:42 IP-FAIL-OVER mysqld: 120309 15:05:42 InnoDB: Initializing buffer pool, size = 128.0M Mar 9 15:05:42 IP-FAIL-OVER mysqld: 120309 15:05:42 InnoDB: Completed initialization of buffer pool Mar 9 15:06:26 IP-FAIL-OVER kernel: [98978.436953] IPv6 addrconf: prefix with wrong length 56 Mar 9 15:06:50 IP-FAIL-OVER mysqld: 120309 15:06:50 InnoDB: highest supported file format is Barracuda. Mar 9 15:09:38 IP-FAIL-OVER /USR/SBIN/CRON[9544]: (root) CMD ( [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -type f -cmin +$(/usr/lib/php5/maxlifetime) -print0 | xargs -n 200 -r -0 rm) Mar 9 15:09:47 IP-FAIL-OVER kernel: [99178.948147] IPv6 addrconf: prefix with wrong length 56 Mar 9 15:13:07 IP-FAIL-OVER kernel: [99379.475730] IPv6 addrconf: prefix with wrong length 56 Mar 9 15:15:22 IP-FAIL-OVER mysqld: 120309 15:15:11 InnoDB: Waiting for the background threads to start Mar 9 15:15:33 IP-FAIL-OVER mysqld: 120309 15:15:12 InnoDB: 1.1.8 started; log sequence number 1600899 Mar 9 15:16:28 IP-FAIL-OVER kernel: [99579.988135] IPv6 addrconf: prefix with wrong length 56 Mar 9 15:17:13 IP-FAIL-OVER /USR/SBIN/CRON[9567]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Mar 9 15:19:48 IP-FAIL-OVER kernel: [99780.499818] IPv6 addrconf: prefix with wrong length 56 Mar 9 15:21:20 IP-FAIL-OVER mysqld: 120309 15:21:20 [Note] Event Scheduler: Loaded 0 events Mar 9 15:21:20 IP-FAIL-OVER mysqld: 120309 15:21:20 [Note] /usr/sbin/mysqld: ready for connections. Mar 9 15:21:20 IP-FAIL-OVER mysqld: Version: '5.5.20-1~dotdeb.0' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Debian) Mar 9 15:21:58 IP-FAIL-OVER mysqld: 120309 15:21:58 [ERROR] /usr/sbin/mysqld: Table './website1_blog/iIajj43jnJ_options' is marked as crashed and should be repaired Mar 9 15:21:58 IP-FAIL-OVER mysqld: 120309 15:21:58 [Warning] Checking table: './website1_blog/iIajj43jnJ_options' Mar 9 15:22:15 IP-FAIL-OVER mysqld: 120309 15:22:15 [ERROR] /usr/sbin/mysqld: Table './website3/wp_options' is marked as crashed and should be repaired Mar 9 15:22:15 IP-FAIL-OVER mysqld: 120309 15:22:15 [ERROR] /usr/sbin/mysqld: Table './website3/wp_options' is marked as crashed and should be repaired Mar 9 15:22:15 IP-FAIL-OVER mysqld: 120309 15:22:15 [ERROR] /usr/sbin/mysqld: Table './website3/wp_options' is marked as crashed and should be repaired Mar 9 15:22:15 IP-FAIL-OVER mysqld: 120309 15:22:15 [ERROR] /usr/sbin/mysqld: Table './website3/wp_options' is marked as crashed and should be repaired Mar 9 15:22:15 IP-FAIL-OVER mysqld: 120309 15:22:15 [ERROR] /usr/sbin/mysqld: Table './website3/wp_options' is marked as crashed and should be repaired Mar 9 15:22:15 IP-FAIL-OVER mysqld: 120309 15:22:15 [ERROR] /usr/sbin/mysqld: Table './website3/wp_options' is marked as crashed and should be repaired Mar 9 15:22:15 IP-FAIL-OVER mysqld: 120309 15:22:15 [ERROR] /usr/sbin/mysqld: Table './website3/wp_options' is marked as crashed and should be repaired Mar 9 15:22:15 IP-FAIL-OVER mysqld: 120309 15:22:15 [Warning] Checking table: './website3/wp_options' Mar 9 15:23:09 IP-FAIL-OVER kernel: [99981.011609] IPv6 addrconf: prefix with wrong length 56 Mar 9 15:23:27 IP-FAIL-OVER mysqld: 120309 15:22:31 [ERROR] /usr/sbin/mysqld: Table './website2_blog/wp_options' is marked as crashed and should be repaired Mar 9 15:23:27 IP-FAIL-OVER mysqld: 120309 15:23:27 [ERROR] /usr/sbin/mysqld: Table './website2_blog/wp_options' is marked as crashed and should be repaired Mar 9 15:23:27 IP-FAIL-OVER mysqld: 120309 15:23:27 [ERROR] /usr/sbin/mysqld: Table './website2_blog/wp_options' is marked as crashed and should be repaired Mar 9 15:23:27 IP-FAIL-OVER mysqld: 120309 15:23:27 [Warning] Checking table: './website2_blog/wp_options' Mar 9 15:26:30 IP-FAIL-OVER kernel: [100181.523301] IPv6 addrconf: prefix with wrong length 56 Mar 9 15:29:50 IP-FAIL-OVER kernel: [100382.034913] IPv6 addrconf: prefix with wrong length 56 Mar 9 15:32:22 IP-FAIL-OVER kernel: [100507.336406] mysqld invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0 Mar 9 15:32:42 IP-FAIL-OVER kernel: [100507.336412] mysqld cpuset=/ mems_allowed=0 Mar 9 15:32:42 IP-FAIL-OVER kernel: [100507.336417] Pid: 9536, comm: mysqld Not tainted 2.6.32-5-amd64 #1 Mar 9 15:32:42 IP-FAIL-OVER kernel: [100507.336420] Call Trace: Mar 9 15:32:42 IP-FAIL-OVER kernel: [100507.336430] [] ? oom_kill_process+0x7f/0x23f Mar 9 15:32:42 IP-FAIL-OVER kernel: [100507.336435] [] ? __out_of_memory+0x12a/0x141 Mar 9 15:32:42 IP-FAIL-OVER kernel: [100507.336440] [] ? out_of_memory+0x140/0x172 Mar 9 15:32:42 IP-FAIL-OVER kernel: [100507.336446] [] ? __alloc_pages_nodemask+0x4ec/0x5fc Mar 9 15:32:42 IP-FAIL-OVER kernel: [100507.336452] [] ? __do_page_cache_readahead+0x9b/0x1b4 Mar 9 15:32:42 IP-FAIL-OVER kernel: [100507.336459] [] ? check_preempt_wakeup+0x0/0x268 Mar 9 15:32:42 IP-FAIL-OVER kernel: [100507.336464] [] ? ra_submit+0x1c/0x20 Mar 9 15:32:42 IP-FAIL-OVER kernel: [100507.336470] [] ? filemap_fault+0x17d/0x2f6 Mar 9 15:32:42 IP-FAIL-OVER kernel: [100507.336476] [] ? __do_fault+0x54/0x3c3 Mar 9 15:32:42 IP-FAIL-OVER kernel: [100507.336483] [] ? xen_force_evtchn_callback+0x9/0xa Mar 9 15:32:42 IP-FAIL-OVER kernel: [100507.336488] [] ? xen_restore_fl_direct_end+0x0/0x1 Mar 9 15:32:42 IP-FAIL-OVER kernel: [100507.336494] [] ? __raw_callee_save_xen_pud_val+0x11/0x1e Mar 9 15:32:42 IP-FAIL-OVER kernel: [100507.336500] [] ? handle_mm_fault+0x3b8/0x80f Mar 9 15:32:42 IP-FAIL-OVER kernel: [100507.336505] [] ? default_wake_function+0x0/0x9 Mar 9 15:32:42 IP-FAIL-OVER kernel: [100507.336513] [] ? do_page_fault+0x2e0/0x2fc Mar 9 15:32:42 IP-FAIL-OVER kernel: [100507.336518] [] ? page_fault+0x25/0x30 Mar 9 15:32:42 IP-FAIL-OVER kernel: [100507.336521] Mem-Info: Mar 9 15:32:42 IP-FAIL-OVER kernel: [100507.336524] Node 0 DMA per-cpu: Mar 9 15:32:42 IP-FAIL-OVER kernel: [100507.336528] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 15:32:42 IP-FAIL-OVER kernel: [100507.336530] Node 0 DMA32 per-cpu: Mar 9 15:32:42 IP-FAIL-OVER kernel: [100507.336534] CPU 0: hi: 186, btch: 31 usd: 89 Mar 9 15:32:42 IP-FAIL-OVER kernel: [100507.336541] active_anon:112060 inactive_anon:112139 isolated_anon:3904 Mar 9 15:32:42 IP-FAIL-OVER kernel: [100507.336542] active_file:59 inactive_file:183 isolated_file:0 Mar 9 15:32:42 IP-FAIL-OVER kernel: [100507.336543] unevictable:0 dirty:0 writeback:283 unstable:0 Mar 9 15:32:42 IP-FAIL-OVER kernel: [100507.336545] free:1961 slab_reclaimable:1617 slab_unreclaimable:2998 Mar 9 15:32:42 IP-FAIL-OVER kernel: [100507.336546] mapped:67 shmem:11 pagetables:11356 bounce:0 Mar 9 15:32:42 IP-FAIL-OVER kernel: [100507.336549] Node 0 DMA free:3944kB min:52kB low:64kB high:76kB active_anon:4812kB inactive_anon:5080kB active_file:28kB inactive_file:64kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13884kB mlocked:0kB dirty:0kB writeback:0kB mapped:20kB shmem:0kB slab_reclaimable:80kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:8kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:137 all_unreclaimable? no Mar 9 15:32:42 IP-FAIL-OVER kernel: [100507.336563] lowmem_reserve[]: 0 970 970 970 Mar 9 15:32:42 IP-FAIL-OVER kernel: [100507.336571] Node 0 DMA32 free:3900kB min:3956kB low:4944kB high:5932kB active_anon:443428kB inactive_anon:443476kB active_file:208kB inactive_file:668kB unevictable:0kB isolated(anon):15616kB isolated(file):0kB present:993840kB mlocked:0kB dirty:0kB writeback:1132kB mapped:248kB shmem:44kB slab_reclaimable:6388kB slab_unreclaimable:11984kB kernel_stack:2048kB pagetables:45416kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1482 all_unreclaimable? no Mar 9 15:32:58 IP-FAIL-OVER kernel: [100507.336585] lowmem_reserve[]: 0 0 0 0 Mar 9 15:32:58 IP-FAIL-OVER kernel: [100507.336592] Node 0 DMA: 13*4kB 4*8kB 1*16kB 0*32kB 0*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3940kB Mar 9 15:32:58 IP-FAIL-OVER kernel: [100507.336609] Node 0 DMA32: 489*4kB 5*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 3900kB Mar 9 15:32:58 IP-FAIL-OVER kernel: [100507.336626] 54125 total pagecache pages Mar 9 15:32:58 IP-FAIL-OVER kernel: [100507.336628] 53866 pages in swap cache Mar 9 15:32:58 IP-FAIL-OVER kernel: [100507.336631] Swap cache stats: add 4061629, delete 4007763, find 309429/693171 Mar 9 15:32:58 IP-FAIL-OVER kernel: [100507.336634] Free swap = 0kB Mar 9 15:32:58 IP-FAIL-OVER kernel: [100507.336636] Total swap = 1947640kB Mar 9 15:32:58 IP-FAIL-OVER kernel: [100507.339841] 256000 pages RAM Mar 9 15:32:58 IP-FAIL-OVER kernel: [100507.339844] 6399 pages reserved Mar 9 15:32:58 IP-FAIL-OVER kernel: [100507.339846] 54424 pages shared Mar 9 15:32:58 IP-FAIL-OVER kernel: [100507.339848] 243026 pages non-shared Mar 9 15:32:58 IP-FAIL-OVER kernel: [100507.339853] Out of memory: kill process 9495 (mysqld) score 173893 or a child Mar 9 15:32:58 IP-FAIL-OVER kernel: [100507.339871] Killed process 9495 (mysqld) Mar 9 15:33:11 IP-FAIL-OVER kernel: [100582.546652] IPv6 addrconf: prefix with wrong length 56 Mar 9 15:36:52 IP-FAIL-OVER kernel: [100783.058528] IPv6 addrconf: prefix with wrong length 56 Mar 9 15:38:14 IP-FAIL-OVER kernel: [100864.237853] apache2 invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0 Mar 9 15:38:15 IP-FAIL-OVER kernel: [100864.237857] apache2 cpuset=/ mems_allowed=0 Mar 9 15:38:15 IP-FAIL-OVER kernel: [100864.237860] Pid: 9593, comm: apache2 Not tainted 2.6.32-5-amd64 #1 Mar 9 15:38:15 IP-FAIL-OVER kernel: [100864.237862] Call Trace: Mar 9 15:38:15 IP-FAIL-OVER kernel: [100864.237868] [] ? oom_kill_process+0x7f/0x23f Mar 9 15:38:15 IP-FAIL-OVER kernel: [100864.237871] [] ? __out_of_memory+0x12a/0x141 Mar 9 15:38:15 IP-FAIL-OVER kernel: [100864.237874] [] ? out_of_memory+0x140/0x172 Mar 9 15:38:15 IP-FAIL-OVER kernel: [100864.237877] [] ? __alloc_pages_nodemask+0x4ec/0x5fc Mar 9 15:38:15 IP-FAIL-OVER kernel: [100864.237882] [] ? handle_mm_fault+0x27a/0x80f Mar 9 15:38:15 IP-FAIL-OVER kernel: [100864.237887] [] ? do_page_fault+0x2e0/0x2fc Mar 9 15:38:15 IP-FAIL-OVER kernel: [100864.237890] [] ? page_fault+0x25/0x30 Mar 9 15:38:15 IP-FAIL-OVER kernel: [100864.237891] Mem-Info: Mar 9 15:38:15 IP-FAIL-OVER kernel: [100864.237893] Node 0 DMA per-cpu: Mar 9 15:38:15 IP-FAIL-OVER kernel: [100864.237895] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 15:38:15 IP-FAIL-OVER kernel: [100864.237896] Node 0 DMA32 per-cpu: Mar 9 15:38:15 IP-FAIL-OVER kernel: [100864.237899] CPU 0: hi: 186, btch: 31 usd: 58 Mar 9 15:38:15 IP-FAIL-OVER kernel: [100864.237902] active_anon:111650 inactive_anon:111526 isolated_anon:4901 Mar 9 15:38:15 IP-FAIL-OVER kernel: [100864.237903] active_file:61 inactive_file:103 isolated_file:0 Mar 9 15:38:15 IP-FAIL-OVER kernel: [100864.237904] unevictable:0 dirty:0 writeback:303 unstable:0 Mar 9 15:38:15 IP-FAIL-OVER kernel: [100864.237904] free:1998 slab_reclaimable:1621 slab_unreclaimable:3030 Mar 9 15:38:15 IP-FAIL-OVER kernel: [100864.237905] mapped:158 shmem:11 pagetables:11352 bounce:0 Mar 9 15:38:15 IP-FAIL-OVER kernel: [100864.237907] Node 0 DMA free:3928kB min:52kB low:64kB high:76kB active_anon:4872kB inactive_anon:5060kB active_file:12kB inactive_file:20kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13884kB mlocked:0kB dirty:0kB writeback:0kB mapped:32kB shmem:0kB slab_reclaimable:80kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:28kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:8 all_unreclaimable? no Mar 9 15:38:15 IP-FAIL-OVER kernel: [100864.237915] lowmem_reserve[]: 0 970 970 970 Mar 9 15:38:15 IP-FAIL-OVER kernel: [100864.237919] Node 0 DMA32 free:4064kB min:3956kB low:4944kB high:5932kB active_anon:441728kB inactive_anon:441044kB active_file:232kB inactive_file:392kB unevictable:0kB isolated(anon):19604kB isolated(file):0kB present:993840kB mlocked:0kB dirty:0kB writeback:1212kB mapped:600kB shmem:44kB slab_reclaimable:6404kB slab_unreclaimable:12112kB kernel_stack:1640kB pagetables:45380kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:377 all_unreclaimable? no Mar 9 15:38:15 IP-FAIL-OVER kernel: [100864.237927] lowmem_reserve[]: 0 0 0 0 Mar 9 15:38:15 IP-FAIL-OVER kernel: [100864.237930] Node 0 DMA: 6*4kB 6*8kB 1*16kB 0*32kB 0*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3928kB Mar 9 15:38:15 IP-FAIL-OVER kernel: [100864.237940] Node 0 DMA32: 534*4kB 9*8kB 2*16kB 1*32kB 0*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 4064kB Mar 9 15:38:15 IP-FAIL-OVER kernel: [100864.237949] 40922 total pagecache pages Mar 9 15:38:15 IP-FAIL-OVER kernel: [100864.237950] 40746 pages in swap cache Mar 9 15:38:15 IP-FAIL-OVER kernel: [100864.237952] Swap cache stats: add 4332257, delete 4291511, find 328979/741242 Mar 9 15:38:15 IP-FAIL-OVER kernel: [100864.237953] Free swap = 0kB Mar 9 15:38:15 IP-FAIL-OVER kernel: [100864.237955] Total swap = 1947640kB Mar 9 15:38:15 IP-FAIL-OVER kernel: [100864.239847] 256000 pages RAM Mar 9 15:38:15 IP-FAIL-OVER kernel: [100864.239849] 6399 pages reserved Mar 9 15:38:15 IP-FAIL-OVER kernel: [100864.239850] 56590 pages shared Mar 9 15:38:15 IP-FAIL-OVER kernel: [100864.239851] 241960 pages non-shared Mar 9 15:38:15 IP-FAIL-OVER kernel: [100864.239854] Out of memory: kill process 559 (apache2) score 60576 or a child Mar 9 15:38:15 IP-FAIL-OVER kernel: [100864.239866] Killed process 9203 (apache2) Mar 9 15:38:43 IP-FAIL-OVER kernel: [100896.867022] apache2 invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0 Mar 9 15:38:54 IP-FAIL-OVER kernel: [100896.867028] apache2 cpuset=/ mems_allowed=0 Mar 9 15:38:54 IP-FAIL-OVER kernel: [100896.867033] Pid: 9282, comm: apache2 Not tainted 2.6.32-5-amd64 #1 Mar 9 15:38:54 IP-FAIL-OVER kernel: [100896.867036] Call Trace: Mar 9 15:38:54 IP-FAIL-OVER kernel: [100896.867045] [] ? oom_kill_process+0x7f/0x23f Mar 9 15:38:54 IP-FAIL-OVER kernel: [100896.867050] [] ? __out_of_memory+0x12a/0x141 Mar 9 15:38:54 IP-FAIL-OVER kernel: [100896.867055] [] ? out_of_memory+0x140/0x172 Mar 9 15:38:54 IP-FAIL-OVER kernel: [100896.867061] [] ? __alloc_pages_nodemask+0x4ec/0x5fc Mar 9 15:38:54 IP-FAIL-OVER kernel: [100896.867068] [] ? bio_alloc+0x10/0x1f Mar 9 15:38:54 IP-FAIL-OVER kernel: [100896.867074] [] ? read_swap_cache_async+0x5d/0xf3 Mar 9 15:38:54 IP-FAIL-OVER kernel: [100896.867080] [] ? swapin_readahead+0x57/0x98 Mar 9 15:38:54 IP-FAIL-OVER kernel: [100896.867086] [] ? __raw_callee_save_xen_pte_val+0x11/0x1e Mar 9 15:38:54 IP-FAIL-OVER kernel: [100896.867093] [] ? handle_mm_fault+0x47f/0x80f Mar 9 15:38:54 IP-FAIL-OVER kernel: [100896.867101] [] ? do_page_fault+0x2e0/0x2fc Mar 9 15:38:54 IP-FAIL-OVER kernel: [100896.867106] [] ? page_fault+0x25/0x30 Mar 9 15:38:54 IP-FAIL-OVER kernel: [100896.867109] Mem-Info: Mar 9 15:38:54 IP-FAIL-OVER kernel: [100896.867112] Node 0 DMA per-cpu: Mar 9 15:38:54 IP-FAIL-OVER kernel: [100896.867116] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 15:38:54 IP-FAIL-OVER kernel: [100896.867118] Node 0 DMA32 per-cpu: Mar 9 15:38:54 IP-FAIL-OVER kernel: [100896.867123] CPU 0: hi: 186, btch: 31 usd: 43 Mar 9 15:38:54 IP-FAIL-OVER kernel: [100896.867129] active_anon:111527 inactive_anon:111556 isolated_anon:4899 Mar 9 15:38:54 IP-FAIL-OVER kernel: [100896.867131] active_file:74 inactive_file:92 isolated_file:0 Mar 9 15:38:54 IP-FAIL-OVER kernel: [100896.867132] unevictable:0 dirty:0 writeback:286 unstable:0 Mar 9 15:38:54 IP-FAIL-OVER kernel: [100896.867134] free:1960 slab_reclaimable:1618 slab_unreclaimable:3032 Mar 9 15:38:54 IP-FAIL-OVER kernel: [100896.867135] mapped:163 shmem:11 pagetables:11367 bounce:0 Mar 9 15:38:54 IP-FAIL-OVER kernel: [100896.867138] Node 0 DMA free:3928kB min:52kB low:64kB high:76kB active_anon:4760kB inactive_anon:4832kB active_file:16kB inactive_file:20kB unevictable:0kB isolated(anon):384kB isolated(file):0kB present:13884kB mlocked:0kB dirty:0kB writeback:20kB mapped:36kB shmem:0kB slab_reclaimable:80kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:28kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 15:38:54 IP-FAIL-OVER kernel: [100896.867152] lowmem_reserve[]: 0 970 970 970 Mar 9 15:38:54 IP-FAIL-OVER kernel: [100896.867160] Node 0 DMA32 free:3912kB min:3956kB low:4944kB high:5932kB active_anon:441348kB inactive_anon:441392kB active_file:280kB inactive_file:348kB unevictable:0kB isolated(anon):19212kB isolated(file):0kB present:993840kB mlocked:0kB dirty:0kB writeback:1124kB mapped:616kB shmem:44kB slab_reclaimable:6392kB slab_unreclaimable:12120kB kernel_stack:1648kB pagetables:45440kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:274 all_unreclaimable? no Mar 9 15:38:54 IP-FAIL-OVER kernel: [100896.867174] lowmem_reserve[]: 0 0 0 0 Mar 9 15:38:54 IP-FAIL-OVER kernel: [100896.867181] Node 0 DMA: 6*4kB 6*8kB 1*16kB 0*32kB 0*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3928kB Mar 9 15:38:54 IP-FAIL-OVER kernel: [100896.867198] Node 0 DMA32: 412*4kB 53*8kB 1*16kB 1*32kB 0*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 3912kB Mar 9 15:38:54 IP-FAIL-OVER kernel: [100896.867215] 44052 total pagecache pages Mar 9 15:38:54 IP-FAIL-OVER kernel: [100896.867217] 43868 pages in swap cache Mar 9 15:38:54 IP-FAIL-OVER kernel: [100896.867220] Swap cache stats: add 4357091, delete 4313223, find 330297/744797 Mar 9 15:38:54 IP-FAIL-OVER kernel: [100896.867223] Free swap = 0kB Mar 9 15:38:54 IP-FAIL-OVER kernel: [100896.867225] Total swap = 1947640kB Mar 9 15:38:54 IP-FAIL-OVER kernel: [100896.868058] 256000 pages RAM Mar 9 15:38:54 IP-FAIL-OVER kernel: [100896.868058] 6399 pages reserved Mar 9 15:38:54 IP-FAIL-OVER kernel: [100896.868058] 57093 pages shared Mar 9 15:38:54 IP-FAIL-OVER kernel: [100896.868058] 241911 pages non-shared Mar 9 15:38:54 IP-FAIL-OVER kernel: [100896.868058] Out of memory: kill process 559 (apache2) score 60545 or a child Mar 9 15:38:54 IP-FAIL-OVER kernel: [100896.868058] Killed process 9207 (apache2) Mar 9 15:40:06 IP-FAIL-OVER kernel: [100983.570242] IPv6 addrconf: prefix with wrong length 56 Mar 9 15:41:01 IP-FAIL-OVER kernel: [101024.232126] apache2 invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0 Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.232133] apache2 cpuset=/ mems_allowed=0 Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.232137] Pid: 9648, comm: apache2 Not tainted 2.6.32-5-amd64 #1 Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.232140] Call Trace: Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.232150] [] ? oom_kill_process+0x7f/0x23f Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.232155] [] ? __out_of_memory+0x12a/0x141 Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.232160] [] ? out_of_memory+0x140/0x172 Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.232166] [] ? __alloc_pages_nodemask+0x4ec/0x5fc Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.232173] [] ? io_schedule+0x93/0xb7 Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.232178] [] ? __do_page_cache_readahead+0x9b/0x1b4 Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.232184] [] ? ra_submit+0x1c/0x20 Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.232189] [] ? filemap_fault+0x11c/0x2f6 Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.232196] [] ? __do_fault+0x54/0x3c3 Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.232203] [] ? _local_bh_enable_ip+0x22/0x8f Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.232209] [] ? inet_csk_accept+0x1bf/0x1e9 Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.232216] [] ? __raw_callee_save_xen_pud_val+0x11/0x1e Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.232221] [] ? handle_mm_fault+0x3b8/0x80f Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.232227] [] ? xen_force_evtchn_callback+0x9/0xa Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.232232] [] ? check_events+0x12/0x20 Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.232239] [] ? pvclock_clocksource_read+0x3a/0x8b Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.232245] [] ? do_page_fault+0x2e0/0x2fc Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.232251] [] ? page_fault+0x25/0x30 Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.232254] Mem-Info: Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.232256] Node 0 DMA per-cpu: Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.232260] CPU 0: hi: 0, btch: 1 usd: 0 Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.232263] Node 0 DMA32 per-cpu: Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.232267] CPU 0: hi: 186, btch: 31 usd: 30 Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.232273] active_anon:111475 inactive_anon:111556 isolated_anon:4930 Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.232275] active_file:3 inactive_file:39 isolated_file:0 Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.232277] unevictable:0 dirty:0 writeback:328 unstable:0 Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.232278] free:1966 slab_reclaimable:1619 slab_unreclaimable:3032 Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.232279] mapped:34 shmem:11 pagetables:11360 bounce:0 Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.232283] Node 0 DMA free:3928kB min:52kB low:64kB high:76kB active_anon:4712kB inactive_anon:4820kB active_file:8kB inactive_file:0kB unevictable:0kB isolated(anon):384kB isolated(file):0kB present:13884kB mlocked:0kB dirty:0kB writeback:32kB mapped:8kB shmem:0kB slab_reclaimable:80kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:28kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.232297] lowmem_reserve[]: 0 970 970 970 Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.232304] Node 0 DMA32 free:3936kB min:3956kB low:4944kB high:5932kB active_anon:441188kB inactive_anon:441404kB active_file:4kB inactive_file:156kB unevictable:0kB isolated(anon):19336kB isolated(file):0kB present:993840kB mlocked:0kB dirty:0kB writeback:1280kB mapped:128kB shmem:44kB slab_reclaimable:6396kB slab_unreclaimable:12120kB kernel_stack:1656kB pagetables:45412kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:37 all_unreclaimable? no Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.232318] lowmem_reserve[]: 0 0 0 0 Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.232325] Node 0 DMA: 4*4kB 7*8kB 1*16kB 0*32kB 0*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3928kB Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.232342] Node 0 DMA32: 498*4kB 9*8kB 3*16kB 1*32kB 0*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 3936kB Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.232359] 43773 total pagecache pages Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.232361] 43719 pages in swap cache Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.232364] Swap cache stats: add 4453154, delete 4409435, find 337055/761525 Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.232367] Free swap = 0kB Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.232369] Total swap = 1947640kB Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.235412] 256000 pages RAM Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.235414] 6399 pages reserved Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.235415] 53139 pages shared Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.235416] 241900 pages non-shared Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.235420] Out of memory: kill process 559 (apache2) score 60545 or a child Mar 9 15:41:28 IP-FAIL-OVER kernel: [101024.235430] Killed process 9210 (apache2) Mar 9 15:43:30 IP-FAIL-OVER kernel: [101184.094047] IPv6 addrconf: prefix with wrong length 56 Mar 9 15:43:39 IP-FAIL-OVER kernel: [101193.294702] IPv6 addrconf: prefix with wrong length 56 Mar 9 15:45:04 IP-FAIL-OVER /USR/SBIN/CRON[9651]: (root) CMD ( [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -type f -cmin +$(/usr/lib/php5/maxlifetime) -print0 | xargs -n 200 -r -0 rm) Mar 9 15:46:42 IP-FAIL-OVER kernel: [101393.794522] IPv6 addrconf: prefix with wrong length 56 Mar 9 15:50:37 IP-FAIL-OVER kernel: imklog 4.6.4, log source = /proc/kmsg started. Mar 9 15:50:37 IP-FAIL-OVER rsyslogd: [origin software="rsyslogd" swVersion="4.6.4" x-pid="594" x-info="http://www.rsyslog.com"] (re)start Mar 9 15:50:37 IP-FAIL-OVER kernel: [ 0.000000] Initializing cgroup subsys cpuset