API tools faq
paste
Advertisement
Guest User

Untitled

a guest
Dec 11th, 2017
61
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 298.36 KB | None | 0 0
raw download clone embed print report
  1. root@monitor:/var/log# more syslog
  2. Sep 27 01:46:04 monitor kernel: nagios invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  3. Sep 27 01:46:04 monitor kernel: Pid: 11513, comm: nagios Not tainted 2.6.24.3 #3
  4. Sep 27 01:46:04 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  5. Sep 27 01:46:04 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  6. Sep 27 01:46:04 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  7. Sep 27 01:46:04 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  8. Sep 27 01:46:04 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  9. Sep 27 01:46:04 monitor kernel: [<c0151cd1>] filemap oomkilladj=0
  10. Sep 27 01:46:04 monitor kernel: Pid: 11513, comm: nagios Not tainted 2.6.24.3 #3
  11. Sep 27 01:46:04 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  12. Sep 27 01:46:04 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  13. Sep 27 01:46:04 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  14. Sep 27 01:46:04 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  15. Sep 27 01:46:04 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  16. Sep 27 01:46:04 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  17. Sep 27 01:46:04 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  18. Sep 27 01:46:04 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  19. Sep 27 01:46:04 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  20. Sep 27 01:46:04 monitor kernel: [<c0172d7a>] vfs_write+0x11a/0x160
  21. Sep 27 01:46:04 monitor kernel: [<c01723a0>] do_sync_write+0x0/0x110
  22. Sep 27 01:46:04 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  23. Sep 27 01:46:04 monitor kernel: [<c043a202>] error_code+0x72/0x78
  24. Sep 27 01:46:04 monitor kernel: [<c0430000>] rpcb_getport_sync+0x0/0x190
  25. Sep 27 01:46:04 monitor kernel: =======================
  26. Sep 27 01:46:04 monitor kernel: DMA per-cpu:
  27. Sep 27 01:46:04 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  28. Sep 27 01:46:04 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  29. Sep 27 01:46:04 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  30. Sep 27 01:46:04 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  31. Sep 27 01:46:04 monitor kernel: Normal per-cpu:
  32. Sep 27 01:46:04 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 102 Cold: hi: 62, btch: 15 usd: 58
  33. Sep 27 01:46:04 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 72 Cold: hi: 62, btch: 15 usd: 51
  34. Sep 27 01:46:04 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 161 Cold: hi: 62, btch: 15 usd: 18
  35. Sep 27 01:46:04 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 64 Cold: hi: 62, btch: 15 usd: 47
  36. Sep 27 01:46:04 monitor kernel: HighMem per-cpu:
  37. Sep 27 01:46:04 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 41 Cold: hi: 62, btch: 15 usd: 59
  38. Sep 27 01:46:04 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 26 Cold: hi: 62, btch: 15 usd: 20
  39. Sep 27 01:46:04 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 29 Cold: hi: 62, btch: 15 usd: 26
  40. Sep 27 01:46:04 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 117 Cold: hi: 62, btch: 15 usd: 61
  41. Sep 27 01:46:04 monitor kernel: Active:429481 inactive:431125 dirty:0 writeback:0 unstable:0
  42. Sep 27 01:46:04 monitor kernel: free:29165 slab:4584 mapped:8 pagetables:3998 bounce:0
  43. Sep 27 01:46:04 monitor kernel: DMA free:10912kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  44. able? yes
  45. Sep 27 01:46:04 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  46. Sep 27 01:46:04 monitor kernel: Normal free:105288kB min:3744kB low:4680kB high:5616kB active:354228kB inactive:352992kB present:894080kB pages_sca
  47. nned:5128736 all_unreclaimable? yes
  48. Sep 27 01:46:04 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  49. Sep 27 01:46:04 monitor kernel: HighMem free:460kB min:512kB low:3916kB high:7324kB active:1363696kB inactive:1371508kB present:3251200kB pages_sca
  50. nned:8557432 all_unreclaimable? yes
  51. Sep 27 01:46:04 monitor kernel: lowmem_reserve[]: 0 0 0 0
  52. Sep 27 01:46:04 monitor kernel: DMA: 84*4kB 78*8kB 64*16kB 47*32kB 24*64kB 14*128kB 2*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10912kB
  53. Sep 27 01:46:04 monitor kernel: Normal: 13628*4kB 5957*8kB 15*16kB 10*32kB 2*64kB 3*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 105288kB
  54. Sep 27 01:46:04 monitor kernel: HighMem: 37*4kB 11*8kB 0*16kB 1*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 460kB
  55. Sep 27 01:46:04 monitor kernel: Swap cache: add 4375595, delete 4375609, find 1542064284/1542140980, race 145+69
  56. Sep 27 01:46:04 monitor kernel: Free swap = 0kB
  57. Sep 27 01:46:04 monitor kernel: Total swap = 3911816kB
  58. Sep 27 01:46:04 monitor kernel: Out of memory: kill process 4465 (postgres) score 9199 or a child
  59. Sep 27 01:46:04 monitor kernel: Killed process 4465 (postgres)
  60. Sep 27 01:46:04 monitor kernel: hald invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  61. Sep 27 01:46:04 monitor kernel: Pid: 3293, comm: hald Not tainted 2.6.24.3 #3
  62. Sep 27 01:46:04 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  63. Sep 27 01:46:04 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  64. Sep 27 01:46:04 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  65. Sep 27 01:46:04 monitor kernel: [<c011a2f2>] update_curr+0x72/0x110
  66. Sep 27 01:46:04 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  67. Sep 27 01:46:04 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  68. Sep 27 01:46:04 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  69. Sep 27 01:46:04 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  70. Sep 27 01:46:04 monitor kernel: [<c0166111>] swap_info_get+0x51/0xd0
  71. Sep 27 01:46:04 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  72. Sep 27 01:46:04 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  73. Sep 27 01:46:04 monitor kernel: [<c017fba4>] sys_poll+0x34/0x80
  74. Sep 27 01:46:04 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  75. Sep 27 01:46:04 monitor kernel: [<c043a202>] error_code+0x72/0x78
  76. Sep 27 01:46:04 monitor kernel: =======================
  77. Sep 27 01:46:04 monitor kernel: DMA per-cpu:
  78. Sep 27 01:46:04 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  79. Sep 27 01:46:04 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  80. Sep 27 01:46:04 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  81. Sep 27 01:46:04 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  82. Sep 27 01:46:04 monitor kernel: Normal per-cpu:
  83. Sep 27 01:46:04 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 102 Cold: hi: 62, btch: 15 usd: 47
  84. Sep 27 01:46:04 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 69 Cold: hi: 62, btch: 15 usd: 57
  85. Sep 27 01:46:04 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 179 Cold: hi: 62, btch: 15 usd: 28
  86. Sep 27 01:46:04 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 51 Cold: hi: 62, btch: 15 usd: 50
  87. Sep 27 01:46:04 monitor kernel: HighMem per-cpu:
  88. Sep 27 01:46:04 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 45 Cold: hi: 62, btch: 15 usd: 47
  89. Sep 27 01:46:04 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 26 Cold: hi: 62, btch: 15 usd: 42
  90. Sep 27 01:46:04 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 164 Cold: hi: 62, btch: 15 usd: 60
  91. Sep 27 01:46:04 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 117 Cold: hi: 62, btch: 15 usd: 24
  92. Sep 27 01:46:04 monitor kernel: Active:360031 inactive:499674 dirty:0 writeback:0 unstable:0
  93. Sep 27 01:46:04 monitor kernel: free:29217 slab:4595 mapped:9 pagetables:3986 bounce:0
  94. Sep 27 01:46:04 monitor kernel: DMA free:10912kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  95. able? yes
  96. Sep 27 01:46:04 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  97. Sep 27 01:46:04 monitor kernel: Normal free:105300kB min:3744kB low:4680kB high:5616kB active:300244kB inactive:412128kB present:894080kB pages_sca
  98. nned:1782844 all_unreclaimable? yes
  99. Sep 27 01:46:04 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  100. Sep 27 01:46:04 monitor kernel: HighMem free:656kB min:512kB low:3916kB high:7324kB active:1139880kB inactive:1586568kB present:3251200kB pages_sca
  101. nned:4709117 all_unreclaimable? yes
  102. Sep 27 01:46:04 monitor kernel: lowmem_reserve[]: 0 0 0 0
  103. Sep 27 01:46:04 monitor kernel: DMA: 84*4kB 78*8kB 64*16kB 47*32kB 24*64kB 14*128kB 2*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10912kB
  104. Sep 27 01:46:04 monitor kernel: Normal: 13659*4kB 5943*8kB 15*16kB 10*32kB 2*64kB 3*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 105300kB
  105. Sep 27 01:46:04 monitor kernel: HighMem: 56*4kB 11*8kB 0*16kB 1*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 536kB
  106. Sep 27 01:46:04 monitor kernel: Swap cache: add 4375595, delete 4375609, find 1542064284/1542140980, race 145+69
  107. Sep 27 01:46:04 monitor kernel: Free swap = 0kB
  108. Sep 27 01:46:04 monitor kernel: Total swap = 3911816kB
  109. Sep 27 01:46:04 monitor kernel: Out of memory: kill process 4434 (pbx_log.pl) score 9129 or a child
  110. Sep 27 01:46:04 monitor kernel: Killed process 4434 (pbx_log.pl)
  111. Sep 27 01:46:04 monitor kernel: python invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  112. Sep 27 01:46:04 monitor kernel: Pid: 4464, comm: python Not tainted 2.6.24.3 #3
  113. Sep 27 01:46:04 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  114. Sep 27 01:46:04 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  115. Sep 27 01:46:04 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  116. Sep 27 01:46:04 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  117. Sep 27 01:46:04 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  118. Sep 27 01:46:04 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  119. Sep 27 01:46:04 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  120. Sep 27 01:46:04 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  121. Sep 27 01:46:04 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  122. Sep 27 01:46:04 monitor kernel: [<c012be98>] sys_rt_sigaction+0x68/0xa0
  123. Sep 27 01:46:04 monitor kernel: [<c017fba4>] sys_poll+0x34/0x80
  124. Sep 27 01:46:04 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  125. Sep 27 01:46:04 monitor kernel: [<c043a202>] error_code+0x72/0x78
  126. Sep 27 01:46:04 monitor kernel: [<c0430000>] rpcb_getport_sync+0x0/0x190
  127. Sep 27 01:46:04 monitor kernel: =======================
  128. Sep 27 01:46:04 monitor kernel: DMA per-cpu:
  129. Sep 27 01:46:04 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  130. Sep 27 01:46:04 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  131. Sep 27 01:46:04 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  132. Sep 27 01:46:04 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  133. Sep 27 01:46:04 monitor kernel: Normal per-cpu:
  134. Sep 27 01:46:04 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 114 Cold: hi: 62, btch: 15 usd: 47
  135. Sep 27 01:46:04 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 64 Cold: hi: 62, btch: 15 usd: 57
  136. Sep 27 01:46:04 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 182 Cold: hi: 62, btch: 15 usd: 29
  137. Sep 27 01:46:04 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 74 Cold: hi: 62, btch: 15 usd: 49
  138. Sep 27 01:46:04 monitor kernel: HighMem per-cpu:
  139. Sep 27 01:46:04 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 54 Cold: hi: 62, btch: 15 usd: 19
  140. Sep 27 01:46:04 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 26 Cold: hi: 62, btch: 15 usd: 47
  141. Sep 27 01:46:04 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 164 Cold: hi: 62, btch: 15 usd: 54
  142. Sep 27 01:46:04 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 117 Cold: hi: 62, btch: 15 usd: 53
  143. Sep 27 01:46:04 monitor kernel: Active:557391 inactive:302111 dirty:0 writeback:0 unstable:0
  144. Sep 27 01:46:04 monitor kernel: free:29171 slab:4606 mapped:8 pagetables:3986 bounce:0
  145. Sep 27 01:46:04 monitor kernel: DMA free:10912kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  146. able? yes
  147. Sep 27 01:46:04 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  148. Sep 27 01:46:04 monitor kernel: Normal free:105296kB min:3744kB low:4680kB high:5616kB active:664460kB inactive:47796kB present:894080kB pages_scan
  149. ned:1600189 all_unreclaimable? yes
  150. Sep 27 01:46:04 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  151. Sep 27 01:46:04 monitor kernel: HighMem free:476kB min:512kB low:3916kB high:7324kB active:1565104kB inactive:1160648kB present:3251200kB pages_sca
  152. nned:16802134 all_unreclaimable? yes
  153. Sep 27 01:46:04 monitor kernel: lowmem_reserve[]: 0 0 0 0
  154. Sep 27 01:46:04 monitor kernel: DMA: 84*4kB 78*8kB 64*16kB 47*32kB 24*64kB 14*128kB 2*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10912kB
  155. Sep 27 01:46:04 monitor kernel: Normal: 13620*4kB 5956*8kB 18*16kB 10*32kB 2*64kB 3*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 105296kB
  156. Sep 27 01:46:04 monitor kernel: HighMem: 41*4kB 11*8kB 0*16kB 1*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 476kB
  157. Sep 27 01:46:04 monitor kernel: Swap cache: add 4375595, delete 4375609, find 1542064284/1542140980, race 145+69
  158. Sep 27 01:46:04 monitor kernel: Free swap = 0kB
  159. Sep 27 01:46:04 monitor kernel: Total swap = 3911816kB
  160. Sep 27 01:46:04 monitor kernel: Out of memory: kill process 4519 (postgres) score 9125 or a child
  161. Sep 27 01:46:04 monitor kernel: Killed process 4519 (postgres)
  162. Sep 27 01:46:04 monitor kernel: xmond-pingWorke invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  163. Sep 27 01:46:04 monitor kernel: Pid: 27205, comm: xmond-pingWorke Not tainted 2.6.24.3 #3
  164. Sep 27 01:46:04 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  165. Sep 27 01:46:04 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  166. Sep 27 01:46:04 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  167. Sep 27 01:46:04 monitor kernel: [<c011abaa>] enqueue_task_fair+0x1a/0x30
  168. Sep 27 01:46:04 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  169. Sep 27 01:46:04 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  170. Sep 27 01:46:04 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  171. Sep 27 01:46:04 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  172. Sep 27 01:46:04 monitor kernel: [<c015f2a2>] free_pgtables+0x22/0xb0
  173. Sep 27 01:46:04 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  174. Sep 27 01:46:04 monitor kernel: [<c0160509>] remove_vma+0x39/0x50
  175. Sep 27 01:46:04 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  176. Sep 27 01:46:04 monitor kernel: [<c0161077>] do_munmap+0x197/0x1f0
  177. Sep 27 01:46:04 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  178. Sep 27 01:46:04 monitor kernel: [<c043a202>] error_code+0x72/0x78
  179. Sep 27 01:46:04 monitor kernel: [<c0430000>] rpcb_getport_sync+0x0/0x190
  180. Sep 27 01:46:04 monitor kernel: =======================
  181. Sep 27 01:46:04 monitor kernel: DMA per-cpu:
  182. Sep 27 01:46:04 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  183. Sep 27 01:46:04 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  184. Sep 27 01:46:04 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  185. Sep 27 01:46:04 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  186. Sep 27 01:46:04 monitor kernel: Normal per-cpu:
  187. Sep 27 01:46:04 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 92 Cold: hi: 62, btch: 15 usd: 46
  188. Sep 27 01:46:04 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 66 Cold: hi: 62, btch: 15 usd: 57
  189. Sep 27 01:46:04 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 165 Cold: hi: 62, btch: 15 usd: 24
  190. Sep 27 01:46:04 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 80 Cold: hi: 62, btch: 15 usd: 44
  191. Sep 27 01:46:04 monitor kernel: HighMem per-cpu:
  192. Sep 27 01:46:04 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 87 Cold: hi: 62, btch: 15 usd: 60
  193. Sep 27 01:46:04 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 47 Cold: hi: 62, btch: 15 usd: 21
  194. Sep 27 01:46:04 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 56 Cold: hi: 62, btch: 15 usd: 23
  195. Sep 27 01:46:04 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 140 Cold: hi: 62, btch: 15 usd: 61
  196. Sep 27 01:46:04 monitor kernel: Active:208871 inactive:650486 dirty:0 writeback:0 unstable:0
  197. Sep 27 01:46:04 monitor kernel: free:29118 slab:4650 mapped:8 pagetables:3977 bounce:0
  198. Sep 27 01:46:04 monitor kernel: DMA free:10912kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  199. able? yes
  200. Sep 27 01:46:04 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  201. Sep 27 01:46:04 monitor kernel: Normal free:105056kB min:3744kB low:4680kB high:5616kB active:350856kB inactive:361228kB present:894080kB pages_sca
  202. nned:2413885 all_unreclaimable? yes
  203. Sep 27 01:46:04 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  204. Sep 27 01:46:04 monitor kernel: HighMem free:504kB min:512kB low:3916kB high:7324kB active:484628kB inactive:2240716kB present:3251200kB pages_scan
  205. ned:16965573 all_unreclaimable? yes
  206. Sep 27 01:46:04 monitor kernel: lowmem_reserve[]: 0 0 0 0
  207. Sep 27 01:46:04 monitor kernel: DMA: 84*4kB 78*8kB 64*16kB 47*32kB 24*64kB 14*128kB 2*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10912kB
  208. Sep 27 01:46:04 monitor kernel: Normal: 13590*4kB 5947*8kB 15*16kB 10*32kB 2*64kB 3*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 105056kB
  209. Sep 27 01:46:04 monitor kernel: HighMem: 32*4kB 17*8kB 1*16kB 1*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 504kB
  210. Sep 27 01:46:04 monitor kernel: Swap cache: add 4375676, delete 4375690, find 1542064288/1542140994, race 145+69
  211. Sep 27 01:46:04 monitor kernel: Free swap = 0kB
  212. Sep 27 01:46:04 monitor kernel: Total swap = 3911816kB
  213. Sep 27 01:46:04 monitor kernel: Out of memory: kill process 4328 (pbx_log.pl) score 8925 or a child
  214. Sep 27 01:46:04 monitor kernel: Killed process 4328 (pbx_log.pl)
  215. Sep 27 01:46:04 monitor kernel: radiusd invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  216. Sep 27 01:46:04 monitor kernel: Pid: 13863, comm: radiusd Not tainted 2.6.24.3 #3
  217. Sep 27 01:46:04 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  218. Sep 27 01:46:04 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  219. Sep 27 01:46:04 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  220. Sep 27 01:46:04 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  221. Sep 27 01:46:04 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  222. Sep 27 01:46:04 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  223. Sep 27 01:46:04 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  224. Sep 27 01:46:04 monitor kernel: [<c0239258>] rb_insert_color+0x68/0xc0
  225. Sep 27 01:46:04 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  226. Sep 27 01:46:04 monitor kernel: [<c0439209>] do_nanosleep+0x59/0x70
  227. Sep 27 01:46:04 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  228. Sep 27 01:46:04 monitor kernel: [<c0138540>] hrtimer_wakeup+0x0/0x20
  229. Sep 27 01:46:04 monitor kernel: [<c0138a38>] sys_nanosleep+0x58/0x60
  230. Sep 27 01:46:04 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  231. Sep 27 01:46:04 monitor kernel: [<c043a202>] error_code+0x72/0x78
  232. Sep 27 01:46:04 monitor kernel: =======================
  233. Sep 27 01:46:04 monitor kernel: DMA per-cpu:
  234. Sep 27 01:46:04 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  235. Sep 27 01:46:04 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  236. Sep 27 01:46:04 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  237. Sep 27 01:46:04 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  238. Sep 27 01:46:04 monitor kernel: Normal per-cpu:
  239. Sep 27 01:46:04 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 92 Cold: hi: 62, btch: 15 usd: 49
  240. Sep 27 01:46:04 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 66 Cold: hi: 62, btch: 15 usd: 60
  241. Sep 27 01:46:04 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 165 Cold: hi: 62, btch: 15 usd: 25
  242. Sep 27 01:46:04 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 77 Cold: hi: 62, btch: 15 usd: 45
  243. Sep 27 01:46:04 monitor kernel: HighMem per-cpu:
  244. Sep 27 01:46:04 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 87 Cold: hi: 62, btch: 15 usd: 60
  245. Sep 27 01:46:04 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 47 Cold: hi: 62, btch: 15 usd: 21
  246. Sep 27 01:46:04 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 56 Cold: hi: 62, btch: 15 usd: 23
  247. Sep 27 01:46:04 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 140 Cold: hi: 62, btch: 15 usd: 61
  248. Sep 27 01:46:04 monitor kernel: Active:579499 inactive:280084 dirty:0 writeback:0 unstable:0
  249. Sep 27 01:46:04 monitor kernel: free:29120 slab:4653 mapped:8 pagetables:3977 bounce:0
  250. Sep 27 01:46:04 monitor kernel: DMA free:10912kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  251. able? yes
  252. Sep 27 01:46:04 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  253. Sep 27 01:46:04 monitor kernel: Normal free:105064kB min:3744kB low:4680kB high:5616kB active:204852kB inactive:493308kB present:894080kB pages_sca
  254. nned:36727014 all_unreclaimable? yes
  255. Sep 27 01:46:04 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  256. Sep 27 01:46:04 monitor kernel: HighMem free:504kB min:512kB low:3916kB high:7324kB active:2113292kB inactive:626900kB present:3251200kB pages_scan
  257. ned:32906415 all_unreclaimable? yes
  258. Sep 27 01:46:04 monitor kernel: lowmem_reserve[]: 0 0 0 0
  259. Sep 27 01:46:04 monitor kernel: DMA: 84*4kB 78*8kB 64*16kB 47*32kB 24*64kB 14*128kB 2*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10912kB
  260. Sep 27 01:46:04 monitor kernel: Normal: 13590*4kB 5948*8kB 15*16kB 10*32kB 2*64kB 3*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 105064kB
  261. Sep 27 01:46:04 monitor kernel: HighMem: 32*4kB 17*8kB 1*16kB 1*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 504kB
  262. Sep 27 01:46:04 monitor kernel: Swap cache: add 4375676, delete 4375690, find 1542064288/1542140994, race 145+69
  263. Sep 27 01:46:04 monitor kernel: Free swap = 0kB
  264. Sep 27 01:46:04 monitor kernel: Total swap = 3911816kB
  265. Sep 27 01:46:04 monitor kernel: Out of memory: kill process 4523 (postgres) score 8887 or a child
  266. Sep 27 01:46:04 monitor kernel: Killed process 4523 (postgres)
  267. Sep 27 01:46:04 monitor kernel: postgres invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  268. Sep 27 01:46:04 monitor kernel: Pid: 27166, comm: postgres Not tainted 2.6.24.3 #3
  269. Sep 27 01:46:04 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  270. Sep 27 01:46:04 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  271. Sep 27 01:46:04 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  272. Sep 27 01:46:04 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  273. Sep 27 01:46:04 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  274. Sep 27 01:46:04 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  275. Sep 27 01:46:04 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  276. Sep 27 01:46:04 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  277. Sep 27 01:46:04 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  278. Sep 27 01:46:04 monitor kernel: [<c0172ed5>] vfs_read+0x115/0x150
  279. Sep 27 01:46:04 monitor kernel: [<c01724b0>] do_sync_read+0x0/0x110
  280. Sep 27 01:46:04 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  281. Sep 27 01:46:04 monitor kernel: [<c043a202>] error_code+0x72/0x78
  282. Sep 27 01:46:04 monitor kernel: [<c0430000>] rpcb_getport_sync+0x0/0x190
  283. Sep 27 01:46:04 monitor kernel: =======================
  284. Sep 27 01:46:04 monitor kernel: DMA per-cpu:
  285. Sep 27 01:46:04 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  286. Sep 27 01:46:04 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  287. Sep 27 01:46:04 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  288. Sep 27 01:46:04 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  289. Sep 27 01:46:04 monitor kernel: Normal per-cpu:
  290. Sep 27 01:46:04 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 92 Cold: hi: 62, btch: 15 usd: 49
  291. Sep 27 01:46:04 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 65 Cold: hi: 62, btch: 15 usd: 60
  292. Sep 27 01:46:04 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 173 Cold: hi: 62, btch: 15 usd: 25
  293. Sep 27 01:46:04 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 77 Cold: hi: 62, btch: 15 usd: 45
  294. Sep 27 01:46:04 monitor kernel: HighMem per-cpu:
  295. Sep 27 01:46:04 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 87 Cold: hi: 62, btch: 15 usd: 60
  296. Sep 27 01:46:04 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 47 Cold: hi: 62, btch: 15 usd: 21
  297. Sep 27 01:46:04 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 73 Cold: hi: 62, btch: 15 usd: 22
  298. Sep 27 01:46:04 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 140 Cold: hi: 62, btch: 15 usd: 61
  299. Sep 27 01:46:04 monitor kernel: Active:481682 inactive:378739 dirty:0 writeback:0 unstable:0
  300. Sep 27 01:46:04 monitor kernel: free:29120 slab:4654 mapped:8 pagetables:3977 bounce:0
  301. Sep 27 01:46:04 monitor kernel: DMA free:10912kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  302. able? yes
  303. Sep 27 01:46:04 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  304. Sep 27 01:46:04 monitor kernel: Normal free:105064kB min:3744kB low:4680kB high:5616kB active:239232kB inactive:461500kB present:894080kB pages_sca
  305. nned:39240188 all_unreclaimable? yes
  306. Sep 27 01:46:04 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  307. Sep 27 01:46:04 monitor kernel: HighMem free:504kB min:512kB low:3916kB high:7324kB active:1687368kB inactive:1053456kB present:3251200kB pages_sca
  308. nned:34086349 all_unreclaimable? yes
  309. Sep 27 01:46:04 monitor kernel: lowmem_reserve[]: 0 0 0 0
  310. Sep 27 01:46:04 monitor kernel: DMA: 84*4kB 78*8kB 64*16kB 47*32kB 24*64kB 14*128kB 2*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10912kB
  311. Sep 27 01:46:04 monitor kernel: Normal: 13590*4kB 5948*8kB 15*16kB 10*32kB 2*64kB 3*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 105064kB
  312. Sep 27 01:46:04 monitor kernel: HighMem: 32*4kB 17*8kB 1*16kB 1*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 504kB
  313. Sep 27 01:46:04 monitor kernel: Swap cache: add 4375676, delete 4375690, find 1542064288/1542140994, race 145+69
  314. Sep 27 01:46:04 monitor kernel: Free swap = 0kB
  315. Sep 27 01:46:04 monitor kernel: Total swap = 3911816kB
  316. Sep 27 01:46:05 monitor kernel: Out of memory: kill process 4528 (postgres) score 8887 or a child
  317. Sep 27 01:46:05 monitor kernel: Killed process 4528 (postgres)
  318. Sep 27 01:46:05 monitor kernel: crond invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  319. Sep 27 01:46:05 monitor kernel: Pid: 13851, comm: crond Not tainted 2.6.24.3 #3
  320. Sep 27 01:46:05 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  321. Sep 27 01:46:05 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  322. Sep 27 01:46:05 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  323. Sep 27 01:46:05 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  324. Sep 27 01:46:05 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  325. Sep 27 01:46:05 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  326. Sep 27 01:46:05 monitor kernel: [<c01736bd>] put_filp+0x3d/0x50
  327. Sep 27 01:46:05 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  328. Sep 27 01:46:05 monitor kernel: [<c017d0d6>] open_namei+0x66/0x5b0
  329. Sep 27 01:46:05 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  330. Sep 27 01:46:05 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  331. Sep 27 01:46:05 monitor kernel: [<c0170dbc>] do_sys_open+0xbc/0xe0
  332. Sep 27 01:46:05 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  333. Sep 27 01:46:05 monitor kernel: [<c043a202>] error_code+0x72/0x78
  334. Sep 27 01:46:05 monitor kernel: [<c0430000>] rpcb_getport_sync+0x0/0x190
  335. Sep 27 01:46:05 monitor kernel: =======================
  336. Sep 27 01:46:05 monitor kernel: DMA per-cpu:
  337. Sep 27 01:46:05 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  338. Sep 27 01:46:05 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  339. Sep 27 01:46:05 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  340. Sep 27 01:46:05 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  341. Sep 27 01:46:05 monitor kernel: Normal per-cpu:
  342. Sep 27 01:46:05 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 92 Cold: hi: 62, btch: 15 usd: 49
  343. Sep 27 01:46:05 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 65 Cold: hi: 62, btch: 15 usd: 60
  344. Sep 27 01:46:05 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 176 Cold: hi: 62, btch: 15 usd: 25
  345. Sep 27 01:46:05 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 77 Cold: hi: 62, btch: 15 usd: 45
  346. Sep 27 01:46:05 monitor kernel: HighMem per-cpu:
  347. Sep 27 01:46:05 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 87 Cold: hi: 62, btch: 15 usd: 60
  348. Sep 27 01:46:05 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 47 Cold: hi: 62, btch: 15 usd: 21
  349. Sep 27 01:46:05 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 91 Cold: hi: 62, btch: 15 usd: 13
  350. Sep 27 01:46:05 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 140 Cold: hi: 62, btch: 15 usd: 61
  351. Sep 27 01:46:05 monitor kernel: Active:486671 inactive:373847 dirty:0 writeback:0 unstable:0
  352. Sep 27 01:46:05 monitor kernel: free:29120 slab:4654 mapped:8 pagetables:3977 bounce:0
  353. Sep 27 01:46:05 monitor kernel: DMA free:10912kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  354. able? yes
  355. Sep 27 01:46:05 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  356. Sep 27 01:46:05 monitor kernel: Normal free:105064kB min:3744kB low:4680kB high:5616kB active:269052kB inactive:432236kB present:894080kB pages_sca
  357. nned:39345344 all_unreclaimable? yes
  358. Sep 27 01:46:05 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  359. Sep 27 01:46:05 monitor kernel: HighMem free:504kB min:512kB low:3916kB high:7324kB active:1677504kB inactive:1063152kB present:3251200kB pages_sca
  360. nned:34158914 all_unreclaimable? yes
  361. Sep 27 01:46:05 monitor kernel: lowmem_reserve[]: 0 0 0 0
  362. Sep 27 01:46:05 monitor kernel: DMA: 84*4kB 78*8kB 64*16kB 47*32kB 24*64kB 14*128kB 2*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10912kB
  363. Sep 27 01:46:05 monitor kernel: Normal: 13590*4kB 5948*8kB 15*16kB 10*32kB 2*64kB 3*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 105064kB
  364. Sep 27 01:46:05 monitor kernel: HighMem: 17*4kB 17*8kB 1*16kB 1*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 444kB
  365. Sep 27 01:46:05 monitor kernel: Swap cache: add 4375676, delete 4375690, find 1542064288/1542140994, race 145+69
  366. Sep 27 01:46:05 monitor kernel: Free swap = 0kB
  367. Sep 27 01:46:05 monitor kernel: Total swap = 3911816kB
  368. Sep 27 01:46:05 monitor kernel: Out of memory: kill process 3821 (pbx_log.pl) score 8061 or a child
  369. Sep 27 01:46:05 monitor kernel: Killed process 3821 (pbx_log.pl)
  370. Sep 27 01:46:05 monitor kernel: nagios invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  371. Sep 27 01:46:05 monitor kernel: Pid: 4529, comm: nagios Not tainted 2.6.24.3 #3
  372. Sep 27 01:46:05 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  373. Sep 27 01:46:05 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  374. Sep 27 01:46:05 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  375. Sep 27 01:46:05 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  376. Sep 27 01:46:05 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  377. Sep 27 01:46:05 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  378. Sep 27 01:46:05 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  379. Sep 27 01:46:05 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  380. Sep 27 01:46:05 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  381. Sep 27 01:46:05 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  382. Sep 27 01:46:05 monitor kernel: [<c043a202>] error_code+0x72/0x78
  383. Sep 27 01:46:05 monitor kernel: [<c0430000>] rpcb_getport_sync+0x0/0x190
  384. Sep 27 01:46:05 monitor kernel: =======================
  385. Sep 27 01:46:05 monitor kernel: DMA per-cpu:
  386. Sep 27 01:46:05 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  387. Sep 27 01:46:05 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  388. Sep 27 01:46:05 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  389. Sep 27 01:46:05 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  390. Sep 27 01:46:05 monitor kernel: Normal per-cpu:
  391. Sep 27 01:46:05 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 92 Cold: hi: 62, btch: 15 usd: 49
  392. Sep 27 01:46:05 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 65 Cold: hi: 62, btch: 15 usd: 60
  393. Sep 27 01:46:05 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 176 Cold: hi: 62, btch: 15 usd: 33
  394. Sep 27 01:46:05 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 76 Cold: hi: 62, btch: 15 usd: 45
  395. Sep 27 01:46:05 monitor kernel: HighMem per-cpu:
  396. Sep 27 01:46:05 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 87 Cold: hi: 62, btch: 15 usd: 50
  397. Sep 27 01:46:05 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 47 Cold: hi: 62, btch: 15 usd: 21
  398. Sep 27 01:46:05 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 91 Cold: hi: 62, btch: 15 usd: 30
  399. Sep 27 01:46:05 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 140 Cold: hi: 62, btch: 15 usd: 61
  400. Sep 27 01:46:05 monitor kernel: Active:200883 inactive:658681 dirty:0 writeback:0 unstable:0
  401. Sep 27 01:46:05 monitor kernel: free:29107 slab:4655 mapped:8 pagetables:3971 bounce:0
  402. Sep 27 01:46:05 monitor kernel: DMA free:10912kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  403. able? yes
  404. Sep 27 01:46:05 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  405. Sep 27 01:46:05 monitor kernel: Normal free:105072kB min:3744kB low:4680kB high:5616kB active:404572kB inactive:302112kB present:894080kB pages_sca
  406. nned:9486334 all_unreclaimable? yes
  407. Sep 27 01:46:05 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  408. Sep 27 01:46:05 monitor kernel: HighMem free:444kB min:512kB low:3916kB high:7324kB active:399052kB inactive:2332740kB present:3251200kB pages_scan
  409. ned:6487559 all_unreclaimable? yes
  410. Sep 27 01:46:05 monitor kernel: lowmem_reserve[]: 0 0 0 0
  411. Sep 27 01:46:05 monitor kernel: DMA: 84*4kB 78*8kB 64*16kB 47*32kB 24*64kB 14*128kB 2*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10912kB
  412. Sep 27 01:46:05 monitor kernel: Normal: 13590*4kB 5949*8kB 15*16kB 10*32kB 2*64kB 3*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 105072kB
  413. Sep 27 01:46:05 monitor kernel: HighMem: 31*4kB 10*8kB 1*16kB 1*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 444kB
  414. Sep 27 01:46:05 monitor kernel: Swap cache: add 4375676, delete 4375690, find 1542064288/1542140994, race 145+69
  415. Sep 27 01:46:05 monitor kernel: Free swap = 0kB
  416. Sep 27 01:46:05 monitor kernel: Total swap = 3911816kB
  417. Sep 27 01:46:05 monitor kernel: Out of memory: kill process 22881 (httpd) score 7429 or a child
  418. Sep 27 01:46:05 monitor kernel: Killed process 22881 (httpd)
  419. Sep 27 01:46:05 monitor kernel: python invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  420. Sep 27 01:46:05 monitor kernel: Pid: 19335, comm: python Not tainted 2.6.24.3 #3
  421. Sep 27 01:46:05 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  422. Sep 27 01:46:05 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  423. Sep 27 01:46:05 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  424. Sep 27 01:46:05 monitor kernel: [<c0151ae1>] filemap_fault+0xf1/0x420
  425. Sep 27 01:46:05 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  426. Sep 27 01:46:05 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  427. Sep 27 01:46:05 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  428. Sep 27 01:46:05 monitor kernel: [<c039bdb6>] sys_socketcall+0xc6/0x280
  429. Sep 27 01:46:05 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  430. Sep 27 01:46:05 monitor kernel: [<c043a202>] error_code+0x72/0x78
  431. Sep 27 01:46:05 monitor kernel: =======================
  432. Sep 27 01:46:05 monitor kernel: DMA per-cpu:
  433. Sep 27 01:46:05 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  434. Sep 27 01:46:05 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  435. Sep 27 01:46:05 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  436. Sep 27 01:46:05 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  437. Sep 27 01:46:05 monitor kernel: Normal per-cpu:
  438. Sep 27 01:46:05 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 103 Cold: hi: 62, btch: 15 usd: 47
  439. Sep 27 01:46:05 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 64 Cold: hi: 62, btch: 15 usd: 60
  440. Sep 27 01:46:05 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 177 Cold: hi: 62, btch: 15 usd: 34
  441. Sep 27 01:46:05 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 74 Cold: hi: 62, btch: 15 usd: 45
  442. Sep 27 01:46:05 monitor kernel: HighMem per-cpu:
  443. Sep 27 01:46:05 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 171 Cold: hi: 62, btch: 15 usd: 49
  444. Sep 27 01:46:05 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 47 Cold: hi: 62, btch: 15 usd: 32
  445. Sep 27 01:46:05 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 91 Cold: hi: 62, btch: 15 usd: 47
  446. Sep 27 01:46:05 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 140 Cold: hi: 62, btch: 15 usd: 54
  447. Sep 27 01:46:05 monitor kernel: Active:163608 inactive:695978 dirty:0 writeback:0 unstable:0
  448. Sep 27 01:46:05 monitor kernel: free:29123 slab:4655 mapped:8 pagetables:3962 bounce:0
  449. Sep 27 01:46:05 monitor kernel: DMA free:10912kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  450. able? yes
  451. Sep 27 01:46:05 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  452. Sep 27 01:46:05 monitor kernel: Normal free:105132kB min:3744kB low:4680kB high:5616kB active:384028kB inactive:320036kB present:894080kB pages_sca
  453. nned:8952280 all_unreclaimable? yes
  454. Sep 27 01:46:05 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  455. Sep 27 01:46:05 monitor kernel: HighMem free:448kB min:512kB low:3916kB high:7324kB active:270404kB inactive:2463876kB present:3251200kB pages_scan
  456. ned:11018918 all_unreclaimable? yes
  457. Sep 27 01:46:05 monitor kernel: lowmem_reserve[]: 0 0 0 0
  458. Sep 27 01:46:05 monitor kernel: DMA: 84*4kB 78*8kB 64*16kB 47*32kB 24*64kB 14*128kB 2*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10912kB
  459. Sep 27 01:46:05 monitor kernel: Normal: 13581*4kB 5961*8kB 15*16kB 10*32kB 2*64kB 3*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 105132kB
  460. Sep 27 01:46:05 monitor kernel: HighMem: 32*4kB 10*8kB 1*16kB 1*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 448kB
  461. Sep 27 01:46:05 monitor kernel: Swap cache: add 4375677, delete 4375691, find 1542064288/1542140994, race 145+69
  462. Sep 27 01:46:05 monitor kernel: Free swap = 0kB
  463. Sep 27 01:46:05 monitor kernel: Total swap = 3911816kB
  464. Sep 27 01:46:05 monitor kernel: Out of memory: kill process 8851 (httpd) score 7395 or a child
  465. Sep 27 01:46:05 monitor kernel: Killed process 8851 (httpd)
  466. Sep 27 01:46:05 monitor kernel: python invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  467. Sep 27 01:46:05 monitor kernel: Pid: 19335, comm: python Not tainted 2.6.24.3 #3
  468. Sep 27 01:46:05 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  469. Sep 27 01:46:05 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  470. Sep 27 01:46:05 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  471. Sep 27 01:46:05 monitor kernel: [<c0151ae1>] filemap_fault+0xf1/0x420
  472. Sep 27 01:46:05 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  473. Sep 27 01:46:05 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  474. Sep 27 01:46:05 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  475. Sep 27 01:46:05 monitor kernel: [<c039bdb6>] sys_socketcall+0xc6/0x280
  476. Sep 27 01:46:05 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  477. Sep 27 01:46:05 monitor kernel: [<c043a202>] error_code+0x72/0x78
  478. Sep 27 01:46:05 monitor kernel: =======================
  479. Sep 27 01:46:05 monitor kernel: DMA per-cpu:
  480. Sep 27 01:46:05 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  481. Sep 27 01:46:05 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  482. Sep 27 01:46:05 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  483. Sep 27 01:46:05 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  484. Sep 27 01:46:05 monitor kernel: Normal per-cpu:
  485. Sep 27 01:46:05 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 103 Cold: hi: 62, btch: 15 usd: 47
  486. Sep 27 01:46:05 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 73 Cold: hi: 62, btch: 15 usd: 60
  487. Sep 27 01:46:05 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 177 Cold: hi: 62, btch: 15 usd: 34
  488. Sep 27 01:46:05 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 74 Cold: hi: 62, btch: 15 usd: 45
  489. Sep 27 01:46:05 monitor kernel: HighMem per-cpu:
  490. Sep 27 01:46:05 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 171 Cold: hi: 62, btch: 15 usd: 49
  491. Sep 27 01:46:05 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 163 Cold: hi: 62, btch: 15 usd: 32
  492. Sep 27 01:46:05 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 91 Cold: hi: 62, btch: 15 usd: 47
  493. Sep 27 01:46:05 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 140 Cold: hi: 62, btch: 15 usd: 55
  494. Sep 27 01:46:05 monitor kernel: Active:350710 inactive:509814 dirty:0 writeback:0 unstable:0
  495. Sep 27 01:46:05 monitor kernel: free:29123 slab:4656 mapped:8 pagetables:3962 bounce:0
  496. Sep 27 01:46:05 monitor kernel: DMA free:10912kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  497. able? yes
  498. Sep 27 01:46:05 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  499. Sep 27 01:46:05 monitor kernel: Normal free:105132kB min:3744kB low:4680kB high:5616kB active:692992kB inactive:14304kB present:894080kB pages_scan
  500. ned:9993593 all_unreclaimable? yes
  501. Sep 27 01:46:05 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  502. Sep 27 01:46:05 monitor kernel: HighMem free:448kB min:512kB low:3916kB high:7324kB active:709848kB inactive:2025064kB present:3251200kB pages_scan
  503. ned:11767616 all_unreclaimable? yes
  504. Sep 27 01:46:05 monitor kernel: lowmem_reserve[]: 0 0 0 0
  505. Sep 27 01:46:05 monitor kernel: DMA: 84*4kB 78*8kB 64*16kB 47*32kB 24*64kB 14*128kB 2*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10912kB
  506. Sep 27 01:46:05 monitor kernel: Normal: 13581*4kB 5961*8kB 15*16kB 10*32kB 2*64kB 3*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 105132kB
  507. Sep 27 01:46:05 monitor kernel: HighMem: 32*4kB 10*8kB 1*16kB 1*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 448kB
  508. Sep 27 01:46:05 monitor kernel: Swap cache: add 4375677, delete 4375691, find 1542064288/1542140994, race 145+69
  509. Sep 27 01:46:05 monitor kernel: Free swap = 0kB
  510. Sep 27 01:46:05 monitor kernel: Total swap = 3911816kB
  511. Sep 27 01:46:05 monitor kernel: Out of memory: kill process 20486 (httpd) score 7395 or a child
  512. Sep 27 01:46:05 monitor kernel: Killed process 20486 (httpd)
  513. Sep 27 02:02:58 monitor kernel: pbx_log.pl invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  514. Sep 27 02:02:58 monitor kernel: Pid: 9128, comm: pbx_log.pl Not tainted 2.6.24.3 #3
  515. Sep 27 02:02:58 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  516. Sep 27 02:02:58 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  517. Sep 27 02:02:58 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  518. Sep 27 02:02:58 monitor kernel: [<c011abaa>] enqueue_task_fair+0x1a/0x30
  519. Sep 27 02:02:58 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  520. Sep 27 02:02:58 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  521. Sep 27 02:02:58 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  522. Sep 27 02:02:58 monitor kernel: [<c011c0ee>] __wake_up+0x3e/0x60
  523. Sep 27 02:02:58 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  524. Sep 27 02:02:58 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  525. Sep 27 02:02:58 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  526. Sep 27 02:02:58 monitor kernel: [<c018815b>] mntput_no_expire+0x1b/0x80
  527. Sep 27 02:02:58 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  528. Sep 27 02:02:58 monitor kernel: [<c043a202>] error_code+0x72/0x78
  529. Sep 27 02:02:58 monitor kernel: [<c0430000>] rpcb_getport_sync+0x0/0x190
  530. Sep 27 02:02:58 monitor kernel: =======================
  531. Sep 27 02:02:58 monitor kernel: DMA per-cpu:
  532. Sep 27 02:02:58 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  533. Sep 27 02:02:58 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  534. Sep 27 02:02:58 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  535. Sep 27 02:02:58 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  536. Sep 27 02:02:58 monitor kernel: Normal per-cpu:
  537. Sep 27 02:02:58 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 125 Cold: hi: 62, btch: 15 usd: 28
  538. Sep 27 02:02:58 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 108 Cold: hi: 62, btch: 15 usd: 48
  539. Sep 27 02:02:58 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 65 Cold: hi: 62, btch: 15 usd: 49
  540. Sep 27 02:02:58 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 60 Cold: hi: 62, btch: 15 usd: 52
  541. Sep 27 02:02:58 monitor kernel: HighMem per-cpu:
  542. Sep 27 02:02:58 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 30 Cold: hi: 62, btch: 15 usd: 29
  543. Sep 27 02:02:58 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 134 Cold: hi: 62, btch: 15 usd: 61
  544. Sep 27 02:02:58 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 56 Cold: hi: 62, btch: 15 usd: 55
  545. Sep 27 02:02:58 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 34 Cold: hi: 62, btch: 15 usd: 47
  546. Sep 27 02:02:58 monitor kernel: Active:446960 inactive:413526 dirty:0 writeback:0 unstable:0
  547. Sep 27 02:02:58 monitor kernel: free:29147 slab:4439 mapped:37 pagetables:4152 bounce:0
  548. Sep 27 02:02:58 monitor kernel: DMA free:10944kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  549. able? yes
  550. Sep 27 02:02:58 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  551. Sep 27 02:02:58 monitor kernel: Normal free:105244kB min:3744kB low:4680kB high:5616kB active:355996kB inactive:356116kB present:894080kB pages_sca
  552. nned:1187013 all_unreclaimable? yes
  553. Sep 27 02:02:58 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  554. Sep 27 02:02:58 monitor kernel: HighMem free:400kB min:512kB low:3916kB high:7324kB active:1431844kB inactive:1297988kB present:3251200kB pages_sca
  555. nned:4861986 all_unreclaimable? yes
  556. Sep 27 02:02:58 monitor kernel: lowmem_reserve[]: 0 0 0 0
  557. Sep 27 02:02:58 monitor kernel: DMA: 80*4kB 72*8kB 62*16kB 45*32kB 23*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10944kB
  558. Sep 27 02:02:58 monitor kernel: Normal: 12889*4kB 6321*8kB 15*16kB 10*32kB 2*64kB 3*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 105244kB
  559. Sep 27 02:02:58 monitor kernel: HighMem: 54*4kB 9*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 400kB
  560. Sep 27 02:02:58 monitor kernel: Swap cache: add 4648795, delete 4648810, find 1542113407/1542204245, race 145+73
  561. Sep 27 02:02:58 monitor kernel: Free swap = 0kB
  562. Sep 27 02:02:58 monitor kernel: Total swap = 3911816kB
  563. Sep 27 02:02:58 monitor kernel: Out of memory: kill process 14479 (pbx_log.pl) score 11834 or a child
  564. Sep 27 02:02:58 monitor kernel: Killed process 14479 (pbx_log.pl)
  565. Sep 27 02:02:58 monitor kernel: pbx_log.pl invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  566. Sep 27 02:02:58 monitor kernel: Pid: 9268, comm: pbx_log.pl Not tainted 2.6.24.3 #3
  567. Sep 27 02:02:58 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  568. Sep 27 02:02:58 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  569. Sep 27 02:02:58 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  570. Sep 27 02:02:58 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  571. Sep 27 02:02:58 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  572. Sep 27 02:02:58 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  573. Sep 27 02:02:58 monitor kernel: [<c040357f>] inet_stream_connect+0x4f/0x270
  574. Sep 27 02:02:58 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  575. Sep 27 02:02:58 monitor kernel: [<c0184390>] d_alloc+0x100/0x180
  576. Sep 27 02:02:58 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  577. Sep 27 02:02:58 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  578. Sep 27 02:02:58 monitor kernel: [<c017fe72>] sys_select+0x182/0x1b0
  579. Sep 27 02:02:58 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  580. Sep 27 02:02:58 monitor kernel: [<c043a202>] error_code+0x72/0x78
  581. Sep 27 02:02:58 monitor kernel: =======================
  582. Sep 27 02:02:58 monitor kernel: DMA per-cpu:
  583. Sep 27 02:02:58 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  584. Sep 27 02:02:58 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  585. Sep 27 02:02:58 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  586. Sep 27 02:02:58 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  587. Sep 27 02:02:58 monitor kernel: Normal per-cpu:
  588. Sep 27 02:02:58 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 112 Cold: hi: 62, btch: 15 usd: 30
  589. Sep 27 02:02:58 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 108 Cold: hi: 62, btch: 15 usd: 49
  590. Sep 27 02:02:58 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 65 Cold: hi: 62, btch: 15 usd: 54
  591. Sep 27 02:02:58 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 60 Cold: hi: 62, btch: 15 usd: 57
  592. Sep 27 02:02:58 monitor kernel: HighMem per-cpu:
  593. Sep 27 02:02:58 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 30 Cold: hi: 62, btch: 15 usd: 33
  594. Sep 27 02:02:58 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 134 Cold: hi: 62, btch: 15 usd: 47
  595. Sep 27 02:02:58 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 56 Cold: hi: 62, btch: 15 usd: 56
  596. Sep 27 02:02:58 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 34 Cold: hi: 62, btch: 15 usd: 50
  597. Sep 27 02:02:58 monitor kernel: Active:155569 inactive:704749 dirty:0 writeback:0 unstable:0
  598. Sep 27 02:02:58 monitor kernel: free:29183 slab:4449 mapped:20 pagetables:4152 bounce:0
  599. Sep 27 02:02:58 monitor kernel: DMA free:10944kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  600. able? yes
  601. Sep 27 02:02:58 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  602. Sep 27 02:02:58 monitor kernel: Normal free:105328kB min:3744kB low:4680kB high:5616kB active:620952kB inactive:77316kB present:894080kB pages_scan
  603. ned:17034592 all_unreclaimable? yes
  604. Sep 27 02:02:58 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  605. Sep 27 02:02:58 monitor kernel: HighMem free:460kB min:512kB low:3916kB high:7324kB active:1324kB inactive:2741552kB present:3251200kB pages_scanne
  606. d:37720282 all_unreclaimable? yes
  607. Sep 27 02:02:58 monitor kernel: lowmem_reserve[]: 0 0 0 0
  608. Sep 27 02:02:58 monitor kernel: DMA: 80*4kB 72*8kB 62*16kB 45*32kB 23*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10944kB
  609. Sep 27 02:02:58 monitor kernel: Normal: 12904*4kB 6322*8kB 16*16kB 10*32kB 2*64kB 3*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 105328kB
  610. Sep 27 02:02:58 monitor kernel: HighMem: 69*4kB 9*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 460kB
  611. Sep 27 02:02:58 monitor kernel: Swap cache: add 4648795, delete 4648810, find 1542113407/1542204245, race 145+73
  612. Sep 27 02:02:58 monitor kernel: Free swap = 0kB
  613. Sep 27 02:02:58 monitor kernel: Total swap = 3911816kB
  614. Sep 27 02:02:58 monitor kernel: Out of memory: kill process 14469 (pbx_log.pl) score 10565 or a child
  615. Sep 27 02:02:58 monitor kernel: Killed process 14469 (pbx_log.pl)
  616. Sep 27 02:02:58 monitor kernel: hald invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  617. Sep 27 02:02:58 monitor kernel: Pid: 3293, comm: hald Not tainted 2.6.24.3 #3
  618. Sep 27 02:02:58 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  619. Sep 27 02:02:58 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  620. Sep 27 02:02:58 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  621. Sep 27 02:02:58 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  622. Sep 27 02:02:58 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  623. Sep 27 02:02:58 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  624. Sep 27 02:02:58 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  625. Sep 27 02:02:58 monitor kernel: [<c017c2d3>] do_path_lookup+0x73/0x1b0
  626. Sep 27 02:02:58 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  627. Sep 27 02:02:58 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  628. Sep 27 02:02:58 monitor kernel: [<c017fba4>] sys_poll+0x34/0x80
  629. Sep 27 02:02:58 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  630. Sep 27 02:02:58 monitor kernel: [<c043a202>] error_code+0x72/0x78
  631. Sep 27 02:02:58 monitor kernel: [<c0430000>] rpcb_getport_sync+0x0/0x190
  632. Sep 27 02:02:58 monitor kernel: =======================
  633. Sep 27 02:02:58 monitor kernel: DMA per-cpu:
  634. Sep 27 02:02:58 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  635. Sep 27 02:02:58 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  636. Sep 27 02:02:58 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  637. Sep 27 02:02:58 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  638. Sep 27 02:02:58 monitor kernel: Normal per-cpu:
  639. Sep 27 02:02:58 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 112 Cold: hi: 62, btch: 15 usd: 30
  640. Sep 27 02:02:58 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 108 Cold: hi: 62, btch: 15 usd: 49
  641. Sep 27 02:02:58 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 65 Cold: hi: 62, btch: 15 usd: 54
  642. Sep 27 02:02:58 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 60 Cold: hi: 62, btch: 15 usd: 57
  643. Sep 27 02:02:58 monitor kernel: HighMem per-cpu:
  644. Sep 27 02:02:58 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 30 Cold: hi: 62, btch: 15 usd: 33
  645. Sep 27 02:02:58 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 134 Cold: hi: 62, btch: 15 usd: 47
  646. Sep 27 02:02:58 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 56 Cold: hi: 62, btch: 15 usd: 56
  647. Sep 27 02:02:58 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 34 Cold: hi: 62, btch: 15 usd: 50
  648. Sep 27 02:02:58 monitor kernel: Active:159174 inactive:701127 dirty:0 writeback:0 unstable:0
  649. Sep 27 02:02:58 monitor kernel: free:29183 slab:4449 mapped:20 pagetables:4152 bounce:0
  650. Sep 27 02:02:58 monitor kernel: DMA free:10944kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  651. able? yes
  652. Sep 27 02:02:58 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  653. Sep 27 02:02:58 monitor kernel: Normal free:105328kB min:3744kB low:4680kB high:5616kB active:624812kB inactive:73352kB present:894080kB pages_scan
  654. ned:17072546 all_unreclaimable? yes
  655. Sep 27 02:02:58 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  656. Sep 27 02:02:58 monitor kernel: HighMem free:460kB min:512kB low:3916kB high:7324kB active:11884kB inactive:2731056kB present:3251200kB pages_scann
  657. ed:37723002 all_unreclaimable? yes
  658. Sep 27 02:02:58 monitor kernel: lowmem_reserve[]: 0 0 0 0
  659. Sep 27 02:02:58 monitor kernel: DMA: 80*4kB 72*8kB 62*16kB 45*32kB 23*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10944kB
  660. Sep 27 02:02:58 monitor kernel: Normal: 12904*4kB 6322*8kB 16*16kB 10*32kB 2*64kB 3*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 105328kB
  661. Sep 27 02:02:58 monitor kernel: HighMem: 69*4kB 9*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 460kB
  662. Sep 27 02:02:58 monitor kernel: Swap cache: add 4648795, delete 4648810, find 1542113407/1542204245, race 145+73
  663. Sep 27 02:02:58 monitor kernel: Free swap = 0kB
  664. Sep 27 02:02:58 monitor kernel: Total swap = 3911816kB
  665. Sep 27 02:02:58 monitor kernel: Out of memory: kill process 14828 (postgres) score 9199 or a child
  666. Sep 27 02:02:58 monitor kernel: Killed process 14828 (postgres)
  667. Sep 27 02:15:30 monitor kernel: pbx_log.pl invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  668. Sep 27 02:15:30 monitor kernel: Pid: 9232, comm: pbx_log.pl Not tainted 2.6.24.3 #3
  669. Sep 27 02:15:30 monitor kernel: [<c01527ad>] 4>] __alloc_pages+0x264/0x360
  670. Sep 27 02:15:30 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  671. Sep 27 02:15:30 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  672. Sep 27 02:15:30 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  673. Sep 27 02:15:30 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  674. Sep 27 02:15:30 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  675. Sep 27 02:15:30 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  676. Sep 27 02:15:30 monitor kernel: [<c0126af8>] sys_gettimeofday+0x28/0x80
  677. Sep 27 02:15:30 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  678. Sep 27 02:15:30 monitor kernel: [<c043a202>] error_code+0x72/0x78
  679. Sep 27 02:15:30 monitor kernel: =======================
  680. Sep 27 02:15:30 monitor kernel: DMA per-cpu:
  681. Sep 27 02:15:30 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  682. Sep 27 02:15:30 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  683. Sep 27 02:15:30 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  684. Sep 27 02:15:30 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  685. Sep 27 02:15:30 monitor kernel: Normal per-cpu:
  686. Sep 27 02:15:30 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 161 Cold: hi: 62, btch: 15 usd: 47
  687. Sep 27 02:15:30 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 162 Cold: hi: 62, btch: 15 usd: 53
  688. Sep 27 02:15:30 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 110 Cold: hi: 62, btch: 15 usd: 37
  689. Sep 27 02:15:30 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 72 Cold: hi: 62, btch: 15 usd: 43
  690. Sep 27 02:15:30 monitor kernel: HighMem per-cpu:
  691. Sep 27 02:15:30 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 4 Cold: hi: 62, btch: 15 usd: 43
  692. Sep 27 02:15:30 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 36 Cold: hi: 62, btch: 15 usd: 18
  693. Sep 27 02:15:30 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 14 Cold: hi: 62, btch: 15 usd: 56
  694. Sep 27 02:15:30 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 30 Cold: hi: 62, btch: 15 usd: 57
  695. Sep 27 02:15:30 monitor kernel: Active:180313 inactive:681268 dirty:0 writeback:0 unstable:0
  696. Sep 27 02:15:30 monitor kernel: free:29150 slab:4458 mapped:21 pagetables:3941 bounce:0
  697. Sep 27 02:15:30 monitor kernel: DMA free:10944kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  698. able? yes
  699. Sep 27 02:15:30 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  700. Sep 27 02:15:30 monitor kernel: Normal free:105184kB min:3744kB low:4680kB high:5616kB active:357440kB inactive:355140kB present:894080kB pages_sca
  701. nned:2853783 all_unreclaimable? yes
  702. Sep 27 02:15:30 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  703. Sep 27 02:15:30 monitor kernel: HighMem free:472kB min:512kB low:3916kB high:7324kB active:363812kB inactive:2369932kB present:3251200kB pages_scan
  704. ned:19636426 all_unreclaimable? yes
  705. Sep 27 02:15:30 monitor kernel: lowmem_reserve[]: 0 0 0 0
  706. Sep 27 02:15:30 monitor kernel: DMA: 80*4kB 72*8kB 62*16kB 45*32kB 23*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10944kB
  707. Sep 27 02:15:30 monitor kernel: Normal: 13540*4kB 5988*8kB 15*16kB 10*32kB 2*64kB 3*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 105184kB
  708. Sep 27 02:15:30 monitor kernel: HighMem: 66*4kB 12*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 472kB
  709. Sep 27 02:15:30 monitor kernel: Swap cache: add 4729362, delete 4729377, find 1542142565/1542235199, race 145+75
  710. Sep 27 02:15:30 monitor kernel: Free swap = 0kB
  711. Sep 27 02:15:30 monitor kernel: Total swap = 3911816kB
  712. Sep 27 02:15:30 monitor kernel: Out of memory: kill process 22164 (pbx_log.pl) score 9049 or a child
  713. Sep 27 02:15:30 monitor kernel: Killed process 22164 (pbx_log.pl)
  714. Sep 27 02:15:30 monitor kernel: postgres invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  715. Sep 27 02:15:30 monitor kernel: Pid: 15852, comm: postgres Not tainted 2.6.24.3 #3
  716. Sep 27 02:15:30 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  717. Sep 27 02:15:30 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  718. Sep 27 02:15:30 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  719. Sep 27 02:15:30 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  720. Sep 27 02:15:30 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  721. Sep 27 02:15:30 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  722. Sep 27 02:15:30 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  723. Sep 27 02:15:30 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  724. Sep 27 02:15:30 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  725. Sep 27 02:15:30 monitor kernel: [<c0126af8>] sys_gettimeofday+0x28/0x80
  726. Sep 27 02:15:30 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  727. Sep 27 02:15:30 monitor kernel: [<c043a202>] error_code+0x72/0x78
  728. Sep 27 02:15:30 monitor kernel: =======================
  729. Sep 27 02:15:30 monitor kernel: DMA per-cpu:
  730. Sep 27 02:15:30 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  731. Sep 27 02:15:30 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  732. Sep 27 02:15:30 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  733. Sep 27 02:15:30 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  734. Sep 27 02:15:30 monitor kernel: Normal per-cpu:
  735. Sep 27 02:15:30 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 161 Cold: hi: 62, btch: 15 usd: 47
  736. Sep 27 02:15:30 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 162 Cold: hi: 62, btch: 15 usd: 53
  737. Sep 27 02:15:30 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 110 Cold: hi: 62, btch: 15 usd: 37
  738. Sep 27 02:15:30 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 72 Cold: hi: 62, btch: 15 usd: 43
  739. Sep 27 02:15:30 monitor kernel: HighMem per-cpu:
  740. Sep 27 02:15:30 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 4 Cold: hi: 62, btch: 15 usd: 43
  741. Sep 27 02:15:30 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 36 Cold: hi: 62, btch: 15 usd: 18
  742. Sep 27 02:15:30 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 14 Cold: hi: 62, btch: 15 usd: 56
  743. Sep 27 02:15:30 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 30 Cold: hi: 62, btch: 15 usd: 57
  744. Sep 27 02:15:30 monitor kernel: Active:179884 inactive:681676 dirty:0 writeback:0 unstable:0
  745. Sep 27 02:15:30 monitor kernel: free:29150 slab:4458 mapped:21 pagetables:3941 bounce:0
  746. Sep 27 02:15:30 monitor kernel: DMA free:10944kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  747. able? yes
  748. Sep 27 02:15:30 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  749. Sep 27 02:15:30 monitor kernel: Normal free:105184kB min:3744kB low:4680kB high:5616kB active:357588kB inactive:354980kB present:894080kB pages_sca
  750. nned:2853891 all_unreclaimable? yes
  751. Sep 27 02:15:30 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  752. Sep 27 02:15:30 monitor kernel: HighMem free:472kB min:512kB low:3916kB high:7324kB active:361948kB inactive:2371924kB present:3251200kB pages_scan
  753. ned:19676906 all_unreclaimable? yes
  754. Sep 27 02:15:30 monitor kernel: lowmem_reserve[]: 0 0 0 0
  755. Sep 27 02:15:30 monitor kernel: DMA: 80*4kB 72*8kB 62*16kB 45*32kB 23*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10944kB
  756. Sep 27 02:15:30 monitor kernel: Normal: 13540*4kB 5988*8kB 15*16kB 10*32kB 2*64kB 3*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 105184kB
  757. Sep 27 02:15:30 monitor kernel: HighMem: 66*4kB 12*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 472kB
  758. Sep 27 02:15:30 monitor kernel: Swap cache: add 4729362, delete 4729377, find 1542142565/1542235199, race 145+75
  759. Sep 27 02:15:30 monitor kernel: Free swap = 0kB
  760. Sep 27 02:15:30 monitor kernel: Total swap = 3911816kB
  761. Sep 27 02:15:30 monitor kernel: Out of memory: kill process 15852 (postgres) score 8955 or a child
  762. Sep 27 02:15:30 monitor kernel: Killed process 15852 (postgres)
  763. Sep 27 02:15:30 monitor kernel: nagios invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  764. Sep 27 02:15:30 monitor kernel: Pid: 11513, comm: nagios Not tainted 2.6.24.3 #3
  765. Sep 27 02:15:30 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  766. Sep 27 02:15:30 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  767. Sep 27 02:15:30 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  768. Sep 27 02:15:30 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  769. Sep 27 02:15:30 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  770. Sep 27 02:15:30 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  771. Sep 27 02:15:30 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  772. Sep 27 02:15:30 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  773. Sep 27 02:15:30 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  774. Sep 27 02:15:30 monitor kernel: [<c0138540>] hrtimer_wakeup+0x0/0x20
  775. Sep 27 02:15:30 monitor kernel: [<c0138a38>] sys_nanosleep+0x58/0x60
  776. Sep 27 02:15:30 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  777. Sep 27 02:15:30 monitor kernel: [<c043a202>] error_code+0x72/0x78
  778. Sep 27 02:15:30 monitor kernel: [<c0430000>] rpcb_getport_sync+0x0/0x190
  779. Sep 27 02:15:30 monitor kernel: =======================
  780. Sep 27 02:15:30 monitor kernel: DMA per-cpu:
  781. Sep 27 02:15:30 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  782. Sep 27 02:15:30 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  783. Sep 27 02:15:30 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  784. Sep 27 02:15:30 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  785. Sep 27 02:15:30 monitor kernel: Normal per-cpu:
  786. Sep 27 02:15:30 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 160 Cold: hi: 62, btch: 15 usd: 47
  787. Sep 27 02:15:30 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 157 Cold: hi: 62, btch: 15 usd: 31
  788. Sep 27 02:15:30 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 110 Cold: hi: 62, btch: 15 usd: 37
  789. Sep 27 02:15:30 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 72 Cold: hi: 62, btch: 15 usd: 36
  790. Sep 27 02:15:30 monitor kernel: HighMem per-cpu:
  791. Sep 27 02:15:30 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 4 Cold: hi: 62, btch: 15 usd: 57
  792. Sep 27 02:15:30 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 87 Cold: hi: 62, btch: 15 usd: 31
  793. Sep 27 02:15:30 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 14 Cold: hi: 62, btch: 15 usd: 56
  794. Sep 27 02:15:30 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 30 Cold: hi: 62, btch: 15 usd: 57
  795. Sep 27 02:15:30 monitor kernel: Active:702936 inactive:158987 dirty:0 writeback:0 unstable:0
  796. Sep 27 02:15:30 monitor kernel: free:29175 slab:4458 mapped:21 pagetables:3935 bounce:0
  797. Sep 27 02:15:30 monitor kernel: DMA free:10944kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  798. able? yes
  799. Sep 27 02:15:30 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  800. Sep 27 02:15:30 monitor kernel: Normal free:105284kB min:3744kB low:4680kB high:5616kB active:358252kB inactive:352816kB present:894080kB pages_sca
  801. nned:1814984 all_unreclaimable? yes
  802. Sep 27 02:15:30 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  803. Sep 27 02:15:30 monitor kernel: HighMem free:472kB min:512kB low:3916kB high:7324kB active:2453492kB inactive:283132kB present:3251200kB pages_scan
  804. ned:23710056 all_unreclaimable? yes
  805. Sep 27 02:15:30 monitor kernel: lowmem_reserve[]: 0 0 0 0
  806. Sep 27 02:15:30 monitor kernel: DMA: 80*4kB 72*8kB 62*16kB 45*32kB 23*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10944kB
  807. Sep 27 02:15:30 monitor kernel: Normal: 13545*4kB 5988*8kB 15*16kB 10*32kB 2*64kB 3*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 105204kB
  808. Sep 27 02:15:30 monitor kernel: HighMem: 66*4kB 12*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 472kB
  809. Sep 27 02:15:30 monitor kernel: Swap cache: add 4729362, delete 4729377, find 1542142565/1542235199, race 145+75
  810. Sep 27 02:15:30 monitor kernel: Free swap = 0kB
  811. Sep 27 02:15:30 monitor kernel: Total swap = 3911816kB
  812. Sep 27 02:15:31 monitor kernel: Out of memory: kill process 15850 (postgres) score 8887 or a child
  813. Sep 27 02:15:31 monitor kernel: Killed process 15850 (postgres)
  814. Sep 27 02:15:31 monitor kernel: nagios invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  815. Sep 27 02:15:31 monitor kernel: Pid: 11513, comm: nagios Not tainted 2.6.24.3 #3
  816. Sep 27 02:15:31 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  817. Sep 27 02:15:31 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  818. Sep 27 02:15:31 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  819. Sep 27 02:15:31 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  820. Sep 27 02:15:31 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  821. Sep 27 02:15:31 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  822. Sep 27 02:15:31 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  823. Sep 27 02:15:31 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  824. Sep 27 02:15:31 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  825. Sep 27 02:15:31 monitor kernel: [<c0138540>] hrtimer_wakeup+0x0/0x20
  826. Sep 27 02:15:31 monitor kernel: [<c0138a38>] sys_nanosleep+0x58/0x60
  827. Sep 27 02:15:31 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  828. Sep 27 02:15:31 monitor kernel: [<c043a202>] error_code+0x72/0x78
  829. Sep 27 02:15:31 monitor kernel: [<c0430000>] rpcb_getport_sync+0x0/0x190
  830. Sep 27 02:15:31 monitor kernel: =======================
  831. Sep 27 02:15:31 monitor kernel: DMA per-cpu:
  832. Sep 27 02:15:31 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  833. Sep 27 02:15:31 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  834. Sep 27 02:15:31 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  835. Sep 27 02:15:31 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  836. Sep 27 02:15:31 monitor kernel: Normal per-cpu:
  837. Sep 27 02:15:31 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 160 Cold: hi: 62, btch: 15 usd: 47
  838. Sep 27 02:15:31 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 157 Cold: hi: 62, btch: 15 usd: 31
  839. Sep 27 02:15:31 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 122 Cold: hi: 62, btch: 15 usd: 37
  840. Sep 27 02:15:31 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 72 Cold: hi: 62, btch: 15 usd: 36
  841. Sep 27 02:15:31 monitor kernel: HighMem per-cpu:
  842. Sep 27 02:15:31 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 4 Cold: hi: 62, btch: 15 usd: 57
  843. Sep 27 02:15:31 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 87 Cold: hi: 62, btch: 15 usd: 31
  844. Sep 27 02:15:31 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 53 Cold: hi: 62, btch: 15 usd: 56
  845. Sep 27 02:15:31 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 30 Cold: hi: 62, btch: 15 usd: 57
  846. Sep 27 02:15:31 monitor kernel: Active:702762 inactive:159113 dirty:0 writeback:0 unstable:0
  847. Sep 27 02:15:31 monitor kernel: free:29175 slab:4458 mapped:21 pagetables:3935 bounce:0
  848. Sep 27 02:15:31 monitor kernel: DMA free:10944kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  849. able? yes
  850. Sep 27 02:15:31 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  851. Sep 27 02:15:31 monitor kernel: Normal free:105284kB min:3744kB low:4680kB high:5616kB active:358464kB inactive:352792kB present:894080kB pages_sca
  852. nned:1830612 all_unreclaimable? yes
  853. Sep 27 02:15:31 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  854. Sep 27 02:15:31 monitor kernel: HighMem free:472kB min:512kB low:3916kB high:7324kB active:2452584kB inactive:283660kB present:3251200kB pages_scan
  855. ned:23742366 all_unreclaimable? yes
  856. Sep 27 02:15:31 monitor kernel: lowmem_reserve[]: 0 0 0 0
  857. Sep 27 02:15:31 monitor kernel: DMA: 80*4kB 72*8kB 62*16kB 45*32kB 23*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10944kB
  858. Sep 27 02:15:31 monitor kernel: Normal: 13545*4kB 5988*8kB 15*16kB 10*32kB 2*64kB 3*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 105204kB
  859. Sep 27 02:15:31 monitor kernel: HighMem: 66*4kB 12*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 472kB
  860. Sep 27 02:15:31 monitor kernel: Swap cache: add 4729362, delete 4729377, find 1542142565/1542235199, race 145+75
  861. Sep 27 02:15:31 monitor kernel: Free swap = 0kB
  862. Sep 27 02:15:31 monitor kernel: Total swap = 3911816kB
  863. Sep 27 02:15:31 monitor kernel: Out of memory: kill process 15851 (postgres) score 8887 or a child
  864. Sep 27 02:15:31 monitor kernel: Killed process 15851 (postgres)
  865. Sep 27 02:15:31 monitor kernel: nagios invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  866. Sep 27 02:15:31 monitor kernel: Pid: 11513, comm: nagios Not tainted 2.6.24.3 #3
  867. Sep 27 02:15:31 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  868. Sep 27 02:15:31 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  869. Sep 27 02:15:31 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  870. Sep 27 02:15:31 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  871. Sep 27 02:15:31 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  872. Sep 27 02:15:31 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  873. Sep 27 02:15:31 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  874. Sep 27 02:15:31 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  875. Sep 27 02:15:31 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  876. Sep 27 02:15:31 monitor kernel: [<c0138540>] hrtimer_wakeup+0x0/0x20
  877. Sep 27 02:15:31 monitor kernel: [<c0138a38>] sys_nanosleep+0x58/0x60
  878. Sep 27 02:15:31 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  879. Sep 27 02:15:31 monitor kernel: [<c043a202>] error_code+0x72/0x78
  880. Sep 27 02:15:31 monitor kernel: [<c0430000>] rpcb_getport_sync+0x0/0x190
  881. Sep 27 02:15:31 monitor kernel: =======================
  882. Sep 27 02:15:31 monitor kernel: DMA per-cpu:
  883. Sep 27 02:15:31 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  884. Sep 27 02:15:31 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  885. Sep 27 02:15:31 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  886. Sep 27 02:15:31 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  887. Sep 27 02:15:31 monitor kernel: Normal per-cpu:
  888. Sep 27 02:15:31 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 160 Cold: hi: 62, btch: 15 usd: 47
  889. Sep 27 02:15:31 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 163 Cold: hi: 62, btch: 15 usd: 31
  890. Sep 27 02:15:31 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 122 Cold: hi: 62, btch: 15 usd: 37
  891. Sep 27 02:15:31 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 72 Cold: hi: 62, btch: 15 usd: 36
  892. Sep 27 02:15:31 monitor kernel: HighMem per-cpu:
  893. Sep 27 02:15:31 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 4 Cold: hi: 62, btch: 15 usd: 57
  894. Sep 27 02:15:31 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 120 Cold: hi: 62, btch: 15 usd: 31
  895. Sep 27 02:15:31 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 53 Cold: hi: 62, btch: 15 usd: 56
  896. Sep 27 02:15:31 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 30 Cold: hi: 62, btch: 15 usd: 57
  897. Sep 27 02:15:31 monitor kernel: Active:702002 inactive:159747 dirty:0 writeback:0 unstable:0
  898. Sep 27 02:15:31 monitor kernel: free:29160 slab:4458 mapped:20 pagetables:3935 bounce:0
  899. Sep 27 02:15:31 monitor kernel: DMA free:10944kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  900. able? yes
  901. Sep 27 02:15:31 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  902. Sep 27 02:15:31 monitor kernel: Normal free:105224kB min:3744kB low:4680kB high:5616kB active:358564kB inactive:352484kB present:894080kB pages_sca
  903. nned:1857336 all_unreclaimable? yes
  904. Sep 27 02:15:31 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  905. Sep 27 02:15:31 monitor kernel: HighMem free:472kB min:512kB low:3916kB high:7324kB active:2449444kB inactive:286504kB present:3251200kB pages_scan
  906. ned:23772994 all_unreclaimable? yes
  907. Sep 27 02:15:31 monitor kernel: lowmem_reserve[]: 0 0 0 0
  908. Sep 27 02:15:31 monitor kernel: DMA: 80*4kB 72*8kB 62*16kB 45*32kB 23*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10944kB
  909. Sep 27 02:15:31 monitor kernel: Normal: 13545*4kB 5988*8kB 15*16kB 10*32kB 2*64kB 3*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 105204kB
  910. Sep 27 02:15:31 monitor kernel: HighMem: 66*4kB 12*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 472kB
  911. Sep 27 02:15:31 monitor kernel: Swap cache: add 4729362, delete 4729377, find 1542142565/1542235199, race 145+75
  912. Sep 27 02:15:31 monitor kernel: Free swap = 0kB
  913. Sep 27 02:15:31 monitor kernel: Total swap = 3911816kB
  914. Sep 27 02:15:31 monitor kernel: Out of memory: kill process 22870 (postgres) score 8887 or a child
  915. Sep 27 02:15:31 monitor kernel: Killed process 22870 (postgres)
  916. Sep 27 02:15:31 monitor kernel: postgres invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  917. Sep 27 02:15:31 monitor kernel: Pid: 17860, comm: postgres Not tainted 2.6.24.3 #3
  918. Sep 27 02:15:31 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  919. Sep 27 02:15:31 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  920. Sep 27 02:15:31 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  921. Sep 27 02:15:31 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  922. Sep 27 02:15:31 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  923. Sep 27 02:15:31 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  924. Sep 27 02:15:31 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  925. Sep 27 02:15:31 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  926. Sep 27 02:15:31 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  927. Sep 27 02:15:31 monitor kernel: [<c011bdc0>] default_wake_function+0x0/0x10
  928. Sep 27 02:15:31 monitor kernel: [<c0124fc5>] sys_waitpid+0x25/0x30
  929. Sep 27 02:15:31 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  930. Sep 27 02:15:31 monitor kernel: [<c043a202>] error_code+0x72/0x78
  931. Sep 27 02:15:31 monitor kernel: [<c0430000>] rpcb_getport_sync+0x0/0x190
  932. Sep 27 02:15:31 monitor kernel: =======================
  933. Sep 27 02:15:31 monitor kernel: DMA per-cpu:
  934. Sep 27 02:15:31 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  935. Sep 27 02:15:31 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  936. Sep 27 02:15:31 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  937. Sep 27 02:15:31 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  938. Sep 27 02:15:31 monitor kernel: Normal per-cpu:
  939. Sep 27 02:15:31 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 171 Cold: hi: 62, btch: 15 usd: 52
  940. Sep 27 02:15:31 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 171 Cold: hi: 62, btch: 15 usd: 27
  941. Sep 27 02:15:31 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 123 Cold: hi: 62, btch: 15 usd: 47
  942. Sep 27 02:15:31 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 69 Cold: hi: 62, btch: 15 usd: 38
  943. Sep 27 02:15:31 monitor kernel: HighMem per-cpu:
  944. Sep 27 02:15:31 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 27 Cold: hi: 62, btch: 15 usd: 49
  945. Sep 27 02:15:31 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 133 Cold: hi: 62, btch: 15 usd: 29
  946. Sep 27 02:15:31 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 53 Cold: hi: 62, btch: 15 usd: 34
  947. Sep 27 02:15:31 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 30 Cold: hi: 62, btch: 15 usd: 49
  948. Sep 27 02:15:31 monitor kernel: Active:482412 inactive:379079 dirty:0 writeback:0 unstable:0
  949. Sep 27 02:15:31 monitor kernel: free:29140 slab:4474 mapped:0 pagetables:3917 bounce:0
  950. Sep 27 02:15:31 monitor kernel: DMA free:10944kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  951. able? yes
  952. Sep 27 02:15:31 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  953. Sep 27 02:15:31 monitor kernel: Normal free:105148kB min:3744kB low:4680kB high:5616kB active:369084kB inactive:342564kB present:894080kB pages_sca
  954. nned:6933250 all_unreclaimable? yes
  955. Sep 27 02:15:31 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  956. Sep 27 02:15:31 monitor kernel: HighMem free:468kB min:512kB low:3916kB high:7324kB active:1560564kB inactive:1173752kB present:3251200kB pages_sca
  957. nned:12799690 all_unreclaimable? yes
  958. Sep 27 02:15:31 monitor kernel: lowmem_reserve[]: 0 0 0 0
  959. Sep 27 02:15:31 monitor kernel: DMA: 80*4kB 72*8kB 62*16kB 45*32kB 23*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10944kB
  960. Sep 27 02:15:31 monitor kernel: Normal: 13501*4kB 6003*8kB 15*16kB 10*32kB 2*64kB 3*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 105148kB
  961. Sep 27 02:15:31 monitor kernel: HighMem: 65*4kB 12*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 468kB
  962. Sep 27 02:15:31 monitor kernel: Swap cache: add 4729371, delete 4729386, find 1542142565/1542235200, race 145+75
  963. Sep 27 02:15:31 monitor kernel: Free swap = 0kB
  964. Sep 27 02:15:31 monitor kernel: Total swap = 3911816kB
  965. Sep 27 02:15:31 monitor kernel: Out of memory: kill process 22726 (pbx_log.pl) score 8315 or a child
  966. Sep 27 02:15:31 monitor kernel: Killed process 22726 (pbx_log.pl)
  967. Sep 27 02:15:31 monitor kernel: postgres invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  968. Sep 27 02:15:31 monitor kernel: Pid: 17860, comm: postgres Not tainted 2.6.24.3 #3
  969. Sep 27 02:15:31 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  970. Sep 27 02:15:31 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  971. Sep 27 02:15:31 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  972. Sep 27 02:15:31 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  973. Sep 27 02:15:31 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  974. Sep 27 02:15:31 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  975. Sep 27 02:15:31 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  976. Sep 27 02:15:31 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  977. Sep 27 02:15:31 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  978. Sep 27 02:15:31 monitor kernel: [<c011bdc0>] default_wake_function+0x0/0x10
  979. Sep 27 02:15:31 monitor kernel: [<c0124fc5>] sys_waitpid+0x25/0x30
  980. Sep 27 02:15:31 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  981. Sep 27 02:15:31 monitor kernel: [<c043a202>] error_code+0x72/0x78
  982. Sep 27 02:15:31 monitor kernel: [<c0430000>] rpcb_getport_sync+0x0/0x190
  983. Sep 27 02:15:31 monitor kernel: =======================
  984. Sep 27 02:15:31 monitor kernel: DMA per-cpu:
  985. Sep 27 02:15:31 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  986. Sep 27 02:15:31 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  987. Sep 27 02:15:31 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  988. Sep 27 02:15:31 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  989. Sep 27 02:15:31 monitor kernel: Normal per-cpu:
  990. Sep 27 02:15:31 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 171 Cold: hi: 62, btch: 15 usd: 52
  991. Sep 27 02:15:31 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 171 Cold: hi: 62, btch: 15 usd: 27
  992. Sep 27 02:15:31 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 123 Cold: hi: 62, btch: 15 usd: 47
  993. Sep 27 02:15:31 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 69 Cold: hi: 62, btch: 15 usd: 38
  994. Sep 27 02:15:31 monitor kernel: HighMem per-cpu:
  995. Sep 27 02:15:31 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 27 Cold: hi: 62, btch: 15 usd: 49
  996. Sep 27 02:15:31 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 133 Cold: hi: 62, btch: 15 usd: 29
  997. Sep 27 02:15:31 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 53 Cold: hi: 62, btch: 15 usd: 34
  998. Sep 27 02:15:31 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 30 Cold: hi: 62, btch: 15 usd: 49
  999. Sep 27 02:15:31 monitor kernel: Active:481897 inactive:379555 dirty:0 writeback:0 unstable:0
  1000. Sep 27 02:15:31 monitor kernel: free:29140 slab:4474 mapped:0 pagetables:3917 bounce:0
  1001. Sep 27 02:15:31 monitor kernel: DMA free:10944kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  1002. able? yes
  1003. Sep 27 02:15:31 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  1004. Sep 27 02:15:31 monitor kernel: Normal free:105148kB min:3744kB low:4680kB high:5616kB active:369084kB inactive:342564kB present:894080kB pages_sca
  1005. nned:6933250 all_unreclaimable? yes
  1006. Sep 27 02:15:31 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  1007. Sep 27 02:15:31 monitor kernel: HighMem free:468kB min:512kB low:3916kB high:7324kB active:1558504kB inactive:1175656kB present:3251200kB pages_sca
  1008. nned:12840490 all_unreclaimable? yes
  1009. Sep 27 02:15:31 monitor kernel: lowmem_reserve[]: 0 0 0 0
  1010. Sep 27 02:15:31 monitor kernel: DMA: 80*4kB 72*8kB 62*16kB 45*32kB 23*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10944kB
  1011. Sep 27 02:15:31 monitor kernel: Normal: 13501*4kB 6003*8kB 15*16kB 10*32kB 2*64kB 3*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 105148kB
  1012. Sep 27 02:15:31 monitor kernel: HighMem: 65*4kB 12*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 468kB
  1013. Sep 27 02:15:31 monitor kernel: Swap cache: add 4729371, delete 4729386, find 1542142565/1542235200, race 145+75
  1014. Sep 27 02:15:31 monitor kernel: Free swap = 0kB
  1015. Sep 27 02:15:31 monitor kernel: Total swap = 3911816kB
  1016. Sep 27 02:15:31 monitor kernel: Out of memory: kill process 22723 (pbx_log.pl) score 7896 or a child
  1017. Sep 27 02:15:31 monitor kernel: Killed process 22723 (pbx_log.pl)
  1018. Sep 27 02:15:31 monitor kernel: postgres invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  1019. Sep 27 02:15:31 monitor kernel: Pid: 17860, comm: postgres Not tainted 2.6.24.3 #3
  1020. Sep 27 02:15:31 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  1021. Sep 27 02:15:31 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  1022. Sep 27 02:15:31 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  1023. Sep 27 02:15:31 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  1024. Sep 27 02:15:31 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  1025. Sep 27 02:15:31 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  1026. Sep 27 02:15:31 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  1027. Sep 27 02:15:31 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  1028. Sep 27 02:15:31 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  1029. Sep 27 02:15:31 monitor kernel: [<c011bdc0>] default_wake_function+0x0/0x10
  1030. Sep 27 02:15:31 monitor kernel: [<c0124fc5>] sys_waitpid+0x25/0x30
  1031. Sep 27 02:15:31 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  1032. Sep 27 02:15:31 monitor kernel: [<c043a202>] error_code+0x72/0x78
  1033. Sep 27 02:15:31 monitor kernel: [<c0430000>] rpcb_getport_sync+0x0/0x190
  1034. Sep 27 02:15:31 monitor kernel: =======================
  1035. Sep 27 02:15:31 monitor kernel: DMA per-cpu:
  1036. Sep 27 02:15:31 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1037. Sep 27 02:15:31 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1038. Sep 27 02:15:31 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1039. Sep 27 02:15:31 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1040. Sep 27 02:15:31 monitor kernel: Normal per-cpu:
  1041. Sep 27 02:15:31 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 171 Cold: hi: 62, btch: 15 usd: 52
  1042. Sep 27 02:15:31 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 171 Cold: hi: 62, btch: 15 usd: 27
  1043. Sep 27 02:15:31 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 123 Cold: hi: 62, btch: 15 usd: 47
  1044. Sep 27 02:15:31 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 69 Cold: hi: 62, btch: 15 usd: 38
  1045. Sep 27 02:15:31 monitor kernel: HighMem per-cpu:
  1046. Sep 27 02:15:31 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 27 Cold: hi: 62, btch: 15 usd: 49
  1047. Sep 27 02:15:31 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 133 Cold: hi: 62, btch: 15 usd: 29
  1048. Sep 27 02:15:31 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 53 Cold: hi: 62, btch: 15 usd: 34
  1049. Sep 27 02:15:31 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 30 Cold: hi: 62, btch: 15 usd: 49
  1050. Sep 27 02:15:31 monitor kernel: Active:468274 inactive:393191 dirty:0 writeback:0 unstable:0
  1051. Sep 27 02:15:31 monitor kernel: free:29140 slab:4474 mapped:0 pagetables:3917 bounce:0
  1052. Sep 27 02:15:31 monitor kernel: DMA free:10944kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  1053. able? yes
  1054. Sep 27 02:15:31 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  1055. Sep 27 02:15:31 monitor kernel: Normal free:105148kB min:3744kB low:4680kB high:5616kB active:369120kB inactive:342492kB present:894080kB pages_sca
  1056. nned:6933360 all_unreclaimable? yes
  1057. Sep 27 02:15:31 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  1058. Sep 27 02:15:31 monitor kernel: HighMem free:468kB min:512kB low:3916kB high:7324kB active:1503976kB inactive:1230272kB present:3251200kB pages_sca
  1059. nned:12886406 all_unreclaimable? yes
  1060. Sep 27 02:15:31 monitor kernel: lowmem_reserve[]: 0 0 0 0
  1061. Sep 27 02:15:31 monitor kernel: DMA: 80*4kB 72*8kB 62*16kB 45*32kB 23*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10944kB
  1062. Sep 27 02:15:31 monitor kernel: Normal: 13501*4kB 6003*8kB 15*16kB 10*32kB 2*64kB 3*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 105148kB
  1063. Sep 27 02:15:31 monitor kernel: HighMem: 65*4kB 12*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 468kB
  1064. Sep 27 02:15:31 monitor kernel: Swap cache: add 4729371, delete 4729386, find 1542142565/1542235200, race 145+75
  1065. Sep 27 02:15:31 monitor kernel: Free swap = 0kB
  1066. Sep 27 02:15:31 monitor kernel: Total swap = 3911816kB
  1067. Sep 27 02:15:31 monitor kernel: Out of memory: kill process 22724 (pbx_log.pl) score 7863 or a child
  1068. Sep 27 02:15:31 monitor kernel: Killed process 22724 (pbx_log.pl)
  1069. Sep 27 02:15:31 monitor kernel: pbx_log.pl invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  1070. Sep 27 02:15:31 monitor kernel: Pid: 9182, comm: pbx_log.pl Not tainted 2.6.24.3 #3
  1071. Sep 27 02:15:31 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  1072. Sep 27 02:15:31 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  1073. Sep 27 02:15:31 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  1074. Sep 27 02:15:31 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  1075. Sep 27 02:15:31 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  1076. Sep 27 02:15:31 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  1077. Sep 27 02:15:31 monitor kernel: [<c040357f>] inet_stream_connect+0x4f/0x270
  1078. Sep 27 02:15:31 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  1079. Sep 27 02:15:31 monitor kernel: [<c0184390>] d_alloc+0x100/0x180
  1080. Sep 27 02:15:31 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  1081. Sep 27 02:15:31 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  1082. Sep 27 02:15:31 monitor kernel: [<c017fe72>] sys_select+0x182/0x1b0
  1083. Sep 27 02:15:31 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  1084. Sep 27 02:15:31 monitor kernel: [<c043a202>] error_code+0x72/0x78
  1085. Sep 27 02:15:31 monitor kernel: =======================
  1086. Sep 27 02:15:31 monitor kernel: DMA per-cpu:
  1087. Sep 27 02:15:31 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1088. Sep 27 02:15:31 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1089. Sep 27 02:15:31 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1090. Sep 27 02:15:31 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1091. Sep 27 02:15:31 monitor kernel: Normal per-cpu:
  1092. Sep 27 02:15:31 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 160 Cold: hi: 62, btch: 15 usd: 52
  1093. Sep 27 02:15:31 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 171 Cold: hi: 62, btch: 15 usd: 27
  1094. Sep 27 02:15:31 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 123 Cold: hi: 62, btch: 15 usd: 47
  1095. Sep 27 02:15:31 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 69 Cold: hi: 62, btch: 15 usd: 38
  1096. Sep 27 02:15:31 monitor kernel: HighMem per-cpu:
  1097. Sep 27 02:15:31 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 27 Cold: hi: 62, btch: 15 usd: 49
  1098. Sep 27 02:15:31 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 133 Cold: hi: 62, btch: 15 usd: 29
  1099. Sep 27 02:15:31 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 53 Cold: hi: 62, btch: 15 usd: 34
  1100. Sep 27 02:15:31 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 30 Cold: hi: 62, btch: 15 usd: 49
  1101. Sep 27 02:15:31 monitor kernel: Active:677762 inactive:183619 dirty:0 writeback:0 unstable:0
  1102. Sep 27 02:15:31 monitor kernel: free:29115 slab:4516 mapped:0 pagetables:3917 bounce:0
  1103. Sep 27 02:15:31 monitor kernel: DMA free:10944kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  1104. able? yes
  1105. Sep 27 02:15:31 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  1106. Sep 27 02:15:31 monitor kernel: Normal free:105048kB min:3744kB low:4680kB high:5616kB active:289612kB inactive:412288kB present:894080kB pages_sca
  1107. nned:101347479 all_unreclaimable? yes
  1108. Sep 27 02:15:31 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  1109. Sep 27 02:15:31 monitor kernel: HighMem free:468kB min:512kB low:3916kB high:7324kB active:2421436kB inactive:322300kB present:3251200kB pages_scan
  1110. ned:27725595 all_unreclaimable? yes
  1111. Sep 27 02:15:31 monitor kernel: lowmem_reserve[]: 0 0 0 0
  1112. Sep 27 02:15:31 monitor kernel: DMA: 80*4kB 72*8kB 62*16kB 45*32kB 23*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10944kB
  1113. Sep 27 02:15:31 monitor kernel: Normal: 13470*4kB 6006*8kB 15*16kB 10*32kB 2*64kB 3*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 105048kB
  1114. Sep 27 02:15:31 monitor kernel: HighMem: 65*4kB 12*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 468kB
  1115. Sep 27 02:15:31 monitor kernel: Swap cache: add 4729371, delete 4729386, find 1542142565/1542235200, race 145+75
  1116. Sep 27 02:15:31 monitor kernel: Free swap = 0kB
  1117. Sep 27 02:15:31 monitor kernel: Total swap = 3911816kB
  1118. Sep 27 02:15:31 monitor kernel: Out of memory: kill process 9093 (pbx_log.pl) score 7526 or a child
  1119. Sep 27 02:15:31 monitor kernel: Killed process 9093 (pbx_log.pl)
  1120. Sep 27 02:15:31 monitor kernel: pbx_log.pl invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  1121. Sep 27 02:15:31 monitor kernel: Pid: 9364, comm: pbx_log.pl Not tainted 2.6.24.3 #3
  1122. Sep 27 02:15:31 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  1123. Sep 27 02:15:31 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  1124. Sep 27 02:15:31 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  1125. Sep 27 02:15:31 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  1126. Sep 27 02:15:31 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  1127. Sep 27 02:15:31 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  1128. Sep 27 02:15:31 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  1129. Sep 27 02:15:31 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  1130. Sep 27 02:15:31 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  1131. Sep 27 02:15:31 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  1132. Sep 27 02:15:31 monitor kernel: [<c043a202>] error_code+0x72/0x78
  1133. Sep 27 02:15:31 monitor kernel: [<c0430000>] rpcb_getport_sync+0x0/0x190
  1134. Sep 27 02:15:31 monitor kernel: =======================
  1135. Sep 27 02:15:31 monitor kernel: DMA per-cpu:
  1136. Sep 27 02:15:31 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1137. Sep 27 02:15:31 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1138. Sep 27 02:15:31 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1139. Sep 27 02:15:31 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1140. Sep 27 02:15:31 monitor kernel: Normal per-cpu:
  1141. Sep 27 02:15:31 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 160 Cold: hi: 62, btch: 15 usd: 52
  1142. Sep 27 02:15:31 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 171 Cold: hi: 62, btch: 15 usd: 27
  1143. Sep 27 02:15:31 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 123 Cold: hi: 62, btch: 15 usd: 47
  1144. Sep 27 02:15:31 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 69 Cold: hi: 62, btch: 15 usd: 38
  1145. Sep 27 02:15:31 monitor kernel: HighMem per-cpu:
  1146. Sep 27 02:15:31 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 27 Cold: hi: 62, btch: 15 usd: 49
  1147. Sep 27 02:15:31 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 133 Cold: hi: 62, btch: 15 usd: 29
  1148. Sep 27 02:15:31 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 53 Cold: hi: 62, btch: 15 usd: 25
  1149. Sep 27 02:15:31 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 30 Cold: hi: 62, btch: 15 usd: 49
  1150. Sep 27 02:15:31 monitor kernel: Active:502583 inactive:360064 dirty:0 writeback:0 unstable:0
  1151. Sep 27 02:15:31 monitor kernel: free:29115 slab:4516 mapped:0 pagetables:3917 bounce:0
  1152. Sep 27 02:15:31 monitor kernel: DMA free:10944kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  1153. able? yes
  1154. Sep 27 02:15:31 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  1155. Sep 27 02:15:31 monitor kernel: Normal free:105048kB min:3744kB low:4680kB high:5616kB active:388712kB inactive:318640kB present:894080kB pages_sca
  1156. nned:103559078 all_unreclaimable? yes
  1157. Sep 27 02:15:31 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  1158. Sep 27 02:15:31 monitor kernel: HighMem free:468kB min:512kB low:3916kB high:7324kB active:1621768kB inactive:1121728kB present:3251200kB pages_sca
  1159. nned:28902437 all_unreclaimable? yes
  1160. Sep 27 02:15:31 monitor kernel: lowmem_reserve[]: 0 0 0 0
  1161. Sep 27 02:15:31 monitor kernel: DMA: 80*4kB 72*8kB 62*16kB 45*32kB 23*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10944kB
  1162. Sep 27 02:15:31 monitor kernel: Normal: 13470*4kB 6006*8kB 15*16kB 10*32kB 2*64kB 3*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 105048kB
  1163. Sep 27 02:15:31 monitor kernel: HighMem: 65*4kB 12*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 468kB
  1164. Sep 27 02:15:31 monitor kernel: Swap cache: add 4729371, delete 4729386, find 1542142565/1542235200, race 145+75
  1165. Sep 27 02:15:31 monitor kernel: Free swap = 0kB
  1166. Sep 27 02:15:31 monitor kernel: Total swap = 3911816kB
  1167. Sep 27 02:15:31 monitor kernel: Out of memory: kill process 9094 (pbx_log.pl) score 7526 or a child
  1168. Sep 27 02:15:31 monitor kernel: Killed process 9094 (pbx_log.pl)
  1169. Sep 27 02:51:35 monitor kernel: init invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  1170. Sep 27 02:51:35 monitor kernel: Pid: 1, comm: init Not tainted 2.6.24.3 #3
  1171. Sep 27 02:51:35 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  1172. Sep 27 02:51:35 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  1173. Sep 27 02:51:35 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  1174. Sep 27 02:51:35 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  1175. Sep 27 02:51:35 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  1176. Sep 27 02:51:35 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  1177. Sep 27 02:51:35 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  1178. Sep 27 02:51:35 monitor kernel: [<c01758a8>] cp_new_stat64+0xf8/0x110
  1179. Sep 27 02:51:35 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  1180. Sep 27 02:51:35 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  1181. Sep 27 02:51:35 monitor kernel: [<c017fe72>] sys_select+0x182/0x1b0
  1182. Sep 27 02:51:35 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  1183. Sep 27 02:51:35 monitor kernel: [<c043a202>] error_code+0x72/0x78
  1184. Sep 27 02:51:35 monitor kernel: =======================
  1185. Sep 27 02:51:35 monitor kernel: DMA per-cpu:
  1186. Sep 27 02:51:35 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1187. Sep 27 02:51:35 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1188. Sep 27 02:51:35 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1189. Sep 27 02:51:35 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1190. Sep 27 02:51:35 monitor kernel: Normal per-cpu:
  1191. Sep 27 02:51:35 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 152 Cold: hi: 62, btch: 15 usd: 25
  1192. Sep 27 02:51:35 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 153 Cold: hi: 62, btch: 15 usd: 55
  1193. Sep 27 02:51:35 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 78 Cold: hi: 62, btch: 15 usd: 48
  1194. Sep 27 02:51:35 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 153 Cold: hi: 62, btch: 15 usd: 53
  1195. Sep 27 02:51:35 monitor kernel: HighMem per-cpu:
  1196. Sep 27 02:51:35 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 118 Cold: hi: 62, btch: 15 usd: 29
  1197. Sep 27 02:51:35 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 49 Cold: hi: 62, btch: 15 usd: 60
  1198. Sep 27 02:51:35 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 55 Cold: hi: 62, btch: 15 usd: 45
  1199. Sep 27 02:51:35 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 27 Cold: hi: 62, btch: 15 usd: 60
  1200. Sep 27 02:51:35 monitor kernel: Active:427228 inactive:433978 dirty:0 writeback:0 unstable:0
  1201. Sep 27 02:51:35 monitor kernel: free:29189 slab:4733 mapped:20 pagetables:3963 bounce:0
  1202. Sep 27 02:51:35 monitor kernel: DMA free:10944kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  1203. able? yes
  1204. Sep 27 02:51:35 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  1205. Sep 27 02:51:35 monitor kernel: Normal free:105320kB min:3744kB low:4680kB high:5616kB active:347836kB inactive:354644kB present:894080kB pages_sca
  1206. nned:5731795 all_unreclaimable? yes
  1207. Sep 27 02:51:35 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  1208. Sep 27 02:51:35 monitor kernel: HighMem free:492kB min:512kB low:3916kB high:7324kB active:1360928kB inactive:1381380kB present:3251200kB pages_sca
  1209. nned:7918489 all_unreclaimable? yes
  1210. Sep 27 02:51:35 monitor kernel: lowmem_reserve[]: 0 0 0 0
  1211. Sep 27 02:51:35 monitor kernel: DMA: 80*4kB 72*8kB 62*16kB 45*32kB 23*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10944kB
  1212. Sep 27 02:51:35 monitor kernel: Normal: 2890*4kB 4266*8kB 909*16kB 225*32kB 132*64kB 88*128kB 51*256kB 6*512kB 0*1024kB 1*2048kB 0*4096kB = 105320k
  1213. B
  1214. Sep 27 02:51:35 monitor kernel: HighMem: 101*4kB 11*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 492kB
  1215. Sep 27 02:51:35 monitor kernel: Swap cache: add 5075914, delete 5075929, find 1542224890/1542328538, race 145+78
  1216. Sep 27 02:51:35 monitor kernel: Free swap = 0kB
  1217. Sep 27 02:51:35 monitor kernel: Total swap = 3911816kB
  1218. Sep 27 02:51:35 monitor kernel: Out of memory: kill process 32177 (pbx_log.pl) score 11988 or a child
  1219. Sep 27 02:51:35 monitor kernel: Killed process 32177 (pbx_log.pl)
  1220. Sep 27 02:51:35 monitor kernel: init invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  1221. Sep 27 02:51:35 monitor kernel: Pid: 1, comm: init Not tainted 2.6.24.3 #3
  1222. Sep 27 02:51:35 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  1223. Sep 27 02:51:35 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  1224. Sep 27 02:51:35 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  1225. Sep 27 02:51:35 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  1226. Sep 27 02:51:35 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  1227. Sep 27 02:51:35 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  1228. Sep 27 02:51:35 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  1229. Sep 27 02:51:35 monitor kernel: [<c01758a8>] cp_new_stat64+0xf8/0x110
  1230. Sep 27 02:51:35 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  1231. Sep 27 02:51:35 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  1232. Sep 27 02:51:35 monitor kernel: [<c017fe72>] sys_select+0x182/0x1b0
  1233. Sep 27 02:51:35 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  1234. Sep 27 02:51:35 monitor kernel: [<c043a202>] error_code+0x72/0x78
  1235. Sep 27 02:51:35 monitor kernel: =======================
  1236. Sep 27 02:51:35 monitor kernel: DMA per-cpu:
  1237. Sep 27 02:51:35 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1238. Sep 27 02:51:35 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1239. Sep 27 02:51:35 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1240. Sep 27 02:51:35 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1241. Sep 27 02:51:35 monitor kernel: Normal per-cpu:
  1242. Sep 27 02:51:35 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 152 Cold: hi: 62, btch: 15 usd: 25
  1243. Sep 27 02:51:35 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 153 Cold: hi: 62, btch: 15 usd: 55
  1244. Sep 27 02:51:35 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 78 Cold: hi: 62, btch: 15 usd: 48
  1245. Sep 27 02:51:35 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 153 Cold: hi: 62, btch: 15 usd: 53
  1246. Sep 27 02:51:35 monitor kernel: HighMem per-cpu:
  1247. Sep 27 02:51:35 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 118 Cold: hi: 62, btch: 15 usd: 29
  1248. Sep 27 02:51:35 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 49 Cold: hi: 62, btch: 15 usd: 60
  1249. Sep 27 02:51:35 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 55 Cold: hi: 62, btch: 15 usd: 45
  1250. Sep 27 02:51:35 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 27 Cold: hi: 62, btch: 15 usd: 60
  1251. Sep 27 02:51:35 monitor kernel: Active:428430 inactive:432802 dirty:0 writeback:0 unstable:0
  1252. Sep 27 02:51:35 monitor kernel: free:29189 slab:4733 mapped:20 pagetables:3963 bounce:0
  1253. Sep 27 02:51:35 monitor kernel: DMA free:10944kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  1254. able? yes
  1255. Sep 27 02:51:35 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  1256. Sep 27 02:51:35 monitor kernel: Normal free:105320kB min:3744kB low:4680kB high:5616kB active:351060kB inactive:351588kB present:894080kB pages_sca
  1257. nned:5818681 all_unreclaimable? yes
  1258. Sep 27 02:51:35 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  1259. Sep 27 02:51:35 monitor kernel: HighMem free:492kB min:512kB low:3916kB high:7324kB active:1362660kB inactive:1379620kB present:3251200kB pages_sca
  1260. nned:7936790 all_unreclaimable? yes
  1261. Sep 27 02:51:35 monitor kernel: lowmem_reserve[]: 0 0 0 0
  1262. Sep 27 02:51:35 monitor kernel: DMA: 80*4kB 72*8kB 62*16kB 45*32kB 23*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10944kB
  1263. Sep 27 02:51:35 monitor kernel: Normal: 2890*4kB 4266*8kB 909*16kB 225*32kB 132*64kB 88*128kB 51*256kB 6*512kB 0*1024kB 1*2048kB 0*4096kB = 105320k
  1264. B
  1265. Sep 27 02:51:35 monitor kernel: HighMem: 101*4kB 11*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 492kB
  1266. Sep 27 02:51:35 monitor kernel: Swap cache: add 5075914, delete 5075929, find 1542224890/1542328538, race 145+78
  1267. Sep 27 02:51:35 monitor kernel: Free swap = 0kB
  1268. Sep 27 02:51:35 monitor kernel: Total swap = 3911816kB
  1269. Sep 27 02:51:37 monitor kernel: Out of memory: kill process 32184 (perl) score 11221 or a child
  1270. Sep 27 02:51:37 monitor kernel: Killed process 32184 (perl)
  1271. Sep 27 02:51:37 monitor kernel: init invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  1272. Sep 27 02:51:37 monitor kernel: Pid: 1, comm: init Not tainted 2.6.24.3 #3
  1273. Sep 27 02:51:37 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  1274. Sep 27 02:51:37 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  1275. Sep 27 02:51:37 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  1276. Sep 27 02:51:37 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  1277. Sep 27 02:51:37 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  1278. Sep 27 02:51:37 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  1279. Sep 27 02:51:37 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  1280. Sep 27 02:51:37 monitor kernel: [<c01758a8>] cp_new_stat64+0xf8/0x110
  1281. Sep 27 02:51:37 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  1282. Sep 27 02:51:37 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  1283. Sep 27 02:51:37 monitor kernel: [<c017fe72>] sys_select+0x182/0x1b0
  1284. Sep 27 02:51:37 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  1285. Sep 27 02:51:37 monitor kernel: [<c043a202>] error_code+0x72/0x78
  1286. Sep 27 02:51:37 monitor kernel: =======================
  1287. Sep 27 02:51:37 monitor kernel: DMA per-cpu:
  1288. Sep 27 02:51:37 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1289. Sep 27 02:51:37 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1290. Sep 27 02:51:37 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1291. Sep 27 02:51:37 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1292. Sep 27 02:51:37 monitor kernel: Normal per-cpu:
  1293. Sep 27 02:51:37 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 152 Cold: hi: 62, btch: 15 usd: 25
  1294. Sep 27 02:51:37 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 153 Cold: hi: 62, btch: 15 usd: 55
  1295. Sep 27 02:51:37 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 78 Cold: hi: 62, btch: 15 usd: 48
  1296. Sep 27 02:51:37 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 153 Cold: hi: 62, btch: 15 usd: 53
  1297. Sep 27 02:51:37 monitor kernel: HighMem per-cpu:
  1298. Sep 27 02:51:37 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 118 Cold: hi: 62, btch: 15 usd: 29
  1299. Sep 27 02:51:37 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 49 Cold: hi: 62, btch: 15 usd: 60
  1300. Sep 27 02:51:37 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 55 Cold: hi: 62, btch: 15 usd: 45
  1301. Sep 27 02:51:37 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 27 Cold: hi: 62, btch: 15 usd: 60
  1302. Sep 27 02:51:37 monitor kernel: Active:428424 inactive:433044 dirty:0 writeback:0 unstable:0
  1303. Sep 27 02:51:37 monitor kernel: free:29189 slab:4733 mapped:20 pagetables:3963 bounce:0
  1304. Sep 27 02:51:37 monitor kernel: DMA free:10944kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  1305. able? yes
  1306. Sep 27 02:51:37 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  1307. Sep 27 02:51:37 monitor kernel: Normal free:105320kB min:3744kB low:4680kB high:5616kB active:350044kB inactive:353276kB present:894080kB pages_sca
  1308. nned:5860338 all_unreclaimable? yes
  1309. Sep 27 02:51:37 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  1310. Sep 27 02:51:37 monitor kernel: HighMem free:492kB min:512kB low:3916kB high:7324kB active:1363652kB inactive:1378900kB present:3251200kB pages_sca
  1311. nned:7960142 all_unreclaimable? yes
  1312. Sep 27 02:51:37 monitor kernel: lowmem_reserve[]: 0 0 0 0
  1313. Sep 27 02:51:37 monitor kernel: DMA: 80*4kB 72*8kB 62*16kB 45*32kB 23*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10944kB
  1314. Sep 27 02:51:37 monitor kernel: Normal: 2890*4kB 4266*8kB 909*16kB 225*32kB 132*64kB 88*128kB 51*256kB 6*512kB 0*1024kB 1*2048kB 0*4096kB = 105320k
  1315. B
  1316. Sep 27 02:51:37 monitor kernel: HighMem: 101*4kB 11*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 492kB
  1317. Sep 27 02:51:37 monitor kernel: Swap cache: add 5075914, delete 5075929, find 1542224890/1542328538, race 145+78
  1318. Sep 27 02:51:37 monitor kernel: Free swap = 0kB
  1319. Sep 27 02:51:37 monitor kernel: Total swap = 3911816kB
  1320. Sep 27 02:51:37 monitor kernel: Out of memory: kill process 32202 (pbx_log.pl) score 10757 or a child
  1321. Sep 27 02:51:37 monitor kernel: Killed process 32202 (pbx_log.pl)
  1322. Sep 27 02:51:37 monitor kernel: pbx_log.pl invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  1323. Sep 27 02:51:37 monitor kernel: Pid: 9243, comm: pbx_log.pl Not tainted 2.6.24.3 #3
  1324. Sep 27 02:51:37 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  1325. Sep 27 02:51:37 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  1326. Sep 27 02:51:37 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  1327. Sep 27 02:51:37 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  1328. Sep 27 02:51:37 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  1329. Sep 27 02:51:37 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  1330. Sep 27 02:51:37 monitor kernel: [<c02b02e0>] e1000_clean_rx_irq+0x0/0x4e0
  1331. Sep 27 02:51:37 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  1332. Sep 27 02:51:37 monitor kernel: [<c0184390>] d_alloc+0x100/0x180
  1333. Sep 27 02:51:37 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  1334. Sep 27 02:51:37 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  1335. Sep 27 02:51:37 monitor kernel: [<c017fe72>] sys_select+0x182/0x1b0
  1336. Sep 27 02:51:37 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  1337. Sep 27 02:51:37 monitor kernel: [<c043a202>] error_code+0x72/0x78
  1338. Sep 27 02:51:37 monitor kernel: =======================
  1339. Sep 27 02:51:37 monitor kernel: DMA per-cpu:
  1340. Sep 27 02:51:37 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1341. Sep 27 02:51:37 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1342. Sep 27 02:51:37 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1343. Sep 27 02:51:37 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1344. Sep 27 02:51:37 monitor kernel: Normal per-cpu:
  1345. Sep 27 02:51:37 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 152 Cold: hi: 62, btch: 15 usd: 25
  1346. Sep 27 02:51:37 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 153 Cold: hi: 62, btch: 15 usd: 55
  1347. Sep 27 02:51:37 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 78 Cold: hi: 62, btch: 15 usd: 48
  1348. Sep 27 02:51:37 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 153 Cold: hi: 62, btch: 15 usd: 53
  1349. Sep 27 02:51:37 monitor kernel: HighMem per-cpu:
  1350. Sep 27 02:51:37 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 118 Cold: hi: 62, btch: 15 usd: 29
  1351. Sep 27 02:51:37 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 49 Cold: hi: 62, btch: 15 usd: 60
  1352. Sep 27 02:51:37 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 55 Cold: hi: 62, btch: 15 usd: 45
  1353. Sep 27 02:51:37 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 27 Cold: hi: 62, btch: 15 usd: 60
  1354. Sep 27 02:51:37 monitor kernel: Active:427312 inactive:433967 dirty:0 writeback:0 unstable:0
  1355. Sep 27 02:51:37 monitor kernel: free:29191 slab:4733 mapped:20 pagetables:3963 bounce:0
  1356. Sep 27 02:51:37 monitor kernel: DMA free:10944kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  1357. able? yes
  1358. Sep 27 02:51:37 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  1359. Sep 27 02:51:37 monitor kernel: Normal free:105328kB min:3744kB low:4680kB high:5616kB active:345236kB inactive:356536kB present:894080kB pages_sca
  1360. nned:1221258 all_unreclaimable? yes
  1361. Sep 27 02:51:37 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  1362. Sep 27 02:51:37 monitor kernel: HighMem free:492kB min:512kB low:3916kB high:7324kB active:1364012kB inactive:1379556kB present:3251200kB pages_sca
  1363. nned:8295485 all_unreclaimable? yes
  1364. Sep 27 02:51:37 monitor kernel: lowmem_reserve[]: 0 0 0 0
  1365. Sep 27 02:51:37 monitor kernel: DMA: 80*4kB 72*8kB 62*16kB 45*32kB 23*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10944kB
  1366. Sep 27 02:51:37 monitor kernel: Normal: 2890*4kB 4269*8kB 909*16kB 225*32kB 132*64kB 88*128kB 51*256kB 6*512kB 0*1024kB 1*2048kB 0*4096kB = 105344k
  1367. B
  1368. Sep 27 02:51:37 monitor kernel: HighMem: 101*4kB 11*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 492kB
  1369. Sep 27 02:51:37 monitor kernel: Swap cache: add 5075914, delete 5075929, find 1542224890/1542328538, race 145+78
  1370. Sep 27 02:51:37 monitor kernel: Free swap = 0kB
  1371. Sep 27 02:51:37 monitor kernel: Total swap = 3911816kB
  1372. Sep 27 02:51:37 monitor kernel: Out of memory: kill process 32183 (pbx_log.pl) score 10610 or a child
  1373. Sep 27 02:51:37 monitor kernel: Killed process 32183 (pbx_log.pl)
  1374. Sep 27 02:51:37 monitor kernel: klogd invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  1375. Sep 27 02:51:37 monitor kernel: Pid: 2186, comm: klogd Not tainted 2.6.24.3 #3
  1376. Sep 27 02:51:37 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  1377. Sep 27 02:51:37 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  1378. Sep 27 02:51:37 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  1379. Sep 27 02:51:37 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  1380. Sep 27 02:51:37 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  1381. Sep 27 02:51:37 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  1382. Sep 27 02:51:37 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  1383. Sep 27 02:51:37 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  1384. Sep 27 02:51:37 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  1385. Sep 27 02:51:37 monitor kernel: [<c0172ed5>] vfs_read+0x115/0x150
  1386. Sep 27 02:51:37 monitor kernel: [<c01a6a00>] proc_reg_read+0x0/0x90
  1387. Sep 27 02:51:37 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  1388. Sep 27 02:51:37 monitor kernel: [<c043a202>] error_code+0x72/0x78
  1389. Sep 27 02:51:37 monitor kernel: =======================
  1390. Sep 27 02:51:37 monitor kernel: DMA per-cpu:
  1391. Sep 27 02:51:37 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1392. Sep 27 02:51:37 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1393. Sep 27 02:51:37 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1394. Sep 27 02:51:37 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1395. Sep 27 02:51:37 monitor kernel: Normal per-cpu:
  1396. Sep 27 02:51:37 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 152 Cold: hi: 62, btch: 15 usd: 25
  1397. Sep 27 02:51:37 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 153 Cold: hi: 62, btch: 15 usd: 55
  1398. Sep 27 02:51:37 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 78 Cold: hi: 62, btch: 15 usd: 48
  1399. Sep 27 02:51:37 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 153 Cold: hi: 62, btch: 15 usd: 53
  1400. Sep 27 02:51:37 monitor kernel: HighMem per-cpu:
  1401. Sep 27 02:51:37 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 118 Cold: hi: 62, btch: 15 usd: 29
  1402. Sep 27 02:51:37 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 49 Cold: hi: 62, btch: 15 usd: 60
  1403. Sep 27 02:51:37 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 55 Cold: hi: 62, btch: 15 usd: 45
  1404. Sep 27 02:51:37 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 27 Cold: hi: 62, btch: 15 usd: 60
  1405. Sep 27 02:51:37 monitor kernel: Active:428388 inactive:432735 dirty:0 writeback:0 unstable:0
  1406. Sep 27 02:51:37 monitor kernel: free:29195 slab:4733 mapped:20 pagetables:3963 bounce:0
  1407. Sep 27 02:51:37 monitor kernel: DMA free:10944kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  1408. able? yes
  1409. Sep 27 02:51:37 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  1410. Sep 27 02:51:37 monitor kernel: Normal free:105344kB min:3744kB low:4680kB high:5616kB active:353100kB inactive:348136kB present:894080kB pages_sca
  1411. nned:1255784 all_unreclaimable? yes
  1412. Sep 27 02:51:37 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  1413. Sep 27 02:51:37 monitor kernel: HighMem free:492kB min:512kB low:3916kB high:7324kB active:1360748kB inactive:1382804kB present:3251200kB pages_sca
  1414. nned:8438889 all_unreclaimable? yes
  1415. Sep 27 02:51:37 monitor kernel: lowmem_reserve[]: 0 0 0 0
  1416. Sep 27 02:51:37 monitor kernel: DMA: 80*4kB 72*8kB 62*16kB 45*32kB 23*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10944kB
  1417. Sep 27 02:51:37 monitor kernel: Normal: 2890*4kB 4270*8kB 909*16kB 225*32kB 132*64kB 88*128kB 51*256kB 6*512kB 0*1024kB 1*2048kB 0*4096kB = 105352k
  1418. B
  1419. Sep 27 02:51:37 monitor kernel: HighMem: 101*4kB 11*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 492kB
  1420. Sep 27 02:51:37 monitor kernel: Swap cache: add 5075914, delete 5075929, find 1542224890/1542328538, race 145+78
  1421. Sep 27 02:51:37 monitor kernel: Free swap = 0kB
  1422. Sep 27 02:51:37 monitor kernel: Total swap = 3911816kB
  1423. Sep 27 02:51:37 monitor kernel: Out of memory: kill process 32203 (pbx_log.pl) score 9945 or a child
  1424. Sep 27 02:51:37 monitor kernel: Killed process 32203 (pbx_log.pl)
  1425. Sep 27 02:51:37 monitor kernel: klogd invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  1426. Sep 27 02:51:37 monitor kernel: Pid: 2186, comm: klogd Not tainted 2.6.24.3 #3
  1427. Sep 27 02:51:37 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  1428. Sep 27 02:51:37 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  1429. Sep 27 02:51:37 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  1430. Sep 27 02:51:37 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  1431. Sep 27 02:51:37 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  1432. Sep 27 02:51:37 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  1433. Sep 27 02:51:37 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  1434. Sep 27 02:51:37 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  1435. Sep 27 02:51:37 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  1436. Sep 27 02:51:37 monitor kernel: [<c0172ed5>] vfs_read+0x115/0x150
  1437. Sep 27 02:51:37 monitor kernel: [<c01a6a00>] proc_reg_read+0x0/0x90
  1438. Sep 27 02:51:37 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  1439. Sep 27 02:51:37 monitor kernel: [<c043a202>] error_code+0x72/0x78
  1440. Sep 27 02:51:37 monitor kernel: =======================
  1441. Sep 27 02:51:37 monitor kernel: DMA per-cpu:
  1442. Sep 27 02:51:37 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1443. Sep 27 02:51:37 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1444. Sep 27 02:51:37 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1445. Sep 27 02:51:37 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1446. Sep 27 02:51:37 monitor kernel: Normal per-cpu:
  1447. Sep 27 02:51:37 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 152 Cold: hi: 62, btch: 15 usd: 25
  1448. Sep 27 02:51:37 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 153 Cold: hi: 62, btch: 15 usd: 55
  1449. Sep 27 02:51:37 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 78 Cold: hi: 62, btch: 15 usd: 48
  1450. Sep 27 02:51:37 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 153 Cold: hi: 62, btch: 15 usd: 53
  1451. Sep 27 02:51:37 monitor kernel: HighMem per-cpu:
  1452. Sep 27 02:51:37 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 118 Cold: hi: 62, btch: 15 usd: 29
  1453. Sep 27 02:51:37 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 49 Cold: hi: 62, btch: 15 usd: 60
  1454. Sep 27 02:51:37 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 55 Cold: hi: 62, btch: 15 usd: 45
  1455. Sep 27 02:51:37 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 27 Cold: hi: 62, btch: 15 usd: 60
  1456. Sep 27 02:51:37 monitor kernel: Active:428336 inactive:432834 dirty:0 writeback:0 unstable:0
  1457. Sep 27 02:51:37 monitor kernel: free:29195 slab:4733 mapped:20 pagetables:3963 bounce:0
  1458. Sep 27 02:51:37 monitor kernel: DMA free:10944kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  1459. able? yes
  1460. Sep 27 02:51:37 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  1461. Sep 27 02:51:37 monitor kernel: Normal free:105344kB min:3744kB low:4680kB high:5616kB active:352492kB inactive:348532kB present:894080kB pages_sca
  1462. nned:1289045 all_unreclaimable? yes
  1463. Sep 27 02:51:37 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  1464. Sep 27 02:51:37 monitor kernel: HighMem free:492kB min:512kB low:3916kB high:7324kB active:1360852kB inactive:1382804kB present:3251200kB pages_sca
  1465. nned:8439357 all_unreclaimable? yes
  1466. Sep 27 02:51:37 monitor kernel: lowmem_reserve[]: 0 0 0 0
  1467. Sep 27 02:51:37 monitor kernel: DMA: 80*4kB 72*8kB 62*16kB 45*32kB 23*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10944kB
  1468. Sep 27 02:51:37 monitor kernel: Normal: 2890*4kB 4270*8kB 909*16kB 225*32kB 132*64kB 88*128kB 51*256kB 6*512kB 0*1024kB 1*2048kB 0*4096kB = 105352k
  1469. B
  1470. Sep 27 02:51:37 monitor kernel: HighMem: 101*4kB 11*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 492kB
  1471. Sep 27 02:51:37 monitor kernel: Swap cache: add 5075914, delete 5075929, find 1542224890/1542328538, race 145+78
  1472. Sep 27 02:51:37 monitor kernel: Free swap = 0kB
  1473. Sep 27 02:51:37 monitor kernel: Total swap = 3911816kB
  1474. Sep 27 02:51:37 monitor kernel: Out of memory: kill process 32176 (pbx_log.pl) score 9247 or a child
  1475. Sep 27 02:51:37 monitor kernel: Killed process 32176 (pbx_log.pl)
  1476. Sep 27 02:51:37 monitor kernel: pbx_log.pl invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  1477. Sep 27 02:51:37 monitor kernel: Pid: 9338, comm: pbx_log.pl Not tainted 2.6.24.3 #3
  1478. Sep 27 02:51:37 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  1479. Sep 27 02:51:37 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  1480. Sep 27 02:51:37 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  1481. Sep 27 02:51:37 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  1482. Sep 27 02:51:37 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  1483. Sep 27 02:51:37 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  1484. Sep 27 02:51:37 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  1485. Sep 27 02:51:37 monitor kernel: [<c022ccef>] blk_run_queue+0x1f/0x70
  1486. Sep 27 02:51:37 monitor kernel: [<c02f62fb>] scsi_end_request+0xab/0xe0
  1487. Sep 27 02:51:37 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  1488. Sep 27 02:51:37 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  1489. Sep 27 02:51:37 monitor kernel: [<c0126af8>] sys_gettimeofday+0x28/0x80
  1490. Sep 27 02:51:37 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  1491. Sep 27 02:51:37 monitor kernel: [<c043a202>] error_code+0x72/0x78
  1492. Sep 27 02:51:37 monitor kernel: [<c0430000>] rpcb_getport_sync+0x0/0x190
  1493. Sep 27 02:51:37 monitor kernel: =======================
  1494. Sep 27 02:51:37 monitor kernel: DMA per-cpu:
  1495. Sep 27 02:51:37 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1496. Sep 27 02:51:37 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1497. Sep 27 02:51:37 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1498. Sep 27 02:51:37 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1499. Sep 27 02:51:37 monitor kernel: Normal per-cpu:
  1500. Sep 27 02:51:37 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 153 Cold: hi: 62, btch: 15 usd: 49
  1501. Sep 27 02:51:37 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 153 Cold: hi: 62, btch: 15 usd: 47
  1502. Sep 27 02:51:37 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 78 Cold: hi: 62, btch: 15 usd: 32
  1503. Sep 27 02:51:37 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 152 Cold: hi: 62, btch: 15 usd: 42
  1504. Sep 27 02:51:37 monitor kernel: HighMem per-cpu:
  1505. Sep 27 02:51:37 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 118 Cold: hi: 62, btch: 15 usd: 29
  1506. Sep 27 02:51:37 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 49 Cold: hi: 62, btch: 15 usd: 60
  1507. Sep 27 02:51:37 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 55 Cold: hi: 62, btch: 15 usd: 45
  1508. Sep 27 02:51:37 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 27 Cold: hi: 62, btch: 15 usd: 60
  1509. Sep 27 02:51:37 monitor kernel: Active:427805 inactive:433079 dirty:0 writeback:0 unstable:0
  1510. Sep 27 02:51:37 monitor kernel: free:29186 slab:4733 mapped:17 pagetables:3963 bounce:0
  1511. Sep 27 02:51:37 monitor kernel: DMA free:10944kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  1512. able? yes
  1513. Sep 27 02:51:37 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  1514. Sep 27 02:51:37 monitor kernel: Normal free:105308kB min:3744kB low:4680kB high:5616kB active:348088kB inactive:351412kB present:894080kB pages_sca
  1515. nned:1437707 all_unreclaimable? yes
  1516. Sep 27 02:51:37 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  1517. Sep 27 02:51:37 monitor kernel: HighMem free:492kB min:512kB low:3916kB high:7324kB active:1363132kB inactive:1380904kB present:3251200kB pages_sca
  1518. nned:8649538 all_unreclaimable? yes
  1519. Sep 27 02:51:37 monitor kernel: lowmem_reserve[]: 0 0 0 0
  1520. Sep 27 02:51:37 monitor kernel: DMA: 80*4kB 72*8kB 62*16kB 45*32kB 23*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10944kB
  1521. Sep 27 02:51:37 monitor kernel: Normal: 2874*4kB 4280*8kB 909*16kB 225*32kB 132*64kB 88*128kB 51*256kB 6*512kB 0*1024kB 1*2048kB 0*4096kB = 105368k
  1522. B
  1523. Sep 27 02:51:37 monitor kernel: HighMem: 101*4kB 11*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 492kB
  1524. Sep 27 02:51:37 monitor kernel: Swap cache: add 5075914, delete 5075929, find 1542224890/1542328538, race 145+78
  1525. Sep 27 02:51:37 monitor kernel: Free swap = 0kB
  1526. Sep 27 02:51:37 monitor kernel: Total swap = 3911816kB
  1527. Sep 27 02:51:37 monitor kernel: Out of memory: kill process 30119 (postgres) score 9151 or a child
  1528. Sep 27 02:51:37 monitor kernel: Killed process 30119 (postgres)
  1529. Sep 27 02:51:37 monitor kernel: nagios invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  1530. Sep 27 02:51:37 monitor kernel: Pid: 11513, comm: nagios Not tainted 2.6.24.3 #3
  1531. Sep 27 02:51:37 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  1532. Sep 27 02:51:37 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  1533. Sep 27 02:51:37 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  1534. Sep 27 02:51:37 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  1535. Sep 27 02:51:37 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  1536. Sep 27 02:51:37 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  1537. Sep 27 02:51:37 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  1538. Sep 27 02:51:37 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  1539. Sep 27 02:51:37 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  1540. Sep 27 02:51:37 monitor kernel: [<c0172e75>] vfs_read+0xb5/0x150
  1541. Sep 27 02:51:37 monitor kernel: [<c0138540>] hrtimer_wakeup+0x0/0x20
  1542. Sep 27 02:51:37 monitor kernel: [<c01724b0>] do_sync_read+0x0/0x110
  1543. Sep 27 02:51:37 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  1544. Sep 27 02:51:37 monitor kernel: [<c043a202>] error_code+0x72/0x78
  1545. Sep 27 02:51:37 monitor kernel: [<c0430000>] rpcb_getport_sync+0x0/0x190
  1546. Sep 27 02:51:37 monitor kernel: =======================
  1547. Sep 27 02:51:37 monitor kernel: DMA per-cpu:
  1548. Sep 27 02:51:37 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1549. Sep 27 02:51:37 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1550. Sep 27 02:51:37 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1551. Sep 27 02:51:37 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1552. Sep 27 02:51:37 monitor kernel: Normal per-cpu:
  1553. Sep 27 02:51:37 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 158 Cold: hi: 62, btch: 15 usd: 56
  1554. Sep 27 02:51:37 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 152 Cold: hi: 62, btch: 15 usd: 60
  1555. Sep 27 02:51:37 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 104 Cold: hi: 62, btch: 15 usd: 57
  1556. Sep 27 02:51:37 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 155 Cold: hi: 62, btch: 15 usd: 49
  1557. Sep 27 02:51:37 monitor kernel: HighMem per-cpu:
  1558. Sep 27 02:51:37 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 119 Cold: hi: 62, btch: 15 usd: 29
  1559. Sep 27 02:51:37 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 49 Cold: hi: 62, btch: 15 usd: 60
  1560. Sep 27 02:51:37 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 78 Cold: hi: 62, btch: 15 usd: 45
  1561. Sep 27 02:51:37 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 27 Cold: hi: 62, btch: 15 usd: 60
  1562. Sep 27 02:51:37 monitor kernel: Active:430737 inactive:429993 dirty:0 writeback:0 unstable:0
  1563. Sep 27 02:51:37 monitor kernel: free:29165 slab:4734 mapped:24 pagetables:3952 bounce:0
  1564. Sep 27 02:51:37 monitor kernel: DMA free:10944kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  1565. able? yes
  1566. Sep 27 02:51:37 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  1567. Sep 27 02:51:37 monitor kernel: Normal free:105224kB min:3744kB low:4680kB high:5616kB active:349364kB inactive:349332kB present:894080kB pages_sca
  1568. nned:4214304 all_unreclaimable? yes
  1569. Sep 27 02:51:37 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  1570. Sep 27 02:51:37 monitor kernel: HighMem free:492kB min:512kB low:3916kB high:7324kB active:1373584kB inactive:1370640kB present:3251200kB pages_sca
  1571. nned:8690038 all_unreclaimable? yes
  1572. Sep 27 02:51:37 monitor kernel: lowmem_reserve[]: 0 0 0 0
  1573. Sep 27 02:51:37 monitor kernel: DMA: 80*4kB 72*8kB 62*16kB 45*32kB 23*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10944kB
  1574. Sep 27 02:51:37 monitor kernel: Normal: 2952*4kB 4241*8kB 894*16kB 222*32kB 133*64kB 89*128kB 51*256kB 6*512kB 0*1024kB 1*2048kB 0*4096kB = 105224k
  1575. B
  1576. Sep 27 02:51:37 monitor kernel: HighMem: 101*4kB 11*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 492kB
  1577. Sep 27 02:51:37 monitor kernel: Swap cache: add 5075914, delete 5075929, find 1542224890/1542328538, race 145+78
  1578. Sep 27 02:51:37 monitor kernel: Free swap = 0kB
  1579. Sep 27 02:51:37 monitor kernel: Total swap = 3911816kB
  1580. Sep 27 02:51:37 monitor kernel: Out of memory: kill process 9134 (pbx_log.pl) score 8972 or a child
  1581. Sep 27 02:51:37 monitor kernel: Killed process 9134 (pbx_log.pl)
  1582. Sep 27 03:30:31 monitor kernel: pbx_log.pl invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  1583. Sep 27 03:30:32 monitor kernel: Pid: 9177, comm: pbx_log.pl Not tainted 2.6.24.3 #3
  1584. Sep 27 03:30:32 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  1585. Sep 27 03:30:32 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  1586. Sep 27 03:30:32 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  1587. Sep 27 03:30:32 monitor kernel: [<c04386ed>] schedule+0x1dd/0x690
  1588. Sep 27 03:30:32 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  1589. Sep 27 03:30:32 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  1590. Sep 27 03:30:32 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  1591. Sep 27 03:30:32 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  1592. Sep 27 03:30:32 monitor kernel: [<c012c4bb>] recalc_sigpending+0xb/0x30
  1593. Sep 27 03:30:32 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  1594. Sep 27 03:30:32 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  1595. Sep 27 03:30:32 monitor kernel: [<c012c4bb>] recalc_sigpending+0xb/0x30
  1596. Sep 27 03:30:32 monitor kernel: [<c012c545>] sigprocmask+0x65/0xf0
  1597. Sep 27 03:30:32 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  1598. Sep 27 03:30:32 monitor kernel: [<c043a202>] error_code+0x72/0x78
  1599. Sep 27 03:30:32 monitor kernel: [<c0430000>] rpcb_getport_sync+0x0/0x190
  1600. Sep 27 03:30:32 monitor kernel: =======================
  1601. Sep 27 03:30:32 monitor kernel: DMA per-cpu:
  1602. Sep 27 03:30:32 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1603. Sep 27 03:30:32 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1604. Sep 27 03:30:32 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1605. Sep 27 03:30:32 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1606. Sep 27 03:30:32 monitor kernel: Normal per-cpu:
  1607. Sep 27 03:30:32 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 120 Cold: hi: 62, btch: 15 usd: 48
  1608. Sep 27 03:30:32 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 82 Cold: hi: 62, btch: 15 usd: 51
  1609. Sep 27 03:30:32 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 85 Cold: hi: 62, btch: 15 usd: 54
  1610. Sep 27 03:30:32 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 179 Cold: hi: 62, btch: 15 usd: 56
  1611. Sep 27 03:30:32 monitor kernel: HighMem per-cpu:
  1612. Sep 27 03:30:32 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 44 Cold: hi: 62, btch: 15 usd: 54
  1613. Sep 27 03:30:32 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 28 Cold: hi: 62, btch: 15 usd: 54
  1614. Sep 27 03:30:32 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 73 Cold: hi: 62, btch: 15 usd: 37
  1615. Sep 27 03:30:32 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 80 Cold: hi: 62, btch: 15 usd: 56
  1616. Sep 27 03:30:32 monitor kernel: Active:432170 inactive:428893 dirty:0 writeback:0 unstable:0
  1617. Sep 27 03:30:32 monitor kernel: free:29107 slab:4620 mapped:8 pagetables:4104 bounce:0
  1618. Sep 27 03:30:32 monitor kernel: DMA free:10944kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  1619. able? yes
  1620. Sep 27 03:30:32 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  1621. Sep 27 03:30:32 monitor kernel: Normal free:105100kB min:3744kB low:4680kB high:5616kB active:358108kB inactive:352412kB present:894080kB pages_sca
  1622. nned:1152405 all_unreclaimable? yes
  1623. Sep 27 03:30:32 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  1624. Sep 27 03:30:32 monitor kernel: HighMem free:384kB min:512kB low:3916kB high:7324kB active:1370572kB inactive:1363160kB present:3251200kB pages_sca
  1625. nned:12083977 all_unreclaimable? yes
  1626. Sep 27 03:30:32 monitor kernel: lowmem_reserve[]: 0 0 0 0
  1627. Sep 27 03:30:32 monitor kernel: DMA: 80*4kB 72*8kB 62*16kB 45*32kB 23*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10944kB
  1628. Sep 27 03:30:32 monitor kernel: Normal: 2975*4kB 4120*8kB 1015*16kB 237*32kB 123*64kB 83*128kB 44*256kB 9*512kB 0*1024kB 1*2048kB 0*4096kB = 105100
  1629. kB
  1630. Sep 27 03:30:32 monitor kernel: HighMem: 28*4kB 14*8kB 0*16kB 3*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 384kB
  1631. Sep 27 03:30:32 monitor kernel: Swap cache: add 5540783, delete 5540797, find 1542326763/1542451673, race 145+83
  1632. Sep 27 03:30:32 monitor kernel: Free swap = 0kB
  1633. Sep 27 03:30:32 monitor kernel: Total swap = 3911816kB
  1634. Sep 27 03:30:32 monitor kernel: Out of memory: kill process 13430 (pbx_log.pl) score 17502 or a child
  1635. Sep 27 03:30:32 monitor kernel: Killed process 13430 (pbx_log.pl)
  1636. Sep 27 03:30:32 monitor kernel: nagios invoked oom-killer: gfp_mask=0x1200d2, order=0, oomkilladj=0
  1637. Sep 27 03:30:32 monitor kernel: Pid: 14033, comm: nagios Not tainted 2.6.24.3 #3
  1638. Sep 27 03:30:32 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  1639. Sep 27 03:30:32 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  1640. Sep 27 03:30:32 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  1641. Sep 27 03:30:32 monitor kernel: [<c015cbe9>] do_wp_page+0x89/0x430
  1642. Sep 27 03:30:32 monitor kernel: [<c015eac5>] handle_mm_fault+0x435/0x5b0
  1643. Sep 27 03:30:32 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  1644. Sep 27 03:30:32 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  1645. Sep 27 03:30:32 monitor kernel: [<c043a202>] error_code+0x72/0x78
  1646. Sep 27 03:30:32 monitor kernel: [<c0430000>] rpcb_getport_sync+0x0/0x190
  1647. Sep 27 03:30:32 monitor kernel: =======================
  1648. Sep 27 03:30:32 monitor kernel: DMA per-cpu:
  1649. Sep 27 03:30:32 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1650. Sep 27 03:30:32 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1651. Sep 27 03:30:32 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1652. Sep 27 03:30:32 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1653. Sep 27 03:30:32 monitor kernel: Normal per-cpu:
  1654. Sep 27 03:30:32 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 120 Cold: hi: 62, btch: 15 usd: 48
  1655. Sep 27 03:30:32 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 82 Cold: hi: 62, btch: 15 usd: 51
  1656. Sep 27 03:30:32 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 86 Cold: hi: 62, btch: 15 usd: 56
  1657. Sep 27 03:30:32 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 179 Cold: hi: 62, btch: 15 usd: 57
  1658. Sep 27 03:30:32 monitor kernel: HighMem per-cpu:
  1659. Sep 27 03:30:32 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 44 Cold: hi: 62, btch: 15 usd: 54
  1660. Sep 27 03:30:32 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 28 Cold: hi: 62, btch: 15 usd: 54
  1661. Sep 27 03:30:32 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 73 Cold: hi: 62, btch: 15 usd: 37
  1662. Sep 27 03:30:32 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 80 Cold: hi: 62, btch: 15 usd: 56
  1663. Sep 27 03:30:32 monitor kernel: Active:437164 inactive:423908 dirty:0 writeback:0 unstable:0
  1664. Sep 27 03:30:32 monitor kernel: free:29109 slab:4620 mapped:8 pagetables:4104 bounce:0
  1665. Sep 27 03:30:32 monitor kernel: DMA free:10944kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  1666. able? yes
  1667. Sep 27 03:30:32 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  1668. Sep 27 03:30:32 monitor kernel: Normal free:105108kB min:3744kB low:4680kB high:5616kB active:347736kB inactive:358340kB present:894080kB pages_sca
  1669. nned:4862433 all_unreclaimable? yes
  1670. Sep 27 03:30:32 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  1671. Sep 27 03:30:32 monitor kernel: HighMem free:384kB min:512kB low:3916kB high:7324kB active:1400772kB inactive:1337292kB present:3251200kB pages_sca
  1672. nned:18203836 all_unreclaimable? yes
  1673. Sep 27 03:30:32 monitor kernel: lowmem_reserve[]: 0 0 0 0
  1674. Sep 27 03:30:32 monitor kernel: DMA: 80*4kB 72*8kB 62*16kB 45*32kB 23*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10944kB
  1675. Sep 27 03:30:32 monitor kernel: Normal: 2975*4kB 4121*8kB 1015*16kB 237*32kB 123*64kB 83*128kB 44*256kB 9*512kB 0*1024kB 1*2048kB 0*4096kB = 105108
  1676. kB
  1677. Sep 27 03:30:32 monitor kernel: HighMem: 28*4kB 14*8kB 0*16kB 3*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 384kB
  1678. Sep 27 03:30:32 monitor kernel: Swap cache: add 5540783, delete 5540797, find 1542326763/1542451673, race 145+83
  1679. Sep 27 03:30:32 monitor kernel: Free swap = 0kB
  1680. Sep 27 03:30:32 monitor kernel: Total swap = 3911816kB
  1681. Sep 27 03:30:32 monitor kernel: Out of memory: kill process 13842 (pbx_log.pl) score 17295 or a child
  1682. Sep 27 03:30:32 monitor kernel: Killed process 13842 (pbx_log.pl)
  1683. Sep 27 03:30:32 monitor kernel: postgres invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  1684. Sep 27 03:30:32 monitor kernel: Pid: 810, comm: postgres Not tainted 2.6.24.3 #3
  1685. Sep 27 03:30:32 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  1686. Sep 27 03:30:32 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  1687. Sep 27 03:30:32 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  1688. Sep 27 03:30:32 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  1689. Sep 27 03:30:32 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  1690. Sep 27 03:30:32 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  1691. Sep 27 03:30:32 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  1692. Sep 27 03:30:32 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  1693. Sep 27 03:30:32 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  1694. Sep 27 03:30:32 monitor kernel: [<c017fe72>] sys_select+0x182/0x1b0
  1695. Sep 27 03:30:32 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  1696. Sep 27 03:30:32 monitor kernel: [<c043a202>] error_code+0x72/0x78
  1697. Sep 27 03:30:32 monitor kernel: [<c0430000>] rpcb_getport_sync+0x0/0x190
  1698. Sep 27 03:30:32 monitor kernel: =======================
  1699. Sep 27 03:30:32 monitor kernel: DMA per-cpu:
  1700. Sep 27 03:30:32 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1701. Sep 27 03:30:32 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1702. Sep 27 03:30:32 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1703. Sep 27 03:30:32 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1704. Sep 27 03:30:32 monitor kernel: Normal per-cpu:
  1705. Sep 27 03:30:32 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 120 Cold: hi: 62, btch: 15 usd: 48
  1706. Sep 27 03:30:32 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 82 Cold: hi: 62, btch: 15 usd: 51
  1707. Sep 27 03:30:32 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 86 Cold: hi: 62, btch: 15 usd: 56
  1708. Sep 27 03:30:32 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 179 Cold: hi: 62, btch: 15 usd: 57
  1709. Sep 27 03:30:32 monitor kernel: HighMem per-cpu:
  1710. Sep 27 03:30:32 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 44 Cold: hi: 62, btch: 15 usd: 54
  1711. Sep 27 03:30:32 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 28 Cold: hi: 62, btch: 15 usd: 54
  1712. Sep 27 03:30:32 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 73 Cold: hi: 62, btch: 15 usd: 37
  1713. Sep 27 03:30:32 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 80 Cold: hi: 62, btch: 15 usd: 56
  1714. Sep 27 03:30:32 monitor kernel: Active:437693 inactive:423280 dirty:0 writeback:0 unstable:0
  1715. Sep 27 03:30:32 monitor kernel: free:29111 slab:4620 mapped:8 pagetables:4104 bounce:0
  1716. Sep 27 03:30:32 monitor kernel: DMA free:10944kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  1717. able? yes
  1718. Sep 27 03:30:32 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  1719. Sep 27 03:30:32 monitor kernel: Normal free:105116kB min:3744kB low:4680kB high:5616kB active:341572kB inactive:361964kB present:894080kB pages_sca
  1720. nned:1546563 all_unreclaimable? yes
  1721. Sep 27 03:30:32 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  1722. Sep 27 03:30:32 monitor kernel: HighMem free:384kB min:512kB low:3916kB high:7324kB active:1409200kB inactive:1331044kB present:3251200kB pages_sca
  1723. nned:19303762 all_unreclaimable? yes
  1724. Sep 27 03:30:32 monitor kernel: lowmem_reserve[]: 0 0 0 0
  1725. Sep 27 03:30:32 monitor kernel: DMA: 80*4kB 72*8kB 62*16kB 45*32kB 23*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10944kB
  1726. Sep 27 03:30:32 monitor kernel: Normal: 2975*4kB 4122*8kB 1015*16kB 237*32kB 123*64kB 83*128kB 44*256kB 9*512kB 0*1024kB 1*2048kB 0*4096kB = 105116
  1727. kB
  1728. Sep 27 03:30:32 monitor kernel: HighMem: 28*4kB 14*8kB 0*16kB 3*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 384kB
  1729. Sep 27 03:30:32 monitor kernel: Swap cache: add 5540783, delete 5540797, find 1542326763/1542451673, race 145+83
  1730. Sep 27 03:30:32 monitor kernel: Free swap = 0kB
  1731. Sep 27 03:30:32 monitor kernel: Total swap = 3911816kB
  1732. Sep 27 03:30:32 monitor kernel: Out of memory: kill process 13301 (pbx_log.pl) score 15047 or a child
  1733. Sep 27 03:30:32 monitor kernel: Killed process 13301 (pbx_log.pl)
  1734. Sep 27 03:30:32 monitor kernel: postgres invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  1735. Sep 27 03:30:32 monitor kernel: Pid: 810, comm: postgres Not tainted 2.6.24.3 #3
  1736. Sep 27 03:30:32 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  1737. Sep 27 03:30:32 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  1738. Sep 27 03:30:32 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  1739. Sep 27 03:30:32 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  1740. Sep 27 03:30:32 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  1741. Sep 27 03:30:32 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  1742. Sep 27 03:30:32 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  1743. Sep 27 03:30:32 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  1744. Sep 27 03:30:32 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  1745. Sep 27 03:30:32 monitor kernel: [<c017fe72>] sys_select+0x182/0x1b0
  1746. Sep 27 03:30:32 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  1747. Sep 27 03:30:32 monitor kernel: [<c043a202>] error_code+0x72/0x78
  1748. Sep 27 03:30:32 monitor kernel: [<c0430000>] rpcb_getport_sync+0x0/0x190
  1749. Sep 27 03:30:32 monitor kernel: =======================
  1750. Sep 27 03:30:32 monitor kernel: DMA per-cpu:
  1751. Sep 27 03:30:32 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1752. Sep 27 03:30:32 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1753. Sep 27 03:30:32 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1754. Sep 27 03:30:32 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1755. Sep 27 03:30:32 monitor kernel: Normal per-cpu:
  1756. Sep 27 03:30:32 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 120 Cold: hi: 62, btch: 15 usd: 48
  1757. Sep 27 03:30:32 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 82 Cold: hi: 62, btch: 15 usd: 51
  1758. Sep 27 03:30:32 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 86 Cold: hi: 62, btch: 15 usd: 56
  1759. Sep 27 03:30:32 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 179 Cold: hi: 62, btch: 15 usd: 57
  1760. Sep 27 03:30:32 monitor kernel: HighMem per-cpu:
  1761. Sep 27 03:30:32 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 44 Cold: hi: 62, btch: 15 usd: 54
  1762. Sep 27 03:30:32 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 28 Cold: hi: 62, btch: 15 usd: 54
  1763. Sep 27 03:30:32 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 73 Cold: hi: 62, btch: 15 usd: 37
  1764. Sep 27 03:30:32 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 80 Cold: hi: 62, btch: 15 usd: 56
  1765. Sep 27 03:30:32 monitor kernel: Active:437709 inactive:423296 dirty:0 writeback:0 unstable:0
  1766. Sep 27 03:30:32 monitor kernel: free:29111 slab:4620 mapped:8 pagetables:4104 bounce:0
  1767. Sep 27 03:30:32 monitor kernel: DMA free:10944kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  1768. able? yes
  1769. Sep 27 03:30:32 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  1770. Sep 27 03:30:32 monitor kernel: Normal free:105116kB min:3744kB low:4680kB high:5616kB active:341592kB inactive:361964kB present:894080kB pages_sca
  1771. nned:1588013 all_unreclaimable? yes
  1772. Sep 27 03:30:32 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  1773. Sep 27 03:30:32 monitor kernel: HighMem free:384kB min:512kB low:3916kB high:7324kB active:1409244kB inactive:1331220kB present:3251200kB pages_sca
  1774. nned:19333897 all_unreclaimable? yes
  1775. Sep 27 03:30:32 monitor kernel: lowmem_reserve[]: 0 0 0 0
  1776. Sep 27 03:30:32 monitor kernel: DMA: 80*4kB 72*8kB 62*16kB 45*32kB 23*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10944kB
  1777. Sep 27 03:30:32 monitor kernel: Normal: 2975*4kB 4122*8kB 1015*16kB 237*32kB 123*64kB 83*128kB 44*256kB 9*512kB 0*1024kB 1*2048kB 0*4096kB = 105116
  1778. kB
  1779. Sep 27 03:30:32 monitor kernel: HighMem: 28*4kB 14*8kB 0*16kB 3*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 384kB
  1780. Sep 27 03:30:32 monitor kernel: Swap cache: add 5540783, delete 5540797, find 1542326763/1542451673, race 145+83
  1781. Sep 27 03:30:32 monitor kernel: Free swap = 0kB
  1782. Sep 27 03:30:32 monitor kernel: Total swap = 3911816kB
  1783. Sep 27 03:30:32 monitor kernel: Out of memory: kill process 13239 (pbx_log.pl) score 14452 or a child
  1784. Sep 27 03:30:32 monitor kernel: Killed process 13239 (pbx_log.pl)
  1785. Sep 27 03:30:32 monitor kernel: nagios invoked oom-killer: gfp_mask=0x1200d2, order=0, oomkilladj=0
  1786. Sep 27 03:30:32 monitor kernel: Pid: 13999, comm: nagios Not tainted 2.6.24.3 #3
  1787. Sep 27 03:30:32 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  1788. Sep 27 03:30:32 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  1789. Sep 27 03:30:32 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  1790. Sep 27 03:30:32 monitor kernel: [<c015cbe9>] do_wp_page+0x89/0x430
  1791. Sep 27 03:30:32 monitor kernel: [<c011a2f2>] update_curr+0x72/0x110
  1792. Sep 27 03:30:32 monitor kernel: [<c015eac5>] handle_mm_fault+0x435/0x5b0
  1793. Sep 27 03:30:32 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  1794. Sep 27 03:30:32 monitor kernel: [<c01795bf>] do_pipe+0x5f/0xc0
  1795. Sep 27 03:30:32 monitor kernel: [<c0102266>] sys_clone+0x36/0x40
  1796. Sep 27 03:30:32 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  1797. Sep 27 03:30:32 monitor kernel: [<c043a202>] error_code+0x72/0x78
  1798. Sep 27 03:30:32 monitor kernel: =======================
  1799. Sep 27 03:30:32 monitor kernel: DMA per-cpu:
  1800. Sep 27 03:30:32 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1801. Sep 27 03:30:32 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1802. Sep 27 03:30:32 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1803. Sep 27 03:30:32 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1804. Sep 27 03:30:32 monitor kernel: Normal per-cpu:
  1805. Sep 27 03:30:32 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 120 Cold: hi: 62, btch: 15 usd: 48
  1806. Sep 27 03:30:32 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 82 Cold: hi: 62, btch: 15 usd: 51
  1807. Sep 27 03:30:32 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 86 Cold: hi: 62, btch: 15 usd: 56
  1808. Sep 27 03:30:32 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 179 Cold: hi: 62, btch: 15 usd: 57
  1809. Sep 27 03:30:32 monitor kernel: HighMem per-cpu:
  1810. Sep 27 03:30:32 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 44 Cold: hi: 62, btch: 15 usd: 54
  1811. Sep 27 03:30:32 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 28 Cold: hi: 62, btch: 15 usd: 54
  1812. Sep 27 03:30:32 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 73 Cold: hi: 62, btch: 15 usd: 37
  1813. Sep 27 03:30:32 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 80 Cold: hi: 62, btch: 15 usd: 56
  1814. Sep 27 03:30:32 monitor kernel: Active:442241 inactive:418983 dirty:0 writeback:0 unstable:0
  1815. Sep 27 03:30:32 monitor kernel: free:29111 slab:4620 mapped:8 pagetables:4104 bounce:0
  1816. Sep 27 03:30:32 monitor kernel: DMA free:10944kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  1817. able? yes
  1818. Sep 27 03:30:32 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  1819. Sep 27 03:30:32 monitor kernel: Normal free:105116kB min:3744kB low:4680kB high:5616kB active:358168kB inactive:345548kB present:894080kB pages_sca
  1820. nned:1161263 all_unreclaimable? yes
  1821. Sep 27 03:30:32 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  1822. Sep 27 03:30:32 monitor kernel: HighMem free:384kB min:512kB low:3916kB high:7324kB active:1410796kB inactive:1330496kB present:3251200kB pages_sca
  1823. nned:19961783 all_unreclaimable? yes
  1824. Sep 27 03:30:32 monitor kernel: lowmem_reserve[]: 0 0 0 0
  1825. Sep 27 03:30:32 monitor kernel: DMA: 80*4kB 72*8kB 62*16kB 45*32kB 23*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10944kB
  1826. Sep 27 03:30:32 monitor kernel: Normal: 2975*4kB 4124*8kB 1015*16kB 237*32kB 123*64kB 83*128kB 44*256kB 9*512kB 0*1024kB 1*2048kB 0*4096kB = 105132
  1827. kB
  1828. Sep 27 03:30:32 monitor kernel: HighMem: 28*4kB 14*8kB 0*16kB 3*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 384kB
  1829. Sep 27 03:30:32 monitor kernel: Swap cache: add 5540783, delete 5540797, find 1542326763/1542451673, race 145+83
  1830. Sep 27 03:30:32 monitor kernel: Free swap = 0kB
  1831. Sep 27 03:30:32 monitor kernel: Total swap = 3911816kB
  1832. Sep 27 03:30:32 monitor kernel: Out of memory: kill process 13305 (pbx_log.pl) score 14239 or a child
  1833. Sep 27 03:30:32 monitor kernel: Killed process 13305 (pbx_log.pl)
  1834. Sep 27 03:30:32 monitor kernel: pbx_log.pl invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  1835. Sep 27 03:30:32 monitor kernel: Pid: 9078, comm: pbx_log.pl Not tainted 2.6.24.3 #3
  1836. Sep 27 03:30:32 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  1837. Sep 27 03:30:32 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  1838. Sep 27 03:30:32 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  1839. Sep 27 03:30:32 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  1840. Sep 27 03:30:32 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  1841. Sep 27 03:30:32 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  1842. Sep 27 03:30:32 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  1843. Sep 27 03:30:32 monitor kernel: [<c0239258>] rb_insert_color+0x68/0xc0
  1844. Sep 27 03:30:32 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  1845. Sep 27 03:30:32 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  1846. Sep 27 03:30:32 monitor kernel: [<c0138540>] hrtimer_wakeup+0x0/0x20
  1847. Sep 27 03:30:32 monitor kernel: [<c0138a38>] sys_nanosleep+0x58/0x60
  1848. Sep 27 03:30:32 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  1849. Sep 27 03:30:32 monitor kernel: [<c043a202>] error_code+0x72/0x78
  1850. Sep 27 03:30:32 monitor kernel: =======================
  1851. Sep 27 03:30:32 monitor kernel: DMA per-cpu:
  1852. Sep 27 03:30:32 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1853. Sep 27 03:30:32 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1854. Sep 27 03:30:32 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1855. Sep 27 03:30:32 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1856. Sep 27 03:30:32 monitor kernel: Normal per-cpu:
  1857. Sep 27 03:30:32 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 120 Cold: hi: 62, btch: 15 usd: 48
  1858. Sep 27 03:30:32 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 81 Cold: hi: 62, btch: 15 usd: 51
  1859. Sep 27 03:30:32 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 86 Cold: hi: 62, btch: 15 usd: 56
  1860. Sep 27 03:30:32 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 179 Cold: hi: 62, btch: 15 usd: 57
  1861. Sep 27 03:30:32 monitor kernel: HighMem per-cpu:
  1862. Sep 27 03:30:32 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 44 Cold: hi: 62, btch: 15 usd: 54
  1863. Sep 27 03:30:32 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 28 Cold: hi: 62, btch: 15 usd: 54
  1864. Sep 27 03:30:32 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 73 Cold: hi: 62, btch: 15 usd: 37
  1865. Sep 27 03:30:32 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 80 Cold: hi: 62, btch: 15 usd: 56
  1866. Sep 27 03:30:32 monitor kernel: Active:447572 inactive:413356 dirty:0 writeback:0 unstable:0
  1867. Sep 27 03:30:32 monitor kernel: free:29117 slab:4621 mapped:8 pagetables:4104 bounce:0
  1868. Sep 27 03:30:32 monitor kernel: DMA free:10944kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  1869. able? yes
  1870. Sep 27 03:30:32 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  1871. Sep 27 03:30:32 monitor kernel: Normal free:105140kB min:3744kB low:4680kB high:5616kB active:380780kB inactive:319364kB present:894080kB pages_sca
  1872. nned:4715292 all_unreclaimable? yes
  1873. Sep 27 03:30:32 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  1874. Sep 27 03:30:32 monitor kernel: HighMem free:384kB min:512kB low:3916kB high:7324kB active:1409508kB inactive:1334060kB present:3251200kB pages_sca
  1875. nned:21259850 all_unreclaimable? yes
  1876. Sep 27 03:30:32 monitor kernel: lowmem_reserve[]: 0 0 0 0
  1877. Sep 27 03:30:32 monitor kernel: DMA: 80*4kB 72*8kB 62*16kB 45*32kB 23*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10944kB
  1878. Sep 27 03:30:32 monitor kernel: Normal: 2975*4kB 4125*8kB 1015*16kB 237*32kB 123*64kB 83*128kB 44*256kB 9*512kB 0*1024kB 1*2048kB 0*4096kB = 105140
  1879. kB
  1880. Sep 27 03:30:32 monitor kernel: HighMem: 28*4kB 14*8kB 0*16kB 3*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 384kB
  1881. Sep 27 03:30:32 monitor kernel: Swap cache: add 5540783, delete 5540797, find 1542326763/1542451673, race 145+83
  1882. Sep 27 03:30:32 monitor kernel: Free swap = 0kB
  1883. Sep 27 03:30:32 monitor kernel: Total swap = 3911816kB
  1884. Sep 27 03:30:32 monitor kernel: Out of memory: kill process 13832 (pbx_log.pl) score 13616 or a child
  1885. Sep 27 03:30:32 monitor kernel: Killed process 13832 (pbx_log.pl)
  1886. Sep 27 03:30:32 monitor kernel: pbx_log.pl invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  1887. Sep 27 03:30:32 monitor kernel: Pid: 9203, comm: pbx_log.pl Not tainted 2.6.24.3 #3
  1888. Sep 27 03:30:32 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  1889. Sep 27 03:30:32 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  1890. Sep 27 03:30:32 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  1891. Sep 27 03:30:32 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  1892. Sep 27 03:30:32 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  1893. Sep 27 03:30:32 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  1894. Sep 27 03:30:32 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  1895. Sep 27 03:30:32 monitor kernel: [<c0239258>] rb_insert_color+0x68/0xc0
  1896. Sep 27 03:30:32 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  1897. Sep 27 03:30:32 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  1898. Sep 27 03:30:32 monitor kernel: [<c0126af8>] sys_gettimeofday+0x28/0x80
  1899. Sep 27 03:30:32 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  1900. Sep 27 03:30:32 monitor kernel: [<c043a202>] error_code+0x72/0x78
  1901. Sep 27 03:30:32 monitor kernel: [<c0430000>] rpcb_getport_sync+0x0/0x190
  1902. Sep 27 03:30:32 monitor kernel: =======================
  1903. Sep 27 03:30:32 monitor kernel: DMA per-cpu:
  1904. Sep 27 03:30:32 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1905. Sep 27 03:30:32 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1906. Sep 27 03:30:32 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1907. Sep 27 03:30:32 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1908. Sep 27 03:30:32 monitor kernel: Normal per-cpu:
  1909. Sep 27 03:30:32 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 120 Cold: hi: 62, btch: 15 usd: 48
  1910. Sep 27 03:30:32 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 80 Cold: hi: 62, btch: 15 usd: 51
  1911. Sep 27 03:30:32 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 87 Cold: hi: 62, btch: 15 usd: 56
  1912. Sep 27 03:30:32 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 179 Cold: hi: 62, btch: 15 usd: 57
  1913. Sep 27 03:30:32 monitor kernel: HighMem per-cpu:
  1914. Sep 27 03:30:32 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 44 Cold: hi: 62, btch: 15 usd: 54
  1915. Sep 27 03:30:32 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 28 Cold: hi: 62, btch: 15 usd: 54
  1916. Sep 27 03:30:32 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 73 Cold: hi: 62, btch: 15 usd: 37
  1917. Sep 27 03:30:32 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 80 Cold: hi: 62, btch: 15 usd: 56
  1918. Sep 27 03:30:32 monitor kernel: Active:381161 inactive:479800 dirty:0 writeback:0 unstable:0
  1919. Sep 27 03:30:32 monitor kernel: free:29119 slab:4621 mapped:8 pagetables:4104 bounce:0
  1920. Sep 27 03:30:32 monitor kernel: DMA free:10944kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  1921. able? yes
  1922. Sep 27 03:30:32 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  1923. Sep 27 03:30:32 monitor kernel: Normal free:105148kB min:3744kB low:4680kB high:5616kB active:116020kB inactive:583852kB present:894080kB pages_sca
  1924. nned:18728916 all_unreclaimable? yes
  1925. Sep 27 03:30:32 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  1926. Sep 27 03:30:32 monitor kernel: HighMem free:384kB min:512kB low:3916kB high:7324kB active:1408624kB inactive:1335348kB present:3251200kB pages_sca
  1927. nned:21354620 all_unreclaimable? yes
  1928. Sep 27 03:30:32 monitor kernel: lowmem_reserve[]: 0 0 0 0
  1929. Sep 27 03:30:32 monitor kernel: DMA: 80*4kB 72*8kB 62*16kB 45*32kB 23*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10944kB
  1930. Sep 27 03:30:32 monitor kernel: Normal: 2975*4kB 4126*8kB 1015*16kB 237*32kB 123*64kB 83*128kB 44*256kB 9*512kB 0*1024kB 1*2048kB 0*4096kB = 105148
  1931. kB
  1932. Sep 27 03:30:32 monitor kernel: HighMem: 28*4kB 14*8kB 0*16kB 3*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 384kB
  1933. Sep 27 03:30:32 monitor kernel: Swap cache: add 5540783, delete 5540797, find 1542326763/1542451673, race 145+83
  1934. Sep 27 03:30:32 monitor kernel: Free swap = 0kB
  1935. Sep 27 03:30:32 monitor kernel: Total swap = 3911816kB
  1936. Sep 27 03:30:32 monitor kernel: Out of memory: kill process 9099 (pbx_log.pl) score 9464 or a child
  1937. Sep 27 03:30:32 monitor kernel: Killed process 9099 (pbx_log.pl)
  1938. Sep 27 03:30:32 monitor kernel: httpd invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  1939. Sep 27 03:30:32 monitor kernel: Pid: 24298, comm: httpd Not tainted 2.6.24.3 #3
  1940. Sep 27 03:30:32 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  1941. Sep 27 03:30:32 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  1942. Sep 27 03:30:32 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  1943. Sep 27 03:30:32 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  1944. Sep 27 03:30:32 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  1945. Sep 27 03:30:32 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  1946. Sep 27 03:30:32 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  1947. Sep 27 03:30:32 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  1948. Sep 27 03:30:32 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  1949. Sep 27 03:30:32 monitor kernel: [<c039bdb6>] sys_socketcall+0xc6/0x280
  1950. Sep 27 03:30:32 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  1951. Sep 27 03:30:32 monitor kernel: [<c043a202>] error_code+0x72/0x78
  1952. Sep 27 03:30:32 monitor kernel: [<c0430000>] rpcb_getport_sync+0x0/0x190
  1953. Sep 27 03:30:32 monitor kernel: =======================
  1954. Sep 27 03:30:32 monitor kernel: DMA per-cpu:
  1955. Sep 27 03:30:32 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1956. Sep 27 03:30:32 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1957. Sep 27 03:30:32 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1958. Sep 27 03:30:32 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  1959. Sep 27 03:30:32 monitor kernel: Normal per-cpu:
  1960. Sep 27 03:30:32 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 120 Cold: hi: 62, btch: 15 usd: 48
  1961. Sep 27 03:30:32 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 79 Cold: hi: 62, btch: 15 usd: 51
  1962. Sep 27 03:30:32 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 87 Cold: hi: 62, btch: 15 usd: 56
  1963. Sep 27 03:30:32 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 179 Cold: hi: 62, btch: 15 usd: 57
  1964. Sep 27 03:30:32 monitor kernel: HighMem per-cpu:
  1965. Sep 27 03:30:32 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 44 Cold: hi: 62, btch: 15 usd: 54
  1966. Sep 27 03:30:32 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 28 Cold: hi: 62, btch: 15 usd: 54
  1967. Sep 27 03:30:32 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 73 Cold: hi: 62, btch: 15 usd: 35
  1968. Sep 27 03:30:32 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 80 Cold: hi: 62, btch: 15 usd: 56
  1969. Sep 27 03:30:32 monitor kernel: Active:521522 inactive:341086 dirty:0 writeback:0 unstable:0
  1970. Sep 27 03:30:32 monitor kernel: free:29119 slab:4622 mapped:8 pagetables:4104 bounce:0
  1971. Sep 27 03:30:32 monitor kernel: DMA free:10944kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  1972. able? yes
  1973. Sep 27 03:30:32 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  1974. Sep 27 03:30:32 monitor kernel: Normal free:105148kB min:3744kB low:4680kB high:5616kB active:529856kB inactive:177508kB present:894080kB pages_sca
  1975. nned:25991200 all_unreclaimable? yes
  1976. Sep 27 03:30:32 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  1977. Sep 27 03:30:32 monitor kernel: HighMem free:384kB min:512kB low:3916kB high:7324kB active:1556232kB inactive:1186836kB present:3251200kB pages_sca
  1978. nned:26421423 all_unreclaimable? yes
  1979. Sep 27 03:30:32 monitor kernel: lowmem_reserve[]: 0 0 0 0
  1980. Sep 27 03:30:32 monitor kernel: DMA: 80*4kB 72*8kB 62*16kB 45*32kB 23*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10944kB
  1981. Sep 27 03:30:32 monitor kernel: Normal: 2975*4kB 4126*8kB 1015*16kB 237*32kB 123*64kB 83*128kB 44*256kB 9*512kB 0*1024kB 1*2048kB 0*4096kB = 105148
  1982. kB
  1983. Sep 27 03:30:32 monitor kernel: HighMem: 28*4kB 14*8kB 0*16kB 3*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 384kB
  1984. Sep 27 03:30:32 monitor kernel: Swap cache: add 5540783, delete 5540797, find 1542326763/1542451673, race 145+83
  1985. Sep 27 03:30:32 monitor kernel: Free swap = 0kB
  1986. Sep 27 03:30:32 monitor kernel: Total swap = 3911816kB
  1987. Sep 27 03:30:32 monitor kernel: Out of memory: kill process 9100 (pbx_log.pl) score 9464 or a child
  1988. Sep 27 03:30:32 monitor kernel: Killed process 9100 (pbx_log.pl)
  1989. Sep 27 04:36:50 monitor kernel: pbx_log.pl invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  1990. Sep 27 04:36:50 monitor kernel: Pid: 9376, comm: pbx_log.pl Not tainted 2.6.24.3 #3
  1991. Sep 27 04:36:50 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  1992. Sep 27 04:36:50 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  1993. Sep 27 04:36:50 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  1994. Sep 27 04:36:50 monitor kernel: [<c011abaa>] enqueue_task_fair+0x1a/0x30
  1995. Sep 27 04:36:50 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  1996. Sep 27 04:36:50 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  1997. Sep 27 04:36:50 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  1998. Sep 27 04:36:50 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  1999. Sep 27 04:36:50 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  2000. Sep 27 04:36:50 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  2001. Sep 27 04:36:50 monitor kernel: [<c0138540>] hrtimer_wakeup+0x0/0x20
  2002. Sep 27 04:36:50 monitor kernel: [<c0138a38>] sys_nanosleep+0x58/0x60
  2003. Sep 27 04:36:50 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  2004. Sep 27 04:36:50 monitor kernel: [<c043a202>] error_code+0x72/0x78
  2005. Sep 27 04:36:50 monitor kernel: =======================
  2006. Sep 27 04:36:50 monitor kernel: DMA per-cpu:
  2007. Sep 27 04:36:50 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2008. Sep 27 04:36:50 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2009. Sep 27 04:36:50 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2010. Sep 27 04:36:50 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2011. Sep 27 04:36:50 monitor kernel: Normal per-cpu:
  2012. Sep 27 04:36:50 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 159 Cold: hi: 62, btch: 15 usd: 51
  2013. Sep 27 04:36:50 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 70 Cold: hi: 62, btch: 15 usd: 58
  2014. Sep 27 04:36:50 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 56 Cold: hi: 62, btch: 15 usd: 49
  2015. Sep 27 04:36:50 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 101 Cold: hi: 62, btch: 15 usd: 56
  2016. Sep 27 04:36:50 monitor kernel: HighMem per-cpu:
  2017. Sep 27 04:36:50 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 35 Cold: hi: 62, btch: 15 usd: 51
  2018. Sep 27 04:36:50 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 15 Cold: hi: 62, btch: 15 usd: 51
  2019. Sep 27 04:36:50 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 37 Cold: hi: 62, btch: 15 usd: 49
  2020. Sep 27 04:36:50 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 145 Cold: hi: 62, btch: 15 usd: 53
  2021. Sep 27 04:36:50 monitor kernel: Active:485414 inactive:376675 dirty:0 writeback:0 unstable:0
  2022. Sep 27 04:36:50 monitor kernel: free:29157 slab:4499 mapped:20 pagetables:3957 bounce:0
  2023. Sep 27 04:36:50 monitor kernel: DMA free:10948kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  2024. able? yes
  2025. Sep 27 04:36:50 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  2026. Sep 27 04:36:50 monitor kernel: Normal free:105324kB min:3744kB low:4680kB high:5616kB active:356688kB inactive:356196kB present:894080kB pages_sca
  2027. nned:3646298 all_unreclaimable? yes
  2028. Sep 27 04:36:50 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  2029. Sep 27 04:36:50 monitor kernel: HighMem free:356kB min:512kB low:3916kB high:7324kB active:1584968kB inactive:1150504kB present:3251200kB pages_sca
  2030. nned:9954016 all_unreclaimable? yes
  2031. Sep 27 04:36:50 monitor kernel: lowmem_reserve[]: 0 0 0 0
  2032. Sep 27 04:36:50 monitor kernel: DMA: 79*4kB 71*8kB 63*16kB 45*32kB 23*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10948kB
  2033. Sep 27 04:36:50 monitor kernel: Normal: 2259*4kB 3826*8kB 1253*16kB 300*32kB 117*64kB 81*128kB 47*256kB 8*512kB 0*1024kB 1*2048kB 0*4096kB = 105324
  2034. kB
  2035. Sep 27 04:36:50 monitor kernel: HighMem: 15*4kB 19*8kB 1*16kB 0*32kB 2*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 356kB
  2036. Sep 27 04:36:50 monitor kernel: Swap cache: add 5916441, delete 5916457, find 1542537405/1542665904, race 145+83
  2037. Sep 27 04:36:50 monitor kernel: Free swap = 0kB
  2038. Sep 27 04:36:50 monitor kernel: Total swap = 3911816kB
  2039. Sep 27 04:36:50 monitor kernel: Out of memory: kill process 12626 (pbx_log.pl) score 27681 or a child
  2040. Sep 27 04:36:50 monitor kernel: Killed process 12626 (pbx_log.pl)
  2041. Sep 27 04:36:50 monitor kernel: postgres invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  2042. Sep 27 04:36:50 monitor kernel: Pid: 12740, comm: postgres Not tainted 2.6.24.3 #3
  2043. Sep 27 04:36:50 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  2044. Sep 27 04:36:50 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  2045. Sep 27 04:36:50 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  2046. Sep 27 04:36:50 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  2047. Sep 27 04:36:50 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  2048. Sep 27 04:36:50 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  2049. Sep 27 04:36:50 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  2050. Sep 27 04:36:50 monitor kernel: [<c014f588>] __lock_page+0x68/0x70
  2051. Sep 27 04:36:50 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  2052. Sep 27 04:36:50 monitor kernel: [<c0160509>] remove_vma+0x39/0x50
  2053. Sep 27 04:36:50 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  2054. Sep 27 04:36:50 monitor kernel: [<c0161077>] do_munmap+0x197/0x1f0
  2055. Sep 27 04:36:50 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  2056. Sep 27 04:36:50 monitor kernel: [<c043a202>] error_code+0x72/0x78
  2057. Sep 27 04:36:50 monitor kernel: [<c0430000>] rpcb_getport_sync+0x0/0x190
  2058. Sep 27 04:36:50 monitor kernel: =======================
  2059. Sep 27 04:36:50 monitor kernel: DMA per-cpu:
  2060. Sep 27 04:36:50 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2061. Sep 27 04:36:50 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2062. Sep 27 04:36:50 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2063. Sep 27 04:36:50 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2064. Sep 27 04:36:50 monitor kernel: Normal per-cpu:
  2065. Sep 27 04:36:50 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 159 Cold: hi: 62, btch: 15 usd: 51
  2066. Sep 27 04:36:50 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 100 Cold: hi: 62, btch: 15 usd: 60
  2067. Sep 27 04:36:50 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 56 Cold: hi: 62, btch: 15 usd: 49
  2068. Sep 27 04:36:50 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 101 Cold: hi: 62, btch: 15 usd: 56
  2069. Sep 27 04:36:50 monitor kernel: HighMem per-cpu:
  2070. Sep 27 04:36:50 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 35 Cold: hi: 62, btch: 15 usd: 51
  2071. Sep 27 04:36:50 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 15 Cold: hi: 62, btch: 15 usd: 51
  2072. Sep 27 04:36:50 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 37 Cold: hi: 62, btch: 15 usd: 49
  2073. Sep 27 04:36:50 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 145 Cold: hi: 62, btch: 15 usd: 53
  2074. Sep 27 04:36:50 monitor kernel: Active:128577 inactive:733707 dirty:0 writeback:0 unstable:0
  2075. Sep 27 04:36:50 monitor kernel: free:29128 slab:4500 mapped:20 pagetables:3957 bounce:0
  2076. Sep 27 04:36:50 monitor kernel: DMA free:10948kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  2077. able? yes
  2078. Sep 27 04:36:50 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  2079. Sep 27 04:36:50 monitor kernel: Normal free:105208kB min:3744kB low:4680kB high:5616kB active:355980kB inactive:353740kB present:894080kB pages_sca
  2080. nned:4283458 all_unreclaimable? yes
  2081. Sep 27 04:36:50 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  2082. Sep 27 04:36:50 monitor kernel: HighMem free:356kB min:512kB low:3916kB high:7324kB active:158328kB inactive:2581088kB present:3251200kB pages_scan
  2083. ned:14431901 all_unreclaimable? yes
  2084. Sep 27 04:36:50 monitor kernel: lowmem_reserve[]: 0 0 0 0
  2085. Sep 27 04:36:50 monitor kernel: DMA: 79*4kB 71*8kB 63*16kB 45*32kB 23*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10948kB
  2086. Sep 27 04:36:50 monitor kernel: Normal: 2228*4kB 3827*8kB 1253*16kB 300*32kB 117*64kB 81*128kB 47*256kB 8*512kB 0*1024kB 1*2048kB 0*4096kB = 105208
  2087. kB
  2088. Sep 27 04:36:50 monitor kernel: HighMem: 15*4kB 19*8kB 1*16kB 0*32kB 2*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 356kB
  2089. Sep 27 04:36:50 monitor kernel: Swap cache: add 5916441, delete 5916457, find 1542537405/1542665904, race 145+83
  2090. Sep 27 04:36:50 monitor kernel: Free swap = 0kB
  2091. Sep 27 04:36:50 monitor kernel: Total swap = 3911816kB
  2092. Sep 27 04:36:50 monitor kernel: Out of memory: kill process 12123 (perl) score 25518 or a child
  2093. Sep 27 04:36:50 monitor kernel: Killed process 12123 (perl)
  2094. Sep 27 04:36:50 monitor kernel: sh invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  2095. Sep 27 04:36:50 monitor kernel: Pid: 12753, comm: sh Not tainted 2.6.24.3 #3
  2096. Sep 27 04:36:50 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  2097. Sep 27 04:36:50 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  2098. Sep 27 04:36:50 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  2099. Sep 27 04:36:50 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  2100. Sep 27 04:36:50 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  2101. Sep 27 04:36:50 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  2102. Sep 27 04:36:50 monitor kernel: [<c0179a2e>] permission+0x7e/0x160
  2103. Sep 27 04:36:50 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  2104. Sep 27 04:36:50 monitor kernel: [<c017d0e9>] open_namei+0x79/0x5b0
  2105. Sep 27 04:36:50 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  2106. Sep 27 04:36:50 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  2107. Sep 27 04:36:50 monitor kernel: [<c0170dbc>] do_sys_open+0xbc/0xe0
  2108. Sep 27 04:36:50 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  2109. Sep 27 04:36:50 monitor kernel: [<c043a202>] error_code+0x72/0x78
  2110. Sep 27 04:36:50 monitor kernel: [<c0430000>] rpcb_getport_sync+0x0/0x190
  2111. Sep 27 04:36:50 monitor kernel: =======================
  2112. Sep 27 04:36:50 monitor kernel: DMA per-cpu:
  2113. Sep 27 04:36:50 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2114. Sep 27 04:36:50 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2115. Sep 27 04:36:50 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2116. Sep 27 04:36:50 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2117. Sep 27 04:36:50 monitor kernel: Normal per-cpu:
  2118. Sep 27 04:36:50 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 159 Cold: hi: 62, btch: 15 usd: 51
  2119. Sep 27 04:36:50 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 82 Cold: hi: 62, btch: 15 usd: 60
  2120. Sep 27 04:36:50 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 56 Cold: hi: 62, btch: 15 usd: 49
  2121. Sep 27 04:36:50 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 101 Cold: hi: 62, btch: 15 usd: 56
  2122. Sep 27 04:36:50 monitor kernel: HighMem per-cpu:
  2123. Sep 27 04:36:50 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 35 Cold: hi: 62, btch: 15 usd: 54
  2124. Sep 27 04:36:50 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 15 Cold: hi: 62, btch: 15 usd: 54
  2125. Sep 27 04:36:50 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 37 Cold: hi: 62, btch: 15 usd: 21
  2126. Sep 27 04:36:50 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 145 Cold: hi: 62, btch: 15 usd: 53
  2127. Sep 27 04:36:50 monitor kernel: Active:470512 inactive:393452 dirty:0 writeback:0 unstable:0
  2128. Sep 27 04:36:50 monitor kernel: free:29128 slab:4518 mapped:8 pagetables:3957 bounce:0
  2129. Sep 27 04:36:50 monitor kernel: DMA free:10948kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  2130. able? yes
  2131. Sep 27 04:36:50 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  2132. Sep 27 04:36:50 monitor kernel: Normal free:105208kB min:3744kB low:4680kB high:5616kB active:450956kB inactive:262140kB present:894080kB pages_sca
  2133. nned:13871974 all_unreclaimable? yes
  2134. Sep 27 04:36:50 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  2135. Sep 27 04:36:50 monitor kernel: HighMem free:356kB min:512kB low:3916kB high:7324kB active:1431092kB inactive:1311668kB present:3251200kB pages_sca
  2136. nned:46678744 all_unreclaimable? yes
  2137. Sep 27 04:36:50 monitor kernel: lowmem_reserve[]: 0 0 0 0
  2138. Sep 27 04:36:50 monitor kernel: DMA: 79*4kB 71*8kB 63*16kB 45*32kB 23*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10948kB
  2139. Sep 27 04:36:50 monitor kernel: Normal: 2228*4kB 3827*8kB 1253*16kB 300*32kB 117*64kB 81*128kB 47*256kB 8*512kB 0*1024kB 1*2048kB 0*4096kB = 105208
  2140. kB
  2141. Sep 27 04:36:50 monitor kernel: HighMem: 15*4kB 19*8kB 1*16kB 0*32kB 2*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 356kB
  2142. Sep 27 04:36:50 monitor kernel: Swap cache: add 5916441, delete 5916457, find 1542537405/1542665904, race 145+83
  2143. Sep 27 04:36:50 monitor kernel: Free swap = 0kB
  2144. Sep 27 04:36:50 monitor kernel: Total swap = 3911816kB
  2145. Sep 27 04:36:50 monitor kernel: Out of memory: kill process 12399 (pbx_log.pl) score 23045 or a child
  2146. Sep 27 04:36:50 monitor kernel: Killed process 12399 (pbx_log.pl)
  2147. Sep 27 04:36:50 monitor kernel: pbx_log.pl invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  2148. Sep 27 04:36:50 monitor kernel: Pid: 9195, comm: pbx_log.pl Not tainted 2.6.24.3 #3
  2149. Sep 27 04:36:50 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  2150. Sep 27 04:36:50 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  2151. Sep 27 04:36:50 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  2152. Sep 27 04:36:50 monitor kernel: [<c011abaa>] enqueue_task_fair+0x1a/0x30
  2153. Sep 27 04:36:50 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  2154. Sep 27 04:36:50 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  2155. Sep 27 04:36:50 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  2156. Sep 27 04:36:50 monitor kernel: [<c011c0ee>] __wake_up+0x3e/0x60
  2157. Sep 27 04:36:50 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  2158. Sep 27 04:36:50 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  2159. Sep 27 04:36:50 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  2160. Sep 27 04:36:50 monitor kernel: [<c0126af8>] sys_gettimeofday+0x28/0x80
  2161. Sep 27 04:36:50 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  2162. Sep 27 04:36:50 monitor kernel: [<c043a202>] error_code+0x72/0x78
  2163. Sep 27 04:36:50 monitor kernel: [<c0430000>] rpcb_getport_sync+0x0/0x190
  2164. Sep 27 04:36:50 monitor kernel: =======================
  2165. Sep 27 04:36:50 monitor kernel: DMA per-cpu:
  2166. Sep 27 04:36:50 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2167. Sep 27 04:36:50 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2168. Sep 27 04:36:50 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2169. Sep 27 04:36:50 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2170. Sep 27 04:36:50 monitor kernel: Normal per-cpu:
  2171. Sep 27 04:36:50 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 159 Cold: hi: 62, btch: 15 usd: 51
  2172. Sep 27 04:36:50 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 80 Cold: hi: 62, btch: 15 usd: 60
  2173. Sep 27 04:36:50 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 56 Cold: hi: 62, btch: 15 usd: 49
  2174. Sep 27 04:36:50 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 101 Cold: hi: 62, btch: 15 usd: 56
  2175. Sep 27 04:36:50 monitor kernel: HighMem per-cpu:
  2176. Sep 27 04:36:50 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 35 Cold: hi: 62, btch: 15 usd: 54
  2177. Sep 27 04:36:50 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 15 Cold: hi: 62, btch: 15 usd: 54
  2178. Sep 27 04:36:50 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 37 Cold: hi: 62, btch: 15 usd: 4
  2179. Sep 27 04:36:50 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 145 Cold: hi: 62, btch: 15 usd: 53
  2180. Sep 27 04:36:50 monitor kernel: Active:347373 inactive:515314 dirty:0 writeback:0 unstable:0
  2181. Sep 27 04:36:50 monitor kernel: free:29117 slab:4518 mapped:10 pagetables:3957 bounce:0
  2182. Sep 27 04:36:50 monitor kernel: DMA free:10948kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  2183. able? yes
  2184. Sep 27 04:36:50 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  2185. Sep 27 04:36:50 monitor kernel: Normal free:105224kB min:3744kB low:4680kB high:5616kB active:353952kB inactive:353464kB present:894080kB pages_sca
  2186. nned:2056094 all_unreclaimable? yes
  2187. Sep 27 04:36:50 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  2188. Sep 27 04:36:50 monitor kernel: HighMem free:296kB min:512kB low:3916kB high:7324kB active:1035540kB inactive:1707792kB present:3251200kB pages_sca
  2189. nned:46845361 all_unreclaimable? yes
  2190. Sep 27 04:36:50 monitor kernel: lowmem_reserve[]: 0 0 0 0
  2191. Sep 27 04:36:50 monitor kernel: DMA: 79*4kB 71*8kB 63*16kB 45*32kB 23*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10948kB
  2192. Sep 27 04:36:50 monitor kernel: Normal: 2228*4kB 3829*8kB 1253*16kB 300*32kB 117*64kB 81*128kB 47*256kB 8*512kB 0*1024kB 1*2048kB 0*4096kB = 105224
  2193. kB
  2194. Sep 27 04:36:50 monitor kernel: HighMem: 0*4kB 19*8kB 1*16kB 0*32kB 2*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 296kB
  2195. Sep 27 04:36:50 monitor kernel: Swap cache: add 5916441, delete 5916457, find 1542537405/1542665904, race 145+83
  2196. Sep 27 04:36:50 monitor kernel: Free swap = 0kB
  2197. Sep 27 04:36:50 monitor kernel: Total swap = 3911816kB
  2198. Sep 27 04:36:50 monitor kernel: Out of memory: kill process 12744 (pbx_log.pl) score 23008 or a child
  2199. Sep 27 04:36:50 monitor kernel: Killed process 12744 (pbx_log.pl)
  2200. Sep 27 04:36:50 monitor kernel: nagios invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  2201. Sep 27 04:36:50 monitor kernel: Pid: 12755, comm: nagios Not tainted 2.6.24.3 #3
  2202. Sep 27 04:36:50 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  2203. Sep 27 04:36:50 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  2204. Sep 27 04:36:50 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  2205. Sep 27 04:36:50 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  2206. Sep 27 04:36:50 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  2207. Sep 27 04:36:50 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  2208. Sep 27 04:36:50 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  2209. Sep 27 04:36:50 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  2210. Sep 27 04:36:50 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  2211. Sep 27 04:36:50 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  2212. Sep 27 04:36:50 monitor kernel: [<c043a202>] error_code+0x72/0x78
  2213. Sep 27 04:36:50 monitor kernel: =======================
  2214. Sep 27 04:36:50 monitor kernel: DMA per-cpu:
  2215. Sep 27 04:36:50 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2216. Sep 27 04:36:50 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2217. Sep 27 04:36:50 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2218. Sep 27 04:36:50 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2219. Sep 27 04:36:50 monitor kernel: Normal per-cpu:
  2220. Sep 27 04:36:50 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 159 Cold: hi: 62, btch: 15 usd: 51
  2221. Sep 27 04:36:50 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 85 Cold: hi: 62, btch: 15 usd: 60
  2222. Sep 27 04:36:50 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 56 Cold: hi: 62, btch: 15 usd: 49
  2223. Sep 27 04:36:50 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 101 Cold: hi: 62, btch: 15 usd: 56
  2224. Sep 27 04:36:50 monitor kernel: HighMem per-cpu:
  2225. Sep 27 04:36:50 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 35 Cold: hi: 62, btch: 15 usd: 54
  2226. Sep 27 04:36:50 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 15 Cold: hi: 62, btch: 15 usd: 54
  2227. Sep 27 04:36:50 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 37 Cold: hi: 62, btch: 15 usd: 4
  2228. Sep 27 04:36:50 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 145 Cold: hi: 62, btch: 15 usd: 53
  2229. Sep 27 04:36:50 monitor kernel: Active:406887 inactive:455482 dirty:0 writeback:0 unstable:0
  2230. Sep 27 04:36:50 monitor kernel: free:29088 slab:4546 mapped:10 pagetables:3957 bounce:0
  2231. Sep 27 04:36:50 monitor kernel: DMA free:10948kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  2232. able? yes
  2233. Sep 27 04:36:50 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  2234. Sep 27 04:36:50 monitor kernel: Normal free:105108kB min:3744kB low:4680kB high:5616kB active:443952kB inactive:261884kB present:894080kB pages_sca
  2235. nned:244425784 all_unreclaimable? yes
  2236. Sep 27 04:36:50 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  2237. Sep 27 04:36:50 monitor kernel: HighMem free:296kB min:512kB low:3916kB high:7324kB active:1183596kB inactive:1559916kB present:3251200kB pages_sca
  2238. nned:46978528 all_unreclaimable? yes
  2239. Sep 27 04:36:50 monitor kernel: lowmem_reserve[]: 0 0 0 0
  2240. Sep 27 04:36:50 monitor kernel: DMA: 79*4kB 71*8kB 63*16kB 45*32kB 23*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10948kB
  2241. Sep 27 04:36:50 monitor kernel: Normal: 2197*4kB 3830*8kB 1253*16kB 300*32kB 117*64kB 81*128kB 47*256kB 8*512kB 0*1024kB 1*2048kB 0*4096kB = 105108
  2242. kB
  2243. Sep 27 04:36:50 monitor kernel: HighMem: 0*4kB 19*8kB 1*16kB 0*32kB 2*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 296kB
  2244. Sep 27 04:36:50 monitor kernel: Swap cache: add 5916441, delete 5916457, find 1542537405/1542665904, race 145+83
  2245. Sep 27 04:36:50 monitor kernel: Free swap = 0kB
  2246. Sep 27 04:36:50 monitor kernel: Total swap = 3911816kB
  2247. Sep 27 04:36:50 monitor kernel: Out of memory: kill process 12718 (pbx_log.pl) score 21500 or a child
  2248. Sep 27 04:36:50 monitor kernel: Killed process 12718 (pbx_log.pl)
  2249. Sep 27 04:36:50 monitor kernel: httpd invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  2250. Sep 27 04:36:50 monitor kernel: Pid: 7957, comm: httpd Not tainted 2.6.24.3 #3
  2251. Sep 27 04:36:50 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  2252. Sep 27 04:36:50 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  2253. Sep 27 04:36:50 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  2254. Sep 27 04:36:50 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  2255. Sep 27 04:36:50 monitor kernel: [<c01b7b70>] ext3_get_block+0x0/0x100
  2256. Sep 27 04:36:50 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  2257. Sep 27 04:36:50 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  2258. Sep 27 04:36:50 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  2259. Sep 27 04:36:50 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  2260. Sep 27 04:36:50 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  2261. Sep 27 04:36:50 monitor kernel: [<c0126af8>] sys_gettimeofday+0x28/0x80
  2262. Sep 27 04:36:50 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  2263. Sep 27 04:36:50 monitor kernel: [<c043a202>] error_code+0x72/0x78
  2264. Sep 27 04:36:50 monitor kernel: [<c0430000>] rpcb_getport_sync+0x0/0x190
  2265. Sep 27 04:36:50 monitor kernel: =======================
  2266. Sep 27 04:36:50 monitor kernel: DMA per-cpu:
  2267. Sep 27 04:36:50 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2268. Sep 27 04:36:50 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2269. Sep 27 04:36:50 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2270. Sep 27 04:36:50 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2271. Sep 27 04:36:50 monitor kernel: Normal per-cpu:
  2272. Sep 27 04:36:50 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 159 Cold: hi: 62, btch: 15 usd: 51
  2273. Sep 27 04:36:50 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 85 Cold: hi: 62, btch: 15 usd: 60
  2274. Sep 27 04:36:50 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 56 Cold: hi: 62, btch: 15 usd: 49
  2275. Sep 27 04:36:50 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 101 Cold: hi: 62, btch: 15 usd: 56
  2276. Sep 27 04:36:50 monitor kernel: HighMem per-cpu:
  2277. Sep 27 04:36:50 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 35 Cold: hi: 62, btch: 15 usd: 54
  2278. Sep 27 04:36:50 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 15 Cold: hi: 62, btch: 15 usd: 54
  2279. Sep 27 04:36:50 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 37 Cold: hi: 62, btch: 15 usd: 4
  2280. Sep 27 04:36:50 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 146 Cold: hi: 62, btch: 15 usd: 26
  2281. Sep 27 04:36:50 monitor kernel: Active:480065 inactive:383094 dirty:0 writeback:0 unstable:0
  2282. Sep 27 04:36:50 monitor kernel: free:29088 slab:4546 mapped:10 pagetables:3957 bounce:0
  2283. Sep 27 04:36:50 monitor kernel: DMA free:10948kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  2284. able? yes
  2285. Sep 27 04:36:50 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  2286. Sep 27 04:36:50 monitor kernel: Normal free:105108kB min:3744kB low:4680kB high:5616kB active:688784kB inactive:19988kB present:894080kB pages_scan
  2287. ned:245016757 all_unreclaimable? yes
  2288. Sep 27 04:36:50 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  2289. Sep 27 04:36:50 monitor kernel: HighMem free:296kB min:512kB low:3916kB high:7324kB active:1231276kB inactive:1512388kB present:3251200kB pages_sca
  2290. nned:47071497 all_unreclaimable? yes
  2291. Sep 27 04:36:50 monitor kernel: lowmem_reserve[]: 0 0 0 0
  2292. Sep 27 04:36:50 monitor kernel: DMA: 79*4kB 71*8kB 63*16kB 45*32kB 23*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10948kB
  2293. Sep 27 04:36:50 monitor kernel: Normal: 2197*4kB 3830*8kB 1253*16kB 300*32kB 117*64kB 81*128kB 47*256kB 8*512kB 0*1024kB 1*2048kB 0*4096kB = 105108
  2294. kB
  2295. Sep 27 04:36:50 monitor kernel: HighMem: 0*4kB 19*8kB 1*16kB 0*32kB 2*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 296kB
  2296. Sep 27 04:36:50 monitor kernel: Swap cache: add 5916441, delete 5916457, find 1542537405/1542665904, race 145+83
  2297. Sep 27 04:36:50 monitor kernel: Free swap = 0kB
  2298. Sep 27 04:36:50 monitor kernel: Total swap = 3911816kB
  2299. Sep 27 04:36:50 monitor kernel: Out of memory: kill process 12701 (pbx_log.pl) score 20768 or a child
  2300. Sep 27 04:36:50 monitor kernel: Killed process 12701 (pbx_log.pl)
  2301. Sep 27 04:36:50 monitor kernel: smokeping.cgi invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  2302. Sep 27 04:36:50 monitor kernel: Pid: 12721, comm: smokeping.cgi Not tainted 2.6.24.3 #3
  2303. Sep 27 04:36:50 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  2304. Sep 27 04:36:50 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  2305. Sep 27 04:36:50 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  2306. Sep 27 04:36:50 monitor kernel: [<c011abaa>] enqueue_task_fair+0x1a/0x30
  2307. Sep 27 04:36:50 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  2308. Sep 27 04:36:50 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  2309. Sep 27 04:36:50 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  2310. Sep 27 04:36:50 monitor kernel: [<c011c0ee>] __wake_up+0x3e/0x60
  2311. Sep 27 04:36:50 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  2312. Sep 27 04:36:50 monitor kernel: [<c01758a8>] cp_new_stat64+0xf8/0x110
  2313. Sep 27 04:36:50 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  2314. Sep 27 04:36:50 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  2315. Sep 27 04:36:50 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  2316. Sep 27 04:36:50 monitor kernel: [<c043a202>] error_code+0x72/0x78
  2317. Sep 27 04:36:50 monitor kernel: [<c0430000>] rpcb_getport_sync+0x0/0x190
  2318. Sep 27 04:36:50 monitor kernel: =======================
  2319. Sep 27 04:36:50 monitor kernel: DMA per-cpu:
  2320. Sep 27 04:36:50 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2321. Sep 27 04:36:50 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2322. Sep 27 04:36:50 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2323. Sep 27 04:36:50 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2324. Sep 27 04:36:50 monitor kernel: Normal per-cpu:
  2325. Sep 27 04:36:50 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 163 Cold: hi: 62, btch: 15 usd: 51
  2326. Sep 27 04:36:50 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 79 Cold: hi: 62, btch: 15 usd: 61
  2327. Sep 27 04:36:50 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 56 Cold: hi: 62, btch: 15 usd: 49
  2328. Sep 27 04:36:50 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 98 Cold: hi: 62, btch: 15 usd: 56
  2329. Sep 27 04:36:50 monitor kernel: HighMem per-cpu:
  2330. Sep 27 04:36:50 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 35 Cold: hi: 62, btch: 15 usd: 37
  2331. Sep 27 04:36:50 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 15 Cold: hi: 62, btch: 15 usd: 45
  2332. Sep 27 04:36:50 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 37 Cold: hi: 62, btch: 15 usd: 57
  2333. Sep 27 04:36:50 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 146 Cold: hi: 62, btch: 15 usd: 34
  2334. Sep 27 04:36:50 monitor kernel: Active:431580 inactive:430608 dirty:0 writeback:0 unstable:0
  2335. Sep 27 04:36:50 monitor kernel: free:29137 slab:4550 mapped:8 pagetables:3957 bounce:0
  2336. Sep 27 04:36:50 monitor kernel: DMA free:10948kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  2337. able? yes
  2338. Sep 27 04:36:50 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  2339. Sep 27 04:36:50 monitor kernel: Normal free:105124kB min:3744kB low:4680kB high:5616kB active:324764kB inactive:388132kB present:894080kB pages_sca
  2340. nned:1218444 all_unreclaimable? yes
  2341. Sep 27 04:36:50 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  2342. Sep 27 04:36:50 monitor kernel: HighMem free:476kB min:512kB low:3916kB high:7324kB active:1401556kB inactive:1334300kB present:3251200kB pages_sca
  2343. nned:18251917 all_unreclaimable? yes
  2344. Sep 27 04:36:50 monitor kernel: lowmem_reserve[]: 0 0 0 0
  2345. Sep 27 04:36:50 monitor kernel: DMA: 79*4kB 71*8kB 63*16kB 45*32kB 23*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10948kB
  2346. Sep 27 04:36:50 monitor kernel: Normal: 2197*4kB 3832*8kB 1253*16kB 300*32kB 117*64kB 81*128kB 47*256kB 8*512kB 0*1024kB 1*2048kB 0*4096kB = 105124
  2347. kB
  2348. Sep 27 04:36:50 monitor kernel: HighMem: 45*4kB 19*8kB 1*16kB 0*32kB 2*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 476kB
  2349. Sep 27 04:36:50 monitor kernel: Swap cache: add 5916441, delete 5916457, find 1542537405/1542665904, race 145+83
  2350. Sep 27 04:36:50 monitor kernel: Free swap = 0kB
  2351. Sep 27 04:36:50 monitor kernel: Total swap = 3911816kB
  2352. Sep 27 04:36:50 monitor kernel: Out of memory: kill process 12702 (pbx_log.pl) score 20147 or a child
  2353. Sep 27 04:36:50 monitor kernel: Killed process 12702 (pbx_log.pl)
  2354. Sep 27 04:36:50 monitor kernel: pbx_log.pl invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  2355. Sep 27 04:36:50 monitor kernel: Pid: 9301, comm: pbx_log.pl Not tainted 2.6.24.3 #3
  2356. Sep 27 04:36:50 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  2357. Sep 27 04:36:50 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  2358. Sep 27 04:36:50 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  2359. Sep 27 04:36:50 monitor kernel: [<c011abaa>] enqueue_task_fair+0x1a/0x30
  2360. Sep 27 04:36:50 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  2361. Sep 27 04:36:50 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  2362. Sep 27 04:36:50 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  2363. Sep 27 04:36:50 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  2364. Sep 27 04:36:50 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  2365. Sep 27 04:36:50 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  2366. Sep 27 04:36:50 monitor kernel: [<c0126af8>] sys_gettimeofday+0x28/0x80
  2367. Sep 27 04:36:50 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  2368. Sep 27 04:36:50 monitor kernel: [<c043a202>] error_code+0x72/0x78
  2369. Sep 27 04:36:50 monitor kernel: [<c0430000>] rpcb_getport_sync+0x0/0x190
  2370. Sep 27 04:36:50 monitor kernel: =======================
  2371. Sep 27 04:36:50 monitor kernel: DMA per-cpu:
  2372. Sep 27 04:36:50 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2373. Sep 27 04:36:50 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2374. Sep 27 04:36:50 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2375. Sep 27 04:36:50 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2376. Sep 27 04:36:50 monitor kernel: Normal per-cpu:
  2377. Sep 27 04:36:50 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 163 Cold: hi: 62, btch: 15 usd: 52
  2378. Sep 27 04:36:50 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 77 Cold: hi: 62, btch: 15 usd: 61
  2379. Sep 27 04:36:50 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 56 Cold: hi: 62, btch: 15 usd: 49
  2380. Sep 27 04:36:50 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 98 Cold: hi: 62, btch: 15 usd: 56
  2381. Sep 27 04:36:50 monitor kernel: HighMem per-cpu:
  2382. Sep 27 04:36:50 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 35 Cold: hi: 62, btch: 15 usd: 5
  2383. Sep 27 04:36:50 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 15 Cold: hi: 62, btch: 15 usd: 45
  2384. Sep 27 04:36:50 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 37 Cold: hi: 62, btch: 15 usd: 57
  2385. Sep 27 04:36:50 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 146 Cold: hi: 62, btch: 15 usd: 34
  2386. Sep 27 04:36:50 monitor kernel: Active:509827 inactive:354065 dirty:0 writeback:0 unstable:0
  2387. Sep 27 04:36:50 monitor kernel: free:29137 slab:4552 mapped:8 pagetables:3957 bounce:0
  2388. Sep 27 04:36:50 monitor kernel: DMA free:10948kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  2389. able? yes
  2390. Sep 27 04:36:50 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  2391. Sep 27 04:36:50 monitor kernel: Normal free:105124kB min:3744kB low:4680kB high:5616kB active:450704kB inactive:262380kB present:894080kB pages_sca
  2392. nned:5847088 all_unreclaimable? yes
  2393. Sep 27 04:36:50 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  2394. Sep 27 04:36:50 monitor kernel: HighMem free:476kB min:512kB low:3916kB high:7324kB active:1588604kB inactive:1153880kB present:3251200kB pages_sca
  2395. nned:50494607 all_unreclaimable? yes
  2396. Sep 27 04:36:50 monitor kernel: lowmem_reserve[]: 0 0 0 0
  2397. Sep 27 04:36:50 monitor kernel: DMA: 79*4kB 71*8kB 63*16kB 45*32kB 23*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10948kB
  2398. Sep 27 04:36:50 monitor kernel: Normal: 2197*4kB 3832*8kB 1253*16kB 300*32kB 117*64kB 81*128kB 47*256kB 8*512kB 0*1024kB 1*2048kB 0*4096kB = 105124
  2399. kB
  2400. Sep 27 04:36:50 monitor kernel: HighMem: 45*4kB 19*8kB 1*16kB 0*32kB 2*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 476kB
  2401. Sep 27 04:36:50 monitor kernel: Swap cache: add 5916441, delete 5916457, find 1542537405/1542665904, race 145+83
  2402. Sep 27 04:36:50 monitor kernel: Free swap = 0kB
  2403. Sep 27 04:36:50 monitor kernel: Total swap = 3911816kB
  2404. Sep 27 04:36:50 monitor kernel: Out of memory: kill process 12748 (pbx_log.pl) score 20147 or a child
  2405. Sep 27 04:36:50 monitor kernel: Killed process 12748 (pbx_log.pl)
  2406. Sep 27 04:36:50 monitor kernel: httpd invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  2407. Sep 27 04:36:50 monitor kernel: Pid: 26883, comm: httpd Not tainted 2.6.24.3 #3
  2408. Sep 27 04:36:50 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  2409. Sep 27 04:36:50 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  2410. Sep 27 04:36:50 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  2411. Sep 27 04:36:50 monitor kernel: [<c0154a5a>] get_page_from_freelist+0x29a/0x410
  2412. Sep 27 04:36:50 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  2413. Sep 27 04:36:50 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  2414. Sep 27 04:36:50 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  2415. Sep 27 04:36:50 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  2416. Sep 27 04:36:50 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  2417. Sep 27 04:36:50 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  2418. Sep 27 04:36:50 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  2419. Sep 27 04:36:50 monitor kernel: [<c043a202>] error_code+0x72/0x78
  2420. Sep 27 04:36:50 monitor kernel: =======================
  2421. Sep 27 04:36:50 monitor kernel: DMA per-cpu:
  2422. Sep 27 04:36:50 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2423. Sep 27 04:36:50 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2424. Sep 27 04:36:50 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2425. Sep 27 04:36:50 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2426. Sep 27 04:36:50 monitor kernel: Normal per-cpu:
  2427. Sep 27 04:36:50 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 163 Cold: hi: 62, btch: 15 usd: 52
  2428. Sep 27 04:36:50 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 73 Cold: hi: 62, btch: 15 usd: 61
  2429. Sep 27 04:36:50 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 56 Cold: hi: 62, btch: 15 usd: 49
  2430. Sep 27 04:36:50 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 98 Cold: hi: 62, btch: 15 usd: 56
  2431. Sep 27 04:36:50 monitor kernel: HighMem per-cpu:
  2432. Sep 27 04:36:50 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 35 Cold: hi: 62, btch: 15 usd: 14
  2433. Sep 27 04:36:50 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 15 Cold: hi: 62, btch: 15 usd: 45
  2434. Sep 27 04:36:50 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 37 Cold: hi: 62, btch: 15 usd: 57
  2435. Sep 27 04:36:50 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 146 Cold: hi: 62, btch: 15 usd: 34
  2436. Sep 27 04:36:50 monitor kernel: Active:511044 inactive:351260 dirty:0 writeback:0 unstable:0
  2437. Sep 27 04:36:50 monitor kernel: free:29126 slab:4556 mapped:10 pagetables:3957 bounce:0
  2438. Sep 27 04:36:50 monitor kernel: DMA free:10948kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  2439. able? yes
  2440. Sep 27 04:36:50 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  2441. Sep 27 04:36:50 monitor kernel: Normal free:105140kB min:3744kB low:4680kB high:5616kB active:409640kB inactive:296244kB present:894080kB pages_sca
  2442. nned:112832493 all_unreclaimable? yes
  2443. Sep 27 04:36:50 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  2444. Sep 27 04:36:50 monitor kernel: HighMem free:416kB min:512kB low:3916kB high:7324kB active:1634536kB inactive:1108908kB present:3251200kB pages_sca
  2445. nned:51013799 all_unreclaimable? yes
  2446. Sep 27 04:36:50 monitor kernel: lowmem_reserve[]: 0 0 0 0
  2447. Sep 27 04:36:50 monitor kernel: DMA: 79*4kB 71*8kB 63*16kB 45*32kB 23*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10948kB
  2448. Sep 27 04:36:50 monitor kernel: Normal: 2197*4kB 3834*8kB 1253*16kB 300*32kB 117*64kB 81*128kB 47*256kB 8*512kB 0*1024kB 1*2048kB 0*4096kB = 105140
  2449. kB
  2450. Sep 27 04:36:50 monitor kernel: HighMem: 30*4kB 19*8kB 1*16kB 0*32kB 2*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 416kB
  2451. Sep 27 04:36:50 monitor kernel: Swap cache: add 5916441, delete 5916457, find 1542537405/1542665904, race 145+83
  2452. Sep 27 04:36:50 monitor kernel: Free swap = 0kB
  2453. Sep 27 04:36:50 monitor kernel: Total swap = 3911816kB
  2454. Sep 27 04:36:50 monitor kernel: Out of memory: kill process 12706 (postgres) score 9323 or a child
  2455. Sep 27 04:36:50 monitor kernel: Killed process 12706 (postgres)
  2456. Sep 27 08:00:24 monitor kernel: pbx_log.pl invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  2457. Sep 27 08:00:24 monitor kernel: Pid: 23102, comm: pbx_log.pl Not tainted 2.6.24.3 #3
  2458. Sep 27 08:00:24 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  2459. Sep 27 08:00:24 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  2460. Sep 27 08:00:24 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  2461. Sep 27 08:00:24 monitor kernel: [<c011abaa>] enqueue_task_fair+0x1a/0x30
  2462. Sep 27 08:00:24 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  2463. Sep 27 08:00:24 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  2464. Sep 27 08:00:24 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  2465. Sep 27 08:00:24 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  2466. Sep 27 08:00:24 monitor kernel: [<c0239281>] rb_insert_color+0x91/0xc0
  2467. Sep 27 08:00:24 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  2468. Sep 27 08:00:24 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  2469. Sep 27 08:00:24 monitor kernel: [<c0138540>] hrtimer_wakeup+0x0/0x20
  2470. Sep 27 08:00:24 monitor kernel: [<c0138a38>] sys_nanosleep+0x58/0x60
  2471. Sep 27 08:00:24 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  2472. Sep 27 08:00:24 monitor kernel: [<c043a202>] error_code+0x72/0x78
  2473. Sep 27 08:00:24 monitor kernel: [<c0430000>] rpcb_getport_sync+0x0/0x190
  2474. Sep 27 08:00:24 monitor kernel: =======================
  2475. Sep 27 08:00:24 monitor kernel: DMA per-cpu:
  2476. Sep 27 08:00:24 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2477. Sep 27 08:00:24 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2478. Sep 27 08:00:24 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2479. Sep 27 08:00:24 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2480. Sep 27 08:00:24 monitor kernel: Normal per-cpu:
  2481. Sep 27 08:00:24 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 89 Cold: hi: 62, btch: 15 usd: 61
  2482. Sep 27 08:00:24 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 44 Cold: hi: 62, btch: 15 usd: 46
  2483. Sep 27 08:00:24 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 37 Cold: hi: 62, btch: 15 usd: 48
  2484. Sep 27 08:00:24 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 41 Cold: hi: 62, btch: 15 usd: 59
  2485. Sep 27 08:00:24 monitor kernel: HighMem per-cpu:
  2486. Sep 27 08:00:24 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 19 Cold: hi: 62, btch: 15 usd: 32
  2487. Sep 27 08:00:24 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 27 Cold: hi: 62, btch: 15 usd: 48
  2488. Sep 27 08:00:24 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 12 Cold: hi: 62, btch: 15 usd: 55
  2489. Sep 27 08:00:24 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 47 Cold: hi: 62, btch: 15 usd: 58
  2490. Sep 27 08:00:24 monitor kernel: Active:458709 inactive:402269 dirty:0 writeback:0 unstable:0
  2491. Sep 27 08:00:24 monitor kernel: free:29193 slab:4534 mapped:22 pagetables:4539 bounce:0
  2492. Sep 27 08:00:24 monitor kernel: DMA free:10984kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  2493. able? yes
  2494. Sep 27 08:00:24 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  2495. Sep 27 08:00:24 monitor kernel: Normal free:105284kB min:3744kB low:4680kB high:5616kB active:355684kB inactive:352928kB present:894080kB pages_sca
  2496. nned:1128145 all_unreclaimable? yes
  2497. Sep 27 08:00:24 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  2498. Sep 27 08:00:24 monitor kernel: HighMem free:504kB min:512kB low:3916kB high:7324kB active:1479152kB inactive:1256148kB present:3251200kB pages_sca
  2499. nned:7073331 all_unreclaimable? yes
  2500. Sep 27 08:00:24 monitor kernel: lowmem_reserve[]: 0 0 0 0
  2501. Sep 27 08:00:24 monitor kernel: DMA: 74*4kB 68*8kB 62*16kB 44*32kB 25*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10984kB
  2502. Sep 27 08:00:24 monitor kernel: Normal: 1163*4kB 2115*8kB 1426*16kB 411*32kB 104*64kB 41*128kB 32*256kB 16*512kB 9*1024kB 5*2048kB 0*4096kB = 10528
  2503. 4kB
  2504. Sep 27 08:00:24 monitor kernel: HighMem: 36*4kB 9*8kB 10*16kB 2*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 504kB
  2505. Sep 27 08:00:24 monitor kernel: Swap cache: add 7706287, delete 7706303, find 1543179890/1543332076, race 145+174
  2506. Sep 27 08:00:24 monitor kernel: Free swap = 0kB
  2507. Sep 27 08:00:24 monitor kernel: Total swap = 3911816kB
  2508. Sep 27 08:00:24 monitor kernel: Out of memory: kill process 23265 (pbx_log.pl) score 15666 or a child
  2509. Sep 27 08:00:24 monitor kernel: Killed process 23265 (pbx_log.pl)
  2510. Sep 27 08:00:24 monitor kernel: mysqld invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  2511. Sep 27 08:00:24 monitor kernel: Pid: 27279, comm: mysqld Not tainted 2.6.24.3 #3
  2512. Sep 27 08:00:24 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  2513. Sep 27 08:00:24 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  2514. Sep 27 08:00:24 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  2515. Sep 27 08:00:24 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  2516. Sep 27 08:00:24 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  2517. Sep 27 08:00:24 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  2518. Sep 27 08:00:24 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  2519. Sep 27 08:00:24 monitor kernel: [<c022cd28>] blk_run_queue+0x58/0x70
  2520. Sep 27 08:00:24 monitor kernel: [<c02f62fb>] scsi_end_request+0xab/0xe0
  2521. Sep 27 08:00:24 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  2522. Sep 27 08:00:24 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  2523. Sep 27 08:00:24 monitor kernel: [<c017fe72>] sys_select+0x182/0x1b0
  2524. Sep 27 08:00:24 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  2525. Sep 27 08:00:24 monitor kernel: [<c043a202>] error_code+0x72/0x78
  2526. Sep 27 08:00:24 monitor kernel: [<c0430000>] rpcb_getport_sync+0x0/0x190
  2527. Sep 27 08:00:24 monitor kernel: =======================
  2528. Sep 27 08:00:24 monitor kernel: DMA per-cpu:
  2529. Sep 27 08:00:24 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2530. Sep 27 08:00:24 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2531. Sep 27 08:00:24 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2532. Sep 27 08:00:24 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2533. Sep 27 08:00:24 monitor kernel: Normal per-cpu:
  2534. Sep 27 08:00:24 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 89 Cold: hi: 62, btch: 15 usd: 61
  2535. Sep 27 08:00:24 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 44 Cold: hi: 62, btch: 15 usd: 46
  2536. Sep 27 08:00:24 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 37 Cold: hi: 62, btch: 15 usd: 48
  2537. Sep 27 08:00:24 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 41 Cold: hi: 62, btch: 15 usd: 61
  2538. Sep 27 08:00:24 monitor kernel: HighMem per-cpu:
  2539. Sep 27 08:00:24 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 19 Cold: hi: 62, btch: 15 usd: 32
  2540. Sep 27 08:00:24 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 27 Cold: hi: 62, btch: 15 usd: 48
  2541. Sep 27 08:00:24 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 12 Cold: hi: 62, btch: 15 usd: 55
  2542. Sep 27 08:00:24 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 47 Cold: hi: 62, btch: 15 usd: 36
  2543. Sep 27 08:00:24 monitor kernel: Active:507789 inactive:354976 dirty:0 writeback:0 unstable:0
  2544. Sep 27 08:00:24 monitor kernel: free:29193 slab:4534 mapped:20 pagetables:4539 bounce:0
  2545. Sep 27 08:00:24 monitor kernel: DMA free:10984kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  2546. able? yes
  2547. Sep 27 08:00:24 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  2548. Sep 27 08:00:24 monitor kernel: Normal free:105284kB min:3744kB low:4680kB high:5616kB active:446440kB inactive:262320kB present:894080kB pages_sca
  2549. nned:3558401 all_unreclaimable? yes
  2550. Sep 27 08:00:24 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  2551. Sep 27 08:00:24 monitor kernel: HighMem free:504kB min:512kB low:3916kB high:7324kB active:1584716kB inactive:1157584kB present:3251200kB pages_sca
  2552. nned:22331931 all_unreclaimable? yes
  2553. Sep 27 08:00:24 monitor kernel: lowmem_reserve[]: 0 0 0 0
  2554. Sep 27 08:00:24 monitor kernel: DMA: 74*4kB 68*8kB 62*16kB 44*32kB 25*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10984kB
  2555. Sep 27 08:00:24 monitor kernel: Normal: 1163*4kB 2115*8kB 1426*16kB 411*32kB 104*64kB 41*128kB 32*256kB 16*512kB 9*1024kB 5*2048kB 0*4096kB = 10528
  2556. 4kB
  2557. Sep 27 08:00:24 monitor kernel: HighMem: 36*4kB 9*8kB 10*16kB 2*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 504kB
  2558. Sep 27 08:00:24 monitor kernel: Swap cache: add 7706287, delete 7706303, find 1543179890/1543332076, race 145+174
  2559. Sep 27 08:00:24 monitor kernel: Free swap = 0kB
  2560. Sep 27 08:00:24 monitor kernel: Total swap = 3911816kB
  2561. Sep 27 08:00:24 monitor kernel: Out of memory: kill process 23266 (pbx_log.pl) score 15666 or a child
  2562. Sep 27 08:00:24 monitor kernel: Killed process 23266 (pbx_log.pl)
  2563. Sep 27 08:08:55 monitor kernel: pbx_log.pl invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  2564. Sep 27 08:08:55 monitor kernel: Pid: 23233, comm: pbx_log.pl Not tainted 2.6.24.3 #3
  2565. Sep 27 08:08:55 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  2566. Sep 27 08:08:55 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  2567. Sep 27 08:08:55 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  2568. Sep 27 08:08:55 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  2569. Sep 27 08:08:55 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  2570. Sep 27 08:08:55 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  2571. Sep 27 08:08:55 monitor kernel: [<c040357f>] inet_stream_connect+0x4f/0x270
  2572. Sep 27 08:08:55 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  2573. Sep 27 08:08:55 monitor kernel: [<c0184390>] d_alloc+0x100/0x180
  2574. Sep 27 08:08:55 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  2575. Sep 27 08:08:55 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  2576. Sep 27 08:08:55 monitor kernel: [<c017fe72>] sys_select+0x182/0x1b0
  2577. Sep 27 08:08:55 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  2578. Sep 27 08:08:55 monitor kernel: [<c043a202>] error_code+0x72/0x78
  2579. Sep 27 08:08:55 monitor kernel: =======================
  2580. Sep 27 08:08:55 monitor kernel: DMA per-cpu:
  2581. Sep 27 08:08:55 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2582. Sep 27 08:08:55 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2583. Sep 27 08:08:55 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2584. Sep 27 08:08:55 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2585. Sep 27 08:08:55 monitor kernel: Normal per-cpu:
  2586. Sep 27 08:08:55 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 68 Cold: hi: 62, btch: 15 usd: 49
  2587. Sep 27 08:08:55 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 118 Cold: hi: 62, btch: 15 usd: 37
  2588. Sep 27 08:08:55 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 153 Cold: hi: 62, btch: 15 usd: 17
  2589. Sep 27 08:08:55 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 73 Cold: hi: 62, btch: 15 usd: 58
  2590. Sep 27 08:08:55 monitor kernel: HighMem per-cpu:
  2591. Sep 27 08:08:55 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 151 Cold: hi: 62, btch: 15 usd: 53
  2592. Sep 27 08:08:55 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 56 Cold: hi: 62, btch: 15 usd: 52
  2593. Sep 27 08:08:55 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 116 Cold: hi: 62, btch: 15 usd: 61
  2594. Sep 27 08:08:55 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 59 Cold: hi: 62, btch: 15 usd: 47
  2595. Sep 27 08:08:55 monitor kernel: Active:430808 inactive:428642 dirty:0 writeback:0 unstable:0
  2596. Sep 27 08:08:55 monitor kernel: free:29162 slab:4700 mapped:27 pagetables:4557 bounce:0
  2597. Sep 27 08:08:55 monitor kernel: DMA free:10984kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  2598. able? yes
  2599. Sep 27 08:08:55 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  2600. Sep 27 08:08:55 monitor kernel: Normal free:105192kB min:3744kB low:4680kB high:5616kB active:362196kB inactive:343604kB present:894080kB pages_sca
  2601. nned:1141179 all_unreclaimable? yes
  2602. Sep 27 08:08:55 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  2603. Sep 27 08:08:55 monitor kernel: HighMem free:472kB min:512kB low:3916kB high:7324kB active:1361036kB inactive:1370964kB present:3251200kB pages_sca
  2604. nned:4315746 all_unreclaimable? yes
  2605. Sep 27 08:08:55 monitor kernel: lowmem_reserve[]: 0 0 0 0
  2606. Sep 27 08:08:55 monitor kernel: DMA: 74*4kB 68*8kB 62*16kB 44*32kB 25*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10984kB
  2607. Sep 27 08:08:55 monitor kernel: Normal: 580*4kB 1991*8kB 1528*16kB 433*32kB 106*64kB 41*128kB 31*256kB 14*512kB 9*1024kB 4*2048kB 1*4096kB = 105192
  2608. kB
  2609. Sep 27 08:08:56 monitor kernel: HighMem: 40*4kB 17*8kB 3*16kB 2*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 472kB
  2610. Sep 27 08:08:56 monitor kernel: Swap cache: add 7770323, delete 7770339, find 1543211598/1543366289, race 145+175
  2611. Sep 27 08:08:56 monitor kernel: Free swap = 0kB
  2612. Sep 27 08:08:56 monitor kernel: Total swap = 3911816kB
  2613. Sep 27 08:08:56 monitor kernel: Out of memory: kill process 23057 (pbx_log.pl) score 14062 or a child
  2614. Sep 27 08:08:56 monitor kernel: Killed process 23057 (pbx_log.pl)
  2615. Sep 27 08:08:56 monitor kernel: pbx_log.pl invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  2616. Sep 27 08:08:56 monitor kernel: Pid: 23218, comm: pbx_log.pl Not tainted 2.6.24.3 #3
  2617. Sep 27 08:08:56 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  2618. Sep 27 08:08:56 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  2619. Sep 27 08:08:56 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  2620. Sep 27 08:08:56 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  2621. Sep 27 08:08:56 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  2622. Sep 27 08:08:56 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  2623. Sep 27 08:08:56 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  2624. Sep 27 08:08:56 monitor kernel: [<c0239281>] rb_insert_color+0x91/0xc0
  2625. Sep 27 08:08:56 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  2626. Sep 27 08:08:56 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  2627. Sep 27 08:08:56 monitor kernel: [<c0104cec>] invalidate_interrupt+0x28/0x30
  2628. Sep 27 08:08:56 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  2629. Sep 27 08:08:56 monitor kernel: [<c01200d8>] copy_process+0x378/0x1100
  2630. Sep 27 08:08:56 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  2631. Sep 27 08:08:56 monitor kernel: [<c043a202>] error_code+0x72/0x78
  2632. Sep 27 08:08:56 monitor kernel: [<c0430000>] rpcb_getport_sync+0x0/0x190
  2633. Sep 27 08:08:56 monitor kernel: =======================
  2634. Sep 27 08:08:56 monitor kernel: DMA per-cpu:
  2635. Sep 27 08:08:56 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2636. Sep 27 08:08:56 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2637. Sep 27 08:08:56 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2638. Sep 27 08:08:56 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2639. Sep 27 08:08:56 monitor kernel: Normal per-cpu:
  2640. Sep 27 08:08:56 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 68 Cold: hi: 62, btch: 15 usd: 51
  2641. Sep 27 08:08:56 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 121 Cold: hi: 62, btch: 15 usd: 51
  2642. Sep 27 08:08:56 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 150 Cold: hi: 62, btch: 15 usd: 48
  2643. Sep 27 08:08:56 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 71 Cold: hi: 62, btch: 15 usd: 51
  2644. Sep 27 08:08:56 monitor kernel: HighMem per-cpu:
  2645. Sep 27 08:08:56 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 151 Cold: hi: 62, btch: 15 usd: 60
  2646. Sep 27 08:08:56 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 56 Cold: hi: 62, btch: 15 usd: 50
  2647. Sep 27 08:08:56 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 116 Cold: hi: 62, btch: 15 usd: 32
  2648. Sep 27 08:08:56 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 59 Cold: hi: 62, btch: 15 usd: 18
  2649. Sep 27 08:08:56 monitor kernel: Active:490758 inactive:370644 dirty:0 writeback:0 unstable:0
  2650. Sep 27 08:08:56 monitor kernel: free:29177 slab:4702 mapped:8 pagetables:4557 bounce:0
  2651. Sep 27 08:08:56 monitor kernel: DMA free:10984kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  2652. able? yes
  2653. Sep 27 08:08:56 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  2654. Sep 27 08:08:56 monitor kernel: Normal free:105252kB min:3744kB low:4680kB high:5616kB active:443492kB inactive:262236kB present:894080kB pages_sca
  2655. nned:11748011 all_unreclaimable? yes
  2656. Sep 27 08:08:56 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  2657. Sep 27 08:08:56 monitor kernel: HighMem free:472kB min:512kB low:3916kB high:7324kB active:1519540kB inactive:1220340kB present:3251200kB pages_sca
  2658. nned:30432098 all_unreclaimable? yes
  2659. Sep 27 08:08:56 monitor kernel: lowmem_reserve[]: 0 0 0 0
  2660. Sep 27 08:08:56 monitor kernel: DMA: 74*4kB 68*8kB 62*16kB 44*32kB 25*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10984kB
  2661. Sep 27 08:08:56 monitor kernel: Normal: 593*4kB 1992*8kB 1528*16kB 433*32kB 106*64kB 41*128kB 31*256kB 14*512kB 9*1024kB 4*2048kB 1*4096kB = 105252
  2662. kB
  2663. Sep 27 08:08:56 monitor kernel: HighMem: 34*4kB 20*8kB 3*16kB 2*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 472kB
  2664. Sep 27 08:08:56 monitor kernel: Swap cache: add 7770323, delete 7770339, find 1543211598/1543366289, race 145+175
  2665. Sep 27 08:08:56 monitor kernel: Free swap = 0kB
  2666. Sep 27 08:08:56 monitor kernel: Total swap = 3911816kB
  2667. Sep 27 08:08:56 monitor kernel: Out of memory: kill process 23058 (pbx_log.pl) score 14062 or a child
  2668. Sep 27 08:08:56 monitor kernel: Killed process 23058 (pbx_log.pl)
  2669. Sep 27 08:16:01 monitor kernel: syslogd invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  2670. Sep 27 08:16:01 monitor kernel: Pid: 2182, comm: syslogd Not tainted 2.6.24.3 #3
  2671. Sep 27 08:16:01 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  2672. Sep 27 08:16:01 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  2673. Sep 27 08:16:01 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  2674. Sep 27 08:16:01 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  2675. Sep 27 08:16:01 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  2676. Sep 27 08:16:01 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  2677. Sep 27 08:16:01 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  2678. Sep 27 08:16:01 monitor kernel: [<c012c4bb>] recalc_sigpending+0xb/0x30
  2679. Sep 27 08:16:01 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  2680. Sep 27 08:16:01 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  2681. Sep 27 08:16:01 monitor kernel: [<c017fd3d>] sys_select+0x4d/0x1b0
  2682. Sep 27 08:16:01 monitor kernel: [<c012c545>] sigprocmask+0x65/0xf0
  2683. Sep 27 08:16:01 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  2684. Sep 27 08:16:01 monitor kernel: [<c043a202>] error_code+0x72/0x78
  2685. Sep 27 08:16:01 monitor kernel: =======================
  2686. Sep 27 08:16:01 monitor kernel: DMA per-cpu:
  2687. Sep 27 08:16:01 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2688. Sep 27 08:16:01 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2689. Sep 27 08:16:01 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2690. Sep 27 08:16:01 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2691. Sep 27 08:16:01 monitor kernel: Normal per-cpu:
  2692. Sep 27 08:16:01 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 82 Cold: hi: 62, btch: 15 usd: 58
  2693. Sep 27 08:16:01 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 44 Cold: hi: 62, btch: 15 usd: 57
  2694. Sep 27 08:16:01 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 144 Cold: hi: 62, btch: 15 usd: 61
  2695. Sep 27 08:16:01 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 112 Cold: hi: 62, btch: 15 usd: 61
  2696. Sep 27 08:16:01 monitor kernel: HighMem per-cpu:
  2697. Sep 27 08:16:01 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 71 Cold: hi: 62, btch: 15 usd: 53
  2698. Sep 27 08:16:01 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 56 Cold: hi: 62, btch: 15 usd: 54
  2699. Sep 27 08:16:01 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 14 Cold: hi: 62, btch: 15 usd: 51
  2700. Sep 27 08:16:01 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 71 Cold: hi: 62, btch: 15 usd: 61
  2701. Sep 27 08:16:01 monitor kernel: Active:435508 inactive:424575 dirty:0 writeback:0 unstable:0
  2702. Sep 27 08:16:01 monitor kernel: free:29149 slab:4689 mapped:21 pagetables:4534 bounce:0
  2703. Sep 27 08:16:01 monitor kernel: DMA free:10984kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  2704. able? yes
  2705. Sep 27 08:16:01 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  2706. Sep 27 08:16:01 monitor kernel: Normal free:105168kB min:3744kB low:4680kB high:5616kB active:362416kB inactive:343736kB present:894080kB pages_sca
  2707. nned:1192826 all_unreclaimable? yes
  2708. Sep 27 08:16:01 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  2709. Sep 27 08:16:01 monitor kernel: HighMem free:444kB min:512kB low:3916kB high:7324kB active:1379616kB inactive:1354564kB present:3251200kB pages_sca
  2710. nned:5926676 all_unreclaimable? yes
  2711. Sep 27 08:16:01 monitor kernel: lowmem_reserve[]: 0 0 0 0
  2712. Sep 27 08:16:01 monitor kernel: DMA: 74*4kB 68*8kB 62*16kB 44*32kB 25*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10984kB
  2713. Sep 27 08:16:01 monitor kernel: Normal: 610*4kB 1997*8kB 1446*16kB 452*32kB 112*64kB 40*128kB 30*256kB 15*512kB 9*1024kB 4*2048kB 1*4096kB = 105168
  2714. kB
  2715. Sep 27 08:16:01 monitor kernel: HighMem: 43*4kB 8*8kB 5*16kB 2*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 444kB
  2716. Sep 27 08:16:01 monitor kernel: Swap cache: add 7841304, delete 7841320, find 1543248111/1543406249, race 145+176
  2717. Sep 27 08:16:01 monitor kernel: Free swap = 0kB
  2718. Sep 27 08:16:01 monitor kernel: Total swap = 3911816kB
  2719. Sep 27 08:16:01 monitor kernel: Out of memory: kill process 23260 (pbx_log.pl) score 14062 or a child
  2720. Sep 27 08:16:01 monitor kernel: Killed process 23260 (pbx_log.pl)
  2721. Sep 27 08:16:01 monitor kernel: syslogd invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  2722. Sep 27 08:16:01 monitor kernel: Pid: 2182, comm: syslogd Not tainted 2.6.24.3 #3
  2723. Sep 27 08:16:01 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  2724. Sep 27 08:16:01 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  2725. Sep 27 08:16:01 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  2726. Sep 27 08:16:01 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  2727. Sep 27 08:16:01 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  2728. Sep 27 08:16:01 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  2729. Sep 27 08:16:01 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  2730. Sep 27 08:16:01 monitor kernel: [<c012c4bb>] recalc_sigpending+0xb/0x30
  2731. Sep 27 08:16:01 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  2732. Sep 27 08:16:01 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  2733. Sep 27 08:16:01 monitor kernel: [<c017fd3d>] sys_select+0x4d/0x1b0
  2734. Sep 27 08:16:01 monitor kernel: [<c012c545>] sigprocmask+0x65/0xf0
  2735. Sep 27 08:16:01 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  2736. Sep 27 08:16:01 monitor kernel: [<c043a202>] error_code+0x72/0x78
  2737. Sep 27 08:16:01 monitor kernel: =======================
  2738. Sep 27 08:16:01 monitor kernel: DMA per-cpu:
  2739. Sep 27 08:16:01 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2740. Sep 27 08:16:01 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2741. Sep 27 08:16:01 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2742. Sep 27 08:16:01 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2743. Sep 27 08:16:01 monitor kernel: Normal per-cpu:
  2744. Sep 27 08:16:01 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 82 Cold: hi: 62, btch: 15 usd: 58
  2745. Sep 27 08:16:01 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 44 Cold: hi: 62, btch: 15 usd: 57
  2746. Sep 27 08:16:01 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 144 Cold: hi: 62, btch: 15 usd: 61
  2747. Sep 27 08:16:01 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 112 Cold: hi: 62, btch: 15 usd: 61
  2748. Sep 27 08:16:01 monitor kernel: HighMem per-cpu:
  2749. Sep 27 08:16:01 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 71 Cold: hi: 62, btch: 15 usd: 53
  2750. Sep 27 08:16:01 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 56 Cold: hi: 62, btch: 15 usd: 54
  2751. Sep 27 08:16:01 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 14 Cold: hi: 62, btch: 15 usd: 51
  2752. Sep 27 08:16:01 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 71 Cold: hi: 62, btch: 15 usd: 61
  2753. Sep 27 08:16:01 monitor kernel: Active:435568 inactive:424497 dirty:0 writeback:0 unstable:0
  2754. Sep 27 08:16:01 monitor kernel: free:29149 slab:4689 mapped:21 pagetables:4534 bounce:0
  2755. Sep 27 08:16:01 monitor kernel: DMA free:10984kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  2756. able? yes
  2757. Sep 27 08:16:01 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  2758. Sep 27 08:16:01 monitor kernel: Normal free:105168kB min:3744kB low:4680kB high:5616kB active:362584kB inactive:343592kB present:894080kB pages_sca
  2759. nned:1193046 all_unreclaimable? yes
  2760. Sep 27 08:16:01 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  2761. Sep 27 08:16:01 monitor kernel: HighMem free:444kB min:512kB low:3916kB high:7324kB active:1379688kB inactive:1354396kB present:3251200kB pages_sca
  2762. nned:5981064 all_unreclaimable? yes
  2763. Sep 27 08:16:01 monitor kernel: lowmem_reserve[]: 0 0 0 0
  2764. Sep 27 08:16:01 monitor kernel: DMA: 74*4kB 68*8kB 62*16kB 44*32kB 25*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10984kB
  2765. Sep 27 08:16:01 monitor kernel: Normal: 610*4kB 1997*8kB 1446*16kB 452*32kB 112*64kB 40*128kB 30*256kB 15*512kB 9*1024kB 4*2048kB 1*4096kB = 105168
  2766. kB
  2767. Sep 27 08:16:01 monitor kernel: HighMem: 43*4kB 8*8kB 5*16kB 2*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 444kB
  2768. Sep 27 08:16:01 monitor kernel: Swap cache: add 7841304, delete 7841320, find 1543248111/1543406249, race 145+176
  2769. Sep 27 08:16:01 monitor kernel: Free swap = 0kB
  2770. Sep 27 08:16:01 monitor kernel: Total swap = 3911816kB
  2771. Sep 27 08:16:01 monitor kernel: Out of memory: kill process 23263 (pbx_log.pl) score 14062 or a child
  2772. Sep 27 08:16:01 monitor kernel: Killed process 23263 (pbx_log.pl)
  2773. Sep 27 08:23:53 monitor kernel: pbx_log.pl invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  2774. Sep 27 08:23:53 monitor kernel: Pid: 23256, comm: pbx_log.pl Not tainted 2.6.24.3 #3
  2775. Sep 27 08:23:53 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  2776. Sep 27 08:23:53 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  2777. Sep 27 08:23:53 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  2778. Sep 27 08:23:53 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  2779. Sep 27 08:23:53 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  2780. Sep 27 08:23:53 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  2781. Sep 27 08:23:53 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  2782. Sep 27 08:23:53 monitor kernel: [<c0239281>] rb_insert_color+0x91/0xc0
  2783. Sep 27 08:23:53 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  2784. Sep 27 08:23:53 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  2785. Sep 27 08:23:53 monitor kernel: [<c0138540>] hrtimer_wakeup+0x0/0x20
  2786. Sep 27 08:23:53 monitor kernel: [<c0138a38>] sys_nanosleep+0x58/0x60
  2787. Sep 27 08:23:53 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  2788. Sep 27 08:23:53 monitor kernel: [<c043a202>] error_code+0x72/0x78
  2789. Sep 27 08:23:53 monitor kernel: [<c0430000>] rpcb_getport_sync+0x0/0x190
  2790. Sep 27 08:23:53 monitor kernel: =======================
  2791. Sep 27 08:23:53 monitor kernel: DMA per-cpu:
  2792. Sep 27 08:23:53 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2793. Sep 27 08:23:53 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2794. Sep 27 08:23:53 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2795. Sep 27 08:23:53 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2796. Sep 27 08:23:53 monitor kernel: Normal per-cpu:
  2797. Sep 27 08:23:53 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 29 Cold: hi: 62, btch: 15 usd: 41
  2798. Sep 27 08:23:53 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 131 Cold: hi: 62, btch: 15 usd: 45
  2799. Sep 27 08:23:53 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 70 Cold: hi: 62, btch: 15 usd: 30
  2800. Sep 27 08:23:53 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 108 Cold: hi: 62, btch: 15 usd: 54
  2801. Sep 27 08:23:53 monitor kernel: HighMem per-cpu:
  2802. Sep 27 08:23:53 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 30 Cold: hi: 62, btch: 15 usd: 59
  2803. Sep 27 08:23:53 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 19 Cold: hi: 62, btch: 15 usd: 34
  2804. Sep 27 08:23:53 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 49 Cold: hi: 62, btch: 15 usd: 50
  2805. Sep 27 08:23:53 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 133 Cold: hi: 62, btch: 15 usd: 35
  2806. Sep 27 08:23:53 monitor kernel: Active:440783 inactive:419193 dirty:0 writeback:0 unstable:0
  2807. Sep 27 08:23:53 monitor kernel: free:29185 slab:4635 mapped:8 pagetables:4533 bounce:0
  2808. Sep 27 08:23:53 monitor kernel: DMA free:11028kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  2809. able? yes
  2810. Sep 27 08:23:53 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  2811. Sep 27 08:23:53 monitor kernel: Normal free:105340kB min:3744kB low:4680kB high:5616kB active:353760kB inactive:351368kB present:894080kB pages_sca
  2812. nned:1997071 all_unreclaimable? yes
  2813. Sep 27 08:23:53 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  2814. Sep 27 08:23:53 monitor kernel: HighMem free:372kB min:512kB low:3916kB high:7324kB active:1409372kB inactive:1325404kB present:3251200kB pages_sca
  2815. nned:7290306 all_unreclaimable? yes
  2816. Sep 27 08:23:53 monitor kernel: lowmem_reserve[]: 0 0 0 0
  2817. Sep 27 08:23:53 monitor kernel: DMA: 65*4kB 64*8kB 59*16kB 45*32kB 27*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 11028kB
  2818. Sep 27 08:23:53 monitor kernel: Normal: 605*4kB 1877*8kB 1416*16kB 463*32kB 132*64kB 40*128kB 30*256kB 15*512kB 9*1024kB 4*2048kB 1*4096kB = 105340
  2819. kB
  2820. Sep 27 08:23:53 monitor kernel: HighMem: 21*4kB 10*8kB 5*16kB 2*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 372kB
  2821. Sep 27 08:23:53 monitor kernel: Swap cache: add 7909858, delete 7909875, find 1543281278/1543442398, race 145+177
  2822. Sep 27 08:23:53 monitor kernel: Free swap = 0kB
  2823. Sep 27 08:23:53 monitor kernel: Total swap = 3911816kB
  2824. Sep 27 08:23:53 monitor kernel: Out of memory: kill process 23244 (pbx_log.pl) score 15132 or a child
  2825. Sep 27 08:23:53 monitor kernel: Killed process 23244 (pbx_log.pl)
  2826. Sep 27 08:23:53 monitor kernel: pbx_log.pl invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  2827. Sep 27 08:23:53 monitor kernel: Pid: 23256, comm: pbx_log.pl Not tainted 2.6.24.3 #3
  2828. Sep 27 08:23:53 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  2829. Sep 27 08:23:53 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  2830. Sep 27 08:23:53 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  2831. Sep 27 08:23:53 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  2832. Sep 27 08:23:53 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  2833. Sep 27 08:23:53 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  2834. Sep 27 08:23:53 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  2835. Sep 27 08:23:53 monitor kernel: [<c0239281>] rb_insert_color+0x91/0xc0
  2836. Sep 27 08:23:53 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  2837. Sep 27 08:23:53 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  2838. Sep 27 08:23:53 monitor kernel: [<c0138540>] hrtimer_wakeup+0x0/0x20
  2839. Sep 27 08:23:53 monitor kernel: [<c0138a38>] sys_nanosleep+0x58/0x60
  2840. Sep 27 08:23:53 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  2841. Sep 27 08:23:53 monitor kernel: [<c043a202>] error_code+0x72/0x78
  2842. Sep 27 08:23:53 monitor kernel: [<c0430000>] rpcb_getport_sync+0x0/0x190
  2843. Sep 27 08:23:53 monitor kernel: =======================
  2844. Sep 27 08:23:53 monitor kernel: DMA per-cpu:
  2845. Sep 27 08:23:53 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2846. Sep 27 08:23:53 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2847. Sep 27 08:23:53 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2848. Sep 27 08:23:53 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2849. Sep 27 08:23:53 monitor kernel: Normal per-cpu:
  2850. Sep 27 08:23:53 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 29 Cold: hi: 62, btch: 15 usd: 41
  2851. Sep 27 08:23:53 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 131 Cold: hi: 62, btch: 15 usd: 45
  2852. Sep 27 08:23:53 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 70 Cold: hi: 62, btch: 15 usd: 30
  2853. Sep 27 08:23:53 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 108 Cold: hi: 62, btch: 15 usd: 54
  2854. Sep 27 08:23:53 monitor kernel: HighMem per-cpu:
  2855. Sep 27 08:23:53 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 30 Cold: hi: 62, btch: 15 usd: 59
  2856. Sep 27 08:23:53 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 19 Cold: hi: 62, btch: 15 usd: 34
  2857. Sep 27 08:23:53 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 49 Cold: hi: 62, btch: 15 usd: 50
  2858. Sep 27 08:23:53 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 133 Cold: hi: 62, btch: 15 usd: 28
  2859. Sep 27 08:23:53 monitor kernel: Active:440869 inactive:419372 dirty:0 writeback:0 unstable:0
  2860. Sep 27 08:23:53 monitor kernel: free:29185 slab:4635 mapped:8 pagetables:4533 bounce:0
  2861. Sep 27 08:23:53 monitor kernel: DMA free:11028kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  2862. able? yes
  2863. Sep 27 08:23:53 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  2864. Sep 27 08:23:53 monitor kernel: Normal free:105340kB min:3744kB low:4680kB high:5616kB active:359396kB inactive:345772kB present:894080kB pages_sca
  2865. nned:2004739 all_unreclaimable? yes
  2866. Sep 27 08:23:53 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  2867. Sep 27 08:23:53 monitor kernel: HighMem free:372kB min:512kB low:3916kB high:7324kB active:1404080kB inactive:1331716kB present:3251200kB pages_sca
  2868. nned:7972910 all_unreclaimable? yes
  2869. Sep 27 08:23:53 monitor kernel: lowmem_reserve[]: 0 0 0 0
  2870. Sep 27 08:23:53 monitor kernel: DMA: 65*4kB 64*8kB 59*16kB 45*32kB 27*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 11028kB
  2871. Sep 27 08:23:53 monitor kernel: Normal: 605*4kB 1877*8kB 1416*16kB 463*32kB 132*64kB 40*128kB 30*256kB 15*512kB 9*1024kB 4*2048kB 1*4096kB = 105340
  2872. kB
  2873. Sep 27 08:23:53 monitor kernel: HighMem: 21*4kB 10*8kB 5*16kB 2*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 372kB
  2874. Sep 27 08:23:53 monitor kernel: Swap cache: add 7909858, delete 7909875, find 1543281278/1543442398, race 145+177
  2875. Sep 27 08:23:53 monitor kernel: Free swap = 0kB
  2876. Sep 27 08:23:53 monitor kernel: Total swap = 3911816kB
  2877. Sep 27 08:23:53 monitor kernel: Out of memory: kill process 23245 (pbx_log.pl) score 15132 or a child
  2878. Sep 27 08:23:53 monitor kernel: Killed process 23245 (pbx_log.pl)
  2879. Sep 27 08:31:00 monitor kernel: postgres invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  2880. Sep 27 08:31:00 monitor kernel: Pid: 19617, comm: postgres Not tainted 2.6.24.3 #3
  2881. Sep 27 08:31:00 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  2882. Sep 27 08:31:00 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  2883. Sep 27 08:31:00 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  2884. Sep 27 08:31:00 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  2885. Sep 27 08:31:00 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  2886. Sep 27 08:31:00 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  2887. Sep 27 08:31:00 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  2888. Sep 27 08:31:00 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  2889. Sep 27 08:31:00 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  2890. Sep 27 08:31:00 monitor kernel: [<c0126af8>] sys_gettimeofday+0x28/0x80
  2891. Sep 27 08:31:00 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  2892. Sep 27 08:31:00 monitor kernel: [<c043a202>] error_code+0x72/0x78
  2893. Sep 27 08:31:00 monitor kernel: [<c0430000>] rpcb_getport_sync+0x0/0x190
  2894. Sep 27 08:31:00 monitor kernel: =======================
  2895. Sep 27 08:31:00 monitor kernel: DMA per-cpu:
  2896. Sep 27 08:31:00 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2897. Sep 27 08:31:00 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2898. Sep 27 08:31:00 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2899. Sep 27 08:31:00 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2900. Sep 27 08:31:00 monitor kernel: Normal per-cpu:
  2901. Sep 27 08:31:00 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 134 Cold: hi: 62, btch: 15 usd: 61
  2902. Sep 27 08:31:00 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 88 Cold: hi: 62, btch: 15 usd: 58
  2903. Sep 27 08:31:00 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 50 Cold: hi: 62, btch: 15 usd: 54
  2904. Sep 27 08:31:00 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 58 Cold: hi: 62, btch: 15 usd: 58
  2905. Sep 27 08:31:00 monitor kernel: HighMem per-cpu:
  2906. Sep 27 08:31:00 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 59 Cold: hi: 62, btch: 15 usd: 60
  2907. Sep 27 08:31:00 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 67 Cold: hi: 62, btch: 15 usd: 59
  2908. Sep 27 08:31:00 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 16 Cold: hi: 62, btch: 15 usd: 30
  2909. Sep 27 08:31:00 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 31 Cold: hi: 62, btch: 15 usd: 51
  2910. Sep 27 08:31:00 monitor kernel: Active:427920 inactive:432458 dirty:0 writeback:0 unstable:0
  2911. Sep 27 08:31:00 monitor kernel: free:29137 slab:4533 mapped:20 pagetables:4533 bounce:0
  2912. Sep 27 08:31:00 monitor kernel: DMA free:11028kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  2913. able? yes
  2914. Sep 27 08:31:00 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  2915. Sep 27 08:31:00 monitor kernel: Normal free:105048kB min:3744kB low:4680kB high:5616kB active:360072kB inactive:346464kB present:894080kB pages_sca
  2916. nned:1142710 all_unreclaimable? yes
  2917. Sep 27 08:31:00 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  2918. Sep 27 08:31:00 monitor kernel: HighMem free:472kB min:512kB low:3916kB high:7324kB active:1351608kB inactive:1383368kB present:3251200kB pages_sca
  2919. nned:7465314 all_unreclaimable? yes
  2920. Sep 27 08:31:00 monitor kernel: lowmem_reserve[]: 0 0 0 0
  2921. Sep 27 08:31:00 monitor kernel: DMA: 65*4kB 64*8kB 59*16kB 45*32kB 27*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 11028kB
  2922. Sep 27 08:31:00 monitor kernel: Normal: 912*4kB 1891*8kB 1416*16kB 408*32kB 132*64kB 39*128kB 29*256kB 14*512kB 10*1024kB 4*2048kB 1*4096kB = 10504
  2923. 8kB
  2924. Sep 27 08:31:00 monitor kernel: HighMem: 50*4kB 12*8kB 3*16kB 2*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 472kB
  2925. Sep 27 08:31:00 monitor kernel: Swap cache: add 7960237, delete 7960254, find 1543314211/1543477359, race 145+177
  2926. Sep 27 08:31:00 monitor kernel: Free swap = 0kB
  2927. Sep 27 08:31:00 monitor kernel: Total swap = 3911816kB
  2928. Sep 27 08:31:00 monitor kernel: Out of memory: kill process 23063 (pbx_log.pl) score 15488 or a child
  2929. Sep 27 08:31:00 monitor kernel: Killed process 23063 (pbx_log.pl)
  2930. Sep 27 08:31:00 monitor kernel: pbx_log.pl invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  2931. Sep 27 08:31:00 monitor kernel: Pid: 23134, comm: pbx_log.pl Not tainted 2.6.24.3 #3
  2932. Sep 27 08:31:00 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  2933. Sep 27 08:31:00 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  2934. Sep 27 08:31:00 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  2935. Sep 27 08:31:00 monitor kernel: [<c0119882>] enqueue_task+0x12/0x30
  2936. Sep 27 08:31:00 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  2937. Sep 27 08:31:00 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  2938. Sep 27 08:31:00 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  2939. Sep 27 08:31:00 monitor kernel: [<c040357f>] inet_stream_connect+0x4f/0x270
  2940. Sep 27 08:31:00 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  2941. Sep 27 08:31:00 monitor kernel: [<c0184390>] d_alloc+0x100/0x180
  2942. Sep 27 08:31:00 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  2943. Sep 27 08:31:00 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  2944. Sep 27 08:31:00 monitor kernel: [<c017fe72>] sys_select+0x182/0x1b0
  2945. Sep 27 08:31:00 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  2946. Sep 27 08:31:00 monitor kernel: [<c043a202>] error_code+0x72/0x78
  2947. Sep 27 08:31:00 monitor kernel: =======================
  2948. Sep 27 08:31:00 monitor kernel: DMA per-cpu:
  2949. Sep 27 08:31:00 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2950. Sep 27 08:31:00 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2951. Sep 27 08:31:00 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2952. Sep 27 08:31:00 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  2953. Sep 27 08:31:00 monitor kernel: Normal per-cpu:
  2954. Sep 27 08:31:00 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 134 Cold: hi: 62, btch: 15 usd: 61
  2955. Sep 27 08:31:00 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 88 Cold: hi: 62, btch: 15 usd: 58
  2956. Sep 27 08:31:00 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 50 Cold: hi: 62, btch: 15 usd: 56
  2957. Sep 27 08:31:00 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 58 Cold: hi: 62, btch: 15 usd: 59
  2958. Sep 27 08:31:00 monitor kernel: HighMem per-cpu:
  2959. Sep 27 08:31:00 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 59 Cold: hi: 62, btch: 15 usd: 60
  2960. Sep 27 08:31:00 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 67 Cold: hi: 62, btch: 15 usd: 59
  2961. Sep 27 08:31:00 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 16 Cold: hi: 62, btch: 15 usd: 30
  2962. Sep 27 08:31:00 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 31 Cold: hi: 62, btch: 15 usd: 19
  2963. Sep 27 08:31:00 monitor kernel: Active:467063 inactive:394087 dirty:0 writeback:0 unstable:0
  2964. Sep 27 08:31:00 monitor kernel: free:29137 slab:4533 mapped:20 pagetables:4533 bounce:0
  2965. Sep 27 08:31:00 monitor kernel: DMA free:11028kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  2966. able? yes
  2967. Sep 27 08:31:00 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  2968. Sep 27 08:31:00 monitor kernel: Normal free:105048kB min:3744kB low:4680kB high:5616kB active:444200kB inactive:262224kB present:894080kB pages_sca
  2969. nned:1353837 all_unreclaimable? yes
  2970. Sep 27 08:31:00 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  2971. Sep 27 08:31:00 monitor kernel: HighMem free:472kB min:512kB low:3916kB high:7324kB active:1424052kB inactive:1314236kB present:3251200kB pages_sca
  2972. nned:12188195 all_unreclaimable? yes
  2973. Sep 27 08:31:00 monitor kernel: lowmem_reserve[]: 0 0 0 0
  2974. Sep 27 08:31:00 monitor kernel: DMA: 65*4kB 64*8kB 59*16kB 45*32kB 27*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 11028kB
  2975. Sep 27 08:31:00 monitor kernel: Normal: 912*4kB 1891*8kB 1416*16kB 408*32kB 132*64kB 39*128kB 29*256kB 14*512kB 10*1024kB 4*2048kB 1*4096kB = 10504
  2976. 8kB
  2977. Sep 27 08:31:00 monitor kernel: HighMem: 50*4kB 12*8kB 3*16kB 2*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 472kB
  2978. Sep 27 08:31:00 monitor kernel: Swap cache: add 7960237, delete 7960254, find 1543314211/1543477359, race 145+177
  2979. Sep 27 08:31:00 monitor kernel: Free swap = 0kB
  2980. Sep 27 08:31:00 monitor kernel: Total swap = 3911816kB
  2981. Sep 27 08:31:00 monitor kernel: Out of memory: kill process 23064 (pbx_log.pl) score 15488 or a child
  2982. Sep 27 08:31:00 monitor kernel: Killed process 23064 (pbx_log.pl)
  2983. Sep 27 08:31:00 monitor kernel: pbx_log.pl invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  2984. Sep 27 08:31:00 monitor kernel: Pid: 23212, comm: pbx_log.pl Not tainted 2.6.24.3 #3
  2985. Sep 27 08:31:00 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  2986. Sep 27 08:31:00 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  2987. Sep 27 08:31:00 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  2988. Sep 27 08:31:00 monitor kernel: [<c011abaa>] enqueue_task_fair+0x1a/0x30
  2989. Sep 27 08:31:00 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  2990. Sep 27 08:31:00 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  2991. Sep 27 08:31:00 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  2992. Sep 27 08:31:00 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  2993. Sep 27 08:31:00 monitor kernel: [<c0239281>] rb_insert_color+0x91/0xc0
  2994. Sep 27 08:31:00 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  2995. Sep 27 08:31:00 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  2996. Sep 27 08:31:00 monitor kernel: [<c0126af8>] sys_gettimeofday+0x28/0x80
  2997. Sep 27 08:31:00 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  2998. Sep 27 08:31:00 monitor kernel: [<c043a202>] error_code+0x72/0x78
  2999. Sep 27 08:31:00 monitor kernel: [<c0430000>] rpcb_getport_sync+0x0/0x190
  3000. Sep 27 08:31:00 monitor kernel: =======================
  3001. Sep 27 08:31:00 monitor kernel: DMA per-cpu:
  3002. Sep 27 08:31:00 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  3003. Sep 27 08:31:00 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  3004. Sep 27 08:31:00 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  3005. Sep 27 08:31:00 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  3006. Sep 27 08:31:00 monitor kernel: Normal per-cpu:
  3007. Sep 27 08:31:00 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 131 Cold: hi: 62, btch: 15 usd: 48
  3008. Sep 27 08:31:00 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 88 Cold: hi: 62, btch: 15 usd: 60
  3009. Sep 27 08:31:00 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 50 Cold: hi: 62, btch: 15 usd: 56
  3010. Sep 27 08:31:00 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 58 Cold: hi: 62, btch: 15 usd: 59
  3011. Sep 27 08:31:00 monitor kernel: HighMem per-cpu:
  3012. Sep 27 08:31:00 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 59 Cold: hi: 62, btch: 15 usd: 53
  3013. Sep 27 08:31:00 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 67 Cold: hi: 62, btch: 15 usd: 53
  3014. Sep 27 08:31:00 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 16 Cold: hi: 62, btch: 15 usd: 52
  3015. Sep 27 08:31:00 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 179 Cold: hi: 62, btch: 15 usd: 29
  3016. Sep 27 08:31:00 monitor kernel: Active:500115 inactive:362085 dirty:0 writeback:0 unstable:0
  3017. Sep 27 08:31:00 monitor kernel: free:29176 slab:4535 mapped:20 pagetables:4533 bounce:0
  3018. Sep 27 08:31:00 monitor kernel: DMA free:11028kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  3019. able? yes
  3020. Sep 27 08:31:00 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  3021. Sep 27 08:31:00 monitor kernel: Normal free:105116kB min:3744kB low:4680kB high:5616kB active:444292kB inactive:262080kB present:894080kB pages_sca
  3022. nned:11560554 all_unreclaimable? yes
  3023. Sep 27 08:31:00 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  3024. Sep 27 08:31:00 monitor kernel: HighMem free:856kB min:512kB low:3916kB high:7324kB active:1555948kB inactive:1186260kB present:3251200kB pages_sca
  3025. nned:0 all_unreclaimable? no
  3026. Sep 27 08:31:00 monitor kernel: lowmem_reserve[]: 0 0 0 0
  3027. Sep 27 08:31:00 monitor kernel: DMA: 65*4kB 64*8kB 59*16kB 45*32kB 27*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 11028kB
  3028. Sep 27 08:31:00 monitor kernel: Normal: 927*4kB 1892*8kB 1416*16kB 408*32kB 132*64kB 39*128kB 29*256kB 14*512kB 10*1024kB 4*2048kB 1*4096kB = 10511
  3029. 6kB
  3030. Sep 27 08:31:00 monitor kernel: HighMem: 219*4kB 13*8kB 3*16kB 2*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1156kB
  3031. Sep 27 08:31:00 monitor kernel: Swap cache: add 7960237, delete 7960254, find 1543314211/1543477359, race 145+177
  3032. Sep 27 08:31:00 monitor kernel: Free swap = 28kB
  3033. Sep 27 08:31:00 monitor kernel: Total swap = 3911816kB
  3034. Sep 27 08:31:00 monitor kernel: Out of memory: kill process 23255 (pbx_log.pl) score 14402 or a child
  3035. Sep 27 08:31:00 monitor kernel: Killed process 23255 (pbx_log.pl)
  3036. Sep 27 08:51:08 monitor kernel: pbx_log.pl invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  3037. Sep 27 08:51:10 monitor kernel: Pid: 23185, comm: pbx_log.pl Not tainted 2.6.24.3 #3
  3038. Sep 27 08:51:10 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  3039. Sep 27 08:51:10 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  3040. Sep 27 08:51:10 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  3041. Sep 27 08:51:10 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  3042. Sep 27 08:51:10 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  3043. Sep 27 08:51:10 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  3044. Sep 27 08:51:10 monitor kernel: [<c02b02e0>] e1000_clean_rx_irq+0x0/0x4e0
  3045. Sep 27 08:51:10 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  3046. Sep 27 08:51:10 monitor kernel: [<c0184390>] d_alloc+0x100/0x180
  3047. Sep 27 08:51:10 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  3048. Sep 27 08:51:10 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  3049. Sep 27 08:51:10 monitor kernel: [<c017fe72>] sys_select+0x182/0x1b0
  3050. Sep 27 08:51:10 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  3051. Sep 27 08:51:10 monitor kernel: [<c043a202>] error_code+0x72/0x78
  3052. Sep 27 08:51:10 monitor kernel: =======================
  3053. Sep 27 08:51:10 monitor kernel: DMA per-cpu:
  3054. Sep 27 08:51:10 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  3055. Sep 27 08:51:10 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  3056. Sep 27 08:51:10 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  3057. Sep 27 08:51:10 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  3058. Sep 27 08:51:10 monitor kernel: Normal per-cpu:
  3059. Sep 27 08:51:10 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 70 Cold: hi: 62, btch: 15 usd: 51
  3060. Sep 27 08:51:10 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 117 Cold: hi: 62, btch: 15 usd: 17
  3061. Sep 27 08:51:10 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 113 Cold: hi: 62, btch: 15 usd: 42
  3062. Sep 27 08:51:10 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 160 Cold: hi: 62, btch: 15 usd: 50
  3063. Sep 27 08:51:10 monitor kernel: HighMem per-cpu:
  3064. Sep 27 08:51:10 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 30 Cold: hi: 62, btch: 15 usd: 57
  3065. Sep 27 08:51:10 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 70 Cold: hi: 62, btch: 15 usd: 55
  3066. Sep 27 08:51:10 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 174 Cold: hi: 62, btch: 15 usd: 59
  3067. Sep 27 08:51:10 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 77 Cold: hi: 62, btch: 15 usd: 61
  3068. Sep 27 08:51:10 monitor kernel: Active:427689 inactive:431507 dirty:0 writeback:0 unstable:0
  3069. Sep 27 08:51:10 monitor kernel: free:29183 slab:4863 mapped:23 pagetables:4563 bounce:0
  3070. Sep 27 08:51:10 monitor kernel: DMA free:11028kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  3071. able? yes
  3072. Sep 27 08:51:10 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  3073. Sep 27 08:51:10 monitor kernel: Normal free:105204kB min:3744kB low:4680kB high:5616kB active:354624kB inactive:349620kB present:894080kB pages_sca
  3074. nned:1256586 all_unreclaimable? yes
  3075. Sep 27 08:51:10 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  3076. Sep 27 08:51:10 monitor kernel: HighMem free:500kB min:512kB low:3916kB high:7324kB active:1356132kB inactive:1376408kB present:3251200kB pages_sca
  3077. nned:5038068 all_unreclaimable? yes
  3078. Sep 27 08:51:10 monitor kernel: lowmem_reserve[]: 0 0 0 0
  3079. Sep 27 08:51:10 monitor kernel: DMA: 65*4kB 64*8kB 59*16kB 45*32kB 27*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 11028kB
  3080. Sep 27 08:51:10 monitor kernel: Normal: 809*4kB 1702*8kB 1420*16kB 467*32kB 132*64kB 38*128kB 28*256kB 13*512kB 11*1024kB 4*2048kB 1*4096kB = 10520
  3081. 4kB
  3082. Sep 27 08:51:10 monitor kernel: HighMem: 103*4kB 11*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 500kB
  3083. Sep 27 08:51:10 monitor kernel: Swap cache: add 8101667, delete 8101683, find 1543392300/1543560278, race 145+177
  3084. Sep 27 08:51:10 monitor kernel: Free swap = 0kB
  3085. Sep 27 08:51:10 monitor kernel: Total swap = 3911816kB
  3086. Sep 27 08:51:10 monitor kernel: Out of memory: kill process 23073 (pbx_log.pl) score 16201 or a child
  3087. Sep 27 08:51:10 monitor kernel: Killed process 23073 (pbx_log.pl)
  3088. Sep 27 09:03:51 monitor kernel: postgres invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  3089. Sep 27 09:03:51 monitor kernel: Pid: 8478, comm: postgres Not tainted 2.6.24.3 #3
  3090. Sep 27 09:03:51 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  3091. Sep 27 09:03:51 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  3092. Sep 27 09:03:51 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  3093. Sep 27 09:03:51 monitor kernel: [<c0151ae1>] filemap_fault+0xf1/0x420
  3094. Sep 27 09:03:51 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  3095. Sep 27 09:03:51 monitor kernel: [<c014f588>] __lock_page+0x68/0x70
  3096. Sep 27 09:03:51 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  3097. Sep 27 09:03:51 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  3098. Sep 27 09:03:51 monitor kernel: [<c0170dbc>] do_sys_open+0xbc/0xe0
  3099. Sep 27 09:03:51 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  3100. Sep 27 09:03:51 monitor kernel: [<c043a202>] error_code+0x72/0x78
  3101. Sep 27 09:03:51 monitor kernel: [<c0430000>] rpcb_getport_sync+0x0/0x190
  3102. Sep 27 09:03:51 monitor kernel: =======================
  3103. Sep 27 09:03:51 monitor kernel: DMA per-cpu:
  3104. Sep 27 09:03:51 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  3105. Sep 27 09:03:51 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  3106. Sep 27 09:03:51 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  3107. Sep 27 09:03:51 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  3108. Sep 27 09:03:51 monitor kernel: Normal per-cpu:
  3109. Sep 27 09:03:51 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 51 Cold: hi: 62, btch: 15 usd: 41
  3110. Sep 27 09:03:51 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 142 Cold: hi: 62, btch: 15 usd: 33
  3111. Sep 27 09:03:51 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 126 Cold: hi: 62, btch: 15 usd: 60
  3112. Sep 27 09:03:51 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 172 Cold: hi: 62, btch: 15 usd: 43
  3113. Sep 27 09:03:51 monitor kernel: HighMem per-cpu:
  3114. Sep 27 09:03:51 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 120 Cold: hi: 62, btch: 15 usd: 52
  3115. Sep 27 09:03:51 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 176 Cold: hi: 62, btch: 15 usd: 54
  3116. Sep 27 09:03:51 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 67 Cold: hi: 62, btch: 15 usd: 35
  3117. Sep 27 09:03:51 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 15 Cold: hi: 62, btch: 15 usd: 49
  3118. Sep 27 09:03:51 monitor kernel: Active:422151 inactive:437826 dirty:0 writeback:0 unstable:0
  3119. Sep 27 09:03:51 monitor kernel: free:29187 slab:4718 mapped:8 pagetables:4551 bounce:0
  3120. Sep 27 09:03:51 monitor kernel: DMA free:11028kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  3121. able? yes
  3122. Sep 27 09:03:51 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  3123. Sep 27 09:03:51 monitor kernel: Normal free:105300kB min:3744kB low:4680kB high:5616kB active:354252kB inactive:350452kB present:894080kB pages_sca
  3124. nned:1740261 all_unreclaimable? yes
  3125. Sep 27 09:03:51 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  3126. Sep 27 09:03:51 monitor kernel: HighMem free:420kB min:512kB low:3916kB high:7324kB active:1334352kB inactive:1400852kB present:3251200kB pages_sca
  3127. nned:13224460 all_unreclaimable? yes
  3128. Sep 27 09:03:51 monitor kernel: lowmem_reserve[]: 0 0 0 0
  3129. Sep 27 09:03:51 monitor kernel: DMA: 65*4kB 64*8kB 59*16kB 45*32kB 27*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 11028kB
  3130. Sep 27 09:03:51 monitor kernel: Normal: 955*4kB 1767*8kB 1419*16kB 436*32kB 132*64kB 38*128kB 28*256kB 13*512kB 11*1024kB 4*2048kB 1*4096kB = 10530
  3131. 0kB
  3132. Sep 27 09:03:51 monitor kernel: HighMem: 81*4kB 2*8kB 1*16kB 0*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 420kB
  3133. Sep 27 09:03:51 monitor kernel: Swap cache: add 8192908, delete 8192925, find 1543440728/1543614125, race 145+179
  3134. Sep 27 09:03:51 monitor kernel: Free swap = 0kB
  3135. Sep 27 09:03:51 monitor kernel: Total swap = 3911816kB
  3136. Sep 27 09:03:51 monitor kernel: Out of memory: kill process 23173 (pbx_log.pl) score 16201 or a child
  3137. Sep 27 09:03:51 monitor kernel: Killed process 23173 (pbx_log.pl)
  3138. Sep 27 09:03:51 monitor kernel: postgres invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  3139. Sep 27 09:03:51 monitor kernel: Pid: 8476, comm: postgres Not tainted 2.6.24.3 #3
  3140. Sep 27 09:03:51 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  3141. Sep 27 09:03:51 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  3142. Sep 27 09:03:51 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  3143. Sep 27 09:03:51 monitor kernel: [<c011a2f2>] update_curr+0x72/0x110
  3144. Sep 27 09:03:51 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  3145. Sep 27 09:03:51 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  3146. Sep 27 09:03:51 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  3147. Sep 27 09:03:51 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  3148. Sep 27 09:03:51 monitor kernel: [<c015cd1f>] do_wp_page+0x1bf/0x430
  3149. Sep 27 09:03:51 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  3150. Sep 27 09:03:51 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  3151. Sep 27 09:03:51 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  3152. Sep 27 09:03:51 monitor kernel: [<c043a202>] error_code+0x72/0x78
  3153. Sep 27 09:03:51 monitor kernel: [<c0430000>] rpcb_getport_sync+0x0/0x190
  3154. Sep 27 09:03:51 monitor kernel: =======================
  3155. Sep 27 09:03:51 monitor kernel: DMA per-cpu:
  3156. Sep 27 09:03:51 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  3157. Sep 27 09:03:51 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  3158. Sep 27 09:03:51 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  3159. Sep 27 09:03:51 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  3160. Sep 27 09:03:51 monitor kernel: Normal per-cpu:
  3161. Sep 27 09:03:51 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 51 Cold: hi: 62, btch: 15 usd: 41
  3162. Sep 27 09:03:51 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 136 Cold: hi: 62, btch: 15 usd: 33
  3163. Sep 27 09:03:51 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 126 Cold: hi: 62, btch: 15 usd: 60
  3164. Sep 27 09:03:51 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 172 Cold: hi: 62, btch: 15 usd: 43
  3165. Sep 27 09:03:51 monitor kernel: HighMem per-cpu:
  3166. Sep 27 09:03:51 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 120 Cold: hi: 62, btch: 15 usd: 52
  3167. Sep 27 09:03:51 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 176 Cold: hi: 62, btch: 15 usd: 22
  3168. Sep 27 09:03:51 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 67 Cold: hi: 62, btch: 15 usd: 35
  3169. Sep 27 09:03:51 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 15 Cold: hi: 62, btch: 15 usd: 49
  3170. Sep 27 09:03:51 monitor kernel: Active:512068 inactive:349665 dirty:0 writeback:0 unstable:0
  3171. Sep 27 09:03:51 monitor kernel: free:29187 slab:4725 mapped:8 pagetables:4551 bounce:0
  3172. Sep 27 09:03:51 monitor kernel: DMA free:11028kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  3173. able? yes
  3174. Sep 27 09:03:51 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  3175. Sep 27 09:03:51 monitor kernel: Normal free:105300kB min:3744kB low:4680kB high:5616kB active:442508kB inactive:261932kB present:894080kB pages_sca
  3176. nned:6123667 all_unreclaimable? yes
  3177. Sep 27 09:03:51 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  3178. Sep 27 09:03:51 monitor kernel: HighMem free:420kB min:512kB low:3916kB high:7324kB active:1605764kB inactive:1136728kB present:3251200kB pages_sca
  3179. nned:38708146 all_unreclaimable? yes
  3180. Sep 27 09:03:51 monitor kernel: lowmem_reserve[]: 0 0 0 0
  3181. Sep 27 09:03:51 monitor kernel: DMA: 65*4kB 64*8kB 59*16kB 45*32kB 27*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 11028kB
  3182. Sep 27 09:03:51 monitor kernel: Normal: 955*4kB 1767*8kB 1419*16kB 436*32kB 132*64kB 38*128kB 28*256kB 13*512kB 11*1024kB 4*2048kB 1*4096kB = 10530
  3183. 0kB
  3184. Sep 27 09:03:51 monitor kernel: HighMem: 81*4kB 2*8kB 1*16kB 0*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 420kB
  3185. Sep 27 09:03:51 monitor kernel: Swap cache: add 8192908, delete 8192925, find 1543440728/1543614125, race 145+179
  3186. Sep 27 09:03:51 monitor kernel: Free swap = 0kB
  3187. Sep 27 09:03:51 monitor kernel: Total swap = 3911816kB
  3188. Sep 27 09:03:51 monitor kernel: Out of memory: kill process 23175 (pbx_log.pl) score 16201 or a child
  3189. Sep 27 09:03:51 monitor kernel: Killed process 23175 (pbx_log.pl)
  3190. Sep 27 09:13:28 monitor kernel: syslogd invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  3191. Sep 27 09:13:29 monitor kernel: Pid: 2182, comm: syslogd Not tainted 2.6.24.3 #3
  3192. Sep 27 09:13:29 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  3193. Sep 27 09:13:29 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  3194. Sep 27 09:13:29 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  3195. Sep 27 09:13:29 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  3196. Sep 27 09:13:29 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  3197. Sep 27 09:13:29 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  3198. Sep 27 09:13:29 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  3199. Sep 27 09:13:29 monitor kernel: [<c012c4bb>] recalc_sigpending+0xb/0x30
  3200. Sep 27 09:13:29 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  3201. Sep 27 09:13:29 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  3202. Sep 27 09:13:29 monitor kernel: [<c017fd3d>] sys_select+0x4d/0x1b0
  3203. Sep 27 09:13:29 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  3204. Sep 27 09:13:29 monitor kernel: [<c043a202>] error_code+0x72/0x78
  3205. Sep 27 09:13:29 monitor kernel: =======================
  3206. Sep 27 09:13:29 monitor kernel: DMA per-cpu:
  3207. Sep 27 09:13:29 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  3208. Sep 27 09:13:29 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  3209. Sep 27 09:13:29 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  3210. Sep 27 09:13:29 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  3211. Sep 27 09:13:29 monitor kernel: Normal per-cpu:
  3212. Sep 27 09:13:29 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 21 Cold: hi: 62, btch: 15 usd: 48
  3213. Sep 27 09:13:29 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 161 Cold: hi: 62, btch: 15 usd: 24
  3214. Sep 27 09:13:29 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 45 Cold: hi: 62, btch: 15 usd: 55
  3215. Sep 27 09:13:29 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 65 Cold: hi: 62, btch: 15 usd: 54
  3216. Sep 27 09:13:29 monitor kernel: HighMem per-cpu:
  3217. Sep 27 09:13:29 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 76 Cold: hi: 62, btch: 15 usd: 51
  3218. Sep 27 09:13:29 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 52 Cold: hi: 62, btch: 15 usd: 53
  3219. Sep 27 09:13:29 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 72 Cold: hi: 62, btch: 15 usd: 56
  3220. Sep 27 09:13:29 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 164 Cold: hi: 62, btch: 15 usd: 56
  3221. Sep 27 09:13:29 monitor kernel: Active:430162 inactive:430288 dirty:0 writeback:0 unstable:0
  3222. Sep 27 09:13:29 monitor kernel: free:29197 slab:4752 mapped:27 pagetables:4554 bounce:0
  3223. Sep 27 09:13:29 monitor kernel: DMA free:11028kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  3224. able? yes
  3225. Sep 27 09:13:29 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  3226. Sep 27 09:13:29 monitor kernel: Normal free:105276kB min:3744kB low:4680kB high:5616kB active:363576kB inactive:342920kB present:894080kB pages_sca
  3227. nned:1106869 all_unreclaimable? yes
  3228. Sep 27 09:13:29 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  3229. Sep 27 09:13:29 monitor kernel: HighMem free:484kB min:512kB low:3916kB high:7324kB active:1357072kB inactive:1378232kB present:3251200kB pages_sca
  3230. nned:4353234 all_unreclaimable? yes
  3231. Sep 27 09:13:29 monitor kernel: lowmem_reserve[]: 0 0 0 0
  3232. Sep 27 09:13:29 monitor kernel: DMA: 65*4kB 64*8kB 59*16kB 45*32kB 27*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 11028kB
  3233. Sep 27 09:13:29 monitor kernel: Normal: 961*4kB 1797*8kB 1417*16kB 426*32kB 133*64kB 38*128kB 28*256kB 13*512kB 11*1024kB 4*2048kB 1*4096kB = 10527
  3234. 6kB
  3235. Sep 27 09:13:29 monitor kernel: HighMem: 63*4kB 19*8kB 1*16kB 0*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 484kB
  3236. Sep 27 09:13:29 monitor kernel: Swap cache: add 8268438, delete 8268455, find 1543478250/1543655462, race 145+182
  3237. Sep 27 09:13:29 monitor kernel: Free swap = 0kB
  3238. Sep 27 09:13:29 monitor kernel: Total swap = 3911816kB
  3239. Sep 27 09:13:29 monitor kernel: Out of memory: kill process 23249 (pbx_log.pl) score 16201 or a child
  3240. Sep 27 09:13:29 monitor kernel: Killed process 23249 (pbx_log.pl)
  3241. Sep 27 09:13:29 monitor kernel: syslogd invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  3242. Sep 27 09:13:29 monitor kernel: Pid: 2182, comm: syslogd Not tainted 2.6.24.3 #3
  3243. Sep 27 09:13:29 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  3244. Sep 27 09:13:29 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  3245. Sep 27 09:13:29 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  3246. Sep 27 09:13:29 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  3247. Sep 27 09:13:29 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  3248. Sep 27 09:13:29 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  3249. Sep 27 09:13:29 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  3250. Sep 27 09:13:29 monitor kernel: [<c012c4bb>] recalc_sigpending+0xb/0x30
  3251. Sep 27 09:13:29 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  3252. Sep 27 09:13:29 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  3253. Sep 27 09:13:29 monitor kernel: [<c017fd3d>] sys_select+0x4d/0x1b0
  3254. Sep 27 09:13:29 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  3255. Sep 27 09:13:29 monitor kernel: [<c043a202>] error_code+0x72/0x78
  3256. Sep 27 09:13:29 monitor kernel: =======================
  3257. Sep 27 09:13:29 monitor kernel: DMA per-cpu:
  3258. Sep 27 09:13:29 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  3259. Sep 27 09:13:29 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  3260. Sep 27 09:13:29 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  3261. Sep 27 09:13:29 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  3262. Sep 27 09:13:29 monitor kernel: Normal per-cpu:
  3263. Sep 27 09:13:29 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 21 Cold: hi: 62, btch: 15 usd: 48
  3264. Sep 27 09:13:29 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 161 Cold: hi: 62, btch: 15 usd: 24
  3265. Sep 27 09:13:29 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 45 Cold: hi: 62, btch: 15 usd: 55
  3266. Sep 27 09:13:29 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 65 Cold: hi: 62, btch: 15 usd: 54
  3267. Sep 27 09:13:29 monitor kernel: HighMem per-cpu:
  3268. Sep 27 09:13:29 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 76 Cold: hi: 62, btch: 15 usd: 51
  3269. Sep 27 09:13:29 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 52 Cold: hi: 62, btch: 15 usd: 53
  3270. Sep 27 09:13:29 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 72 Cold: hi: 62, btch: 15 usd: 56
  3271. Sep 27 09:13:29 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 164 Cold: hi: 62, btch: 15 usd: 40
  3272. Sep 27 09:13:29 monitor kernel: Active:432759 inactive:427839 dirty:0 writeback:0 unstable:0
  3273. Sep 27 09:13:29 monitor kernel: free:29197 slab:4753 mapped:27 pagetables:4554 bounce:0
  3274. Sep 27 09:13:29 monitor kernel: DMA free:11028kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  3275. able? yes
  3276. Sep 27 09:13:29 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  3277. Sep 27 09:13:29 monitor kernel: Normal free:105276kB min:3744kB low:4680kB high:5616kB active:369348kB inactive:337216kB present:894080kB pages_sca
  3278. nned:1115669 all_unreclaimable? yes
  3279. Sep 27 09:13:29 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  3280. Sep 27 09:13:29 monitor kernel: HighMem free:484kB min:512kB low:3916kB high:7324kB active:1361688kB inactive:1374140kB present:3251200kB pages_sca
  3281. nned:5225647 all_unreclaimable? yes
  3282. Sep 27 09:13:29 monitor kernel: lowmem_reserve[]: 0 0 0 0
  3283. Sep 27 09:13:29 monitor kernel: DMA: 65*4kB 64*8kB 59*16kB 45*32kB 27*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 11028kB
  3284. Sep 27 09:13:29 monitor kernel: Normal: 961*4kB 1797*8kB 1417*16kB 426*32kB 133*64kB 38*128kB 28*256kB 13*512kB 11*1024kB 4*2048kB 1*4096kB = 10527
  3285. 6kB
  3286. Sep 27 09:13:29 monitor kernel: HighMem: 63*4kB 19*8kB 1*16kB 0*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 484kB
  3287. Sep 27 09:13:29 monitor kernel: Swap cache: add 8268438, delete 8268455, find 1543478250/1543655462, race 145+182
  3288. Sep 27 09:13:29 monitor kernel: Free swap = 0kB
  3289. Sep 27 09:13:29 monitor kernel: Total swap = 3911816kB
  3290. Sep 27 09:13:29 monitor kernel: Out of memory: kill process 23250 (pbx_log.pl) score 16201 or a child
  3291. Sep 27 09:13:29 monitor kernel: Killed process 23250 (pbx_log.pl)
  3292. Sep 27 09:23:23 monitor kernel: xmond-communica invoked oom-killer: gfp_mask=0x1200d2, order=0, oomkilladj=0
  3293. Sep 27 09:23:23 monitor kernel: Pid: 27218, comm: xmond-communica Not tainted 2.6.24.3 #3
  3294. Sep 27 09:23:23 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  3295. Sep 27 09:23:23 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  3296. Sep 27 09:23:23 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  3297. Sep 27 09:23:23 monitor kernel: [<c016595e>] read_swap_cache_async+0x9e/0xe0
  3298. Sep 27 09:23:23 monitor kernel: [<c015cfe3>] swapin_readahead+0x53/0x70
  3299. Sep 27 09:23:23 monitor kernel: [<c015eb2d>] handle_mm_fault+0x49d/0x5b0
  3300. Sep 27 09:23:23 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  3301. Sep 27 09:23:23 monitor kernel: [<c039bebb>] sys_socketcall+0x1cb/0x280
  3302. Sep 27 09:23:23 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  3303. Sep 27 09:23:23 monitor kernel: [<c043a202>] error_code+0x72/0x78
  3304. Sep 27 09:23:23 monitor kernel: [<c0430000>] rpcb_getport_sync+0x0/0x190
  3305. Sep 27 09:23:23 monitor kernel: =======================
  3306. Sep 27 09:23:23 monitor kernel: DMA per-cpu:
  3307. Sep 27 09:23:23 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  3308. Sep 27 09:23:23 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  3309. Sep 27 09:23:23 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  3310. Sep 27 09:23:23 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  3311. Sep 27 09:23:23 monitor kernel: Normal per-cpu:
  3312. Sep 27 09:23:23 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 74 Cold: hi: 62, btch: 15 usd: 50
  3313. Sep 27 09:23:23 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 66 Cold: hi: 62, btch: 15 usd: 61
  3314. Sep 27 09:23:23 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 174 Cold: hi: 62, btch: 15 usd: 55
  3315. Sep 27 09:23:23 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 28 Cold: hi: 62, btch: 15 usd: 52
  3316. Sep 27 09:23:23 monitor kernel: HighMem per-cpu:
  3317. Sep 27 09:23:23 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 71 Cold: hi: 62, btch: 15 usd: 53
  3318. Sep 27 09:23:23 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 19 Cold: hi: 62, btch: 15 usd: 52
  3319. Sep 27 09:23:23 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 57 Cold: hi: 62, btch: 15 usd: 46
  3320. Sep 27 09:23:23 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 10 Cold: hi: 62, btch: 15 usd: 47
  3321. Sep 27 09:23:23 monitor kernel: Active:769779 inactive:89218 dirty:0 writeback:0 unstable:0
  3322. Sep 27 09:23:23 monitor kernel: free:29124 slab:4672 mapped:21 pagetables:4657 bounce:0
  3323. Sep 27 09:23:23 monitor kernel: DMA free:11028kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  3324. able? yes
  3325. Sep 27 09:23:23 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  3326. Sep 27 09:23:23 monitor kernel: Normal free:104984kB min:3744kB low:4680kB high:5616kB active:354660kB inactive:350972kB present:894080kB pages_sca
  3327. nned:1196745 all_unreclaimable? yes
  3328. Sep 27 09:23:23 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  3329. Sep 27 09:23:23 monitor kernel: HighMem free:484kB min:512kB low:3916kB high:7324kB active:2724456kB inactive:5900kB present:3251200kB pages_scanne
  3330. d:4811359 all_unreclaimable? yes
  3331. Sep 27 09:23:23 monitor kernel: lowmem_reserve[]: 0 0 0 0
  3332. Sep 27 09:23:23 monitor kernel: DMA: 65*4kB 64*8kB 59*16kB 45*32kB 27*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 11028kB
  3333. Sep 27 09:23:23 monitor kernel: Normal: 832*4kB 1784*8kB 1438*16kB 425*32kB 133*64kB 38*128kB 28*256kB 13*512kB 11*1024kB 4*2048kB 1*4096kB = 10496
  3334. 0kB
  3335. Sep 27 09:23:23 monitor kernel: HighMem: 63*4kB 21*8kB 0*16kB 0*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 484kB
  3336. Sep 27 09:23:23 monitor kernel: Swap cache: add 8336371, delete 8336388, find 1543522579/1543703206, race 145+182
  3337. Sep 27 09:23:23 monitor kernel: Free swap = 0kB
  3338. Sep 27 09:23:23 monitor kernel: Total swap = 3911816kB
  3339. Sep 27 09:23:23 monitor kernel: Out of memory: kill process 23089 (pbx_log.pl) score 17270 or a child
  3340. Sep 27 09:23:23 monitor kernel: Killed process 23089 (pbx_log.pl)
  3341. Sep 27 09:23:23 monitor kernel: hald invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  3342. Sep 27 09:23:23 monitor kernel: Pid: 3293, comm: hald Not tainted 2.6.24.3 #3
  3343. Sep 27 09:23:23 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  3344. Sep 27 09:23:23 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  3345. Sep 27 09:23:23 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  3346. Sep 27 09:23:23 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  3347. Sep 27 09:23:23 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  3348. Sep 27 09:23:23 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  3349. Sep 27 09:23:23 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  3350. Sep 27 09:23:23 monitor kernel: [<c017c2d3>] do_path_lookup+0x73/0x1b0
  3351. Sep 27 09:23:23 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  3352. Sep 27 09:23:23 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  3353. Sep 27 09:23:23 monitor kernel: [<c0126af8>] sys_gettimeofday+0x28/0x80
  3354. Sep 27 09:23:23 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  3355. Sep 27 09:23:23 monitor kernel: [<c043a202>] error_code+0x72/0x78
  3356. Sep 27 09:23:23 monitor kernel: [<c0430000>] rpcb_getport_sync+0x0/0x190
  3357. Sep 27 09:23:23 monitor kernel: =======================
  3358. Sep 27 09:23:23 monitor kernel: DMA per-cpu:
  3359. Sep 27 09:23:23 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  3360. Sep 27 09:23:23 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  3361. Sep 27 09:23:23 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  3362. Sep 27 09:23:23 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  3363. Sep 27 09:23:23 monitor kernel: Normal per-cpu:
  3364. Sep 27 09:23:23 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 66 Cold: hi: 62, btch: 15 usd: 50
  3365. Sep 27 09:23:23 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 66 Cold: hi: 62, btch: 15 usd: 47
  3366. Sep 27 09:23:23 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 174 Cold: hi: 62, btch: 15 usd: 60
  3367. Sep 27 09:23:23 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 27 Cold: hi: 62, btch: 15 usd: 56
  3368. Sep 27 09:23:23 monitor kernel: HighMem per-cpu:
  3369. Sep 27 09:23:23 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 71 Cold: hi: 62, btch: 15 usd: 53
  3370. Sep 27 09:23:23 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 19 Cold: hi: 62, btch: 15 usd: 52
  3371. Sep 27 09:23:23 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 57 Cold: hi: 62, btch: 15 usd: 46
  3372. Sep 27 09:23:23 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 10 Cold: hi: 62, btch: 15 usd: 15
  3373. Sep 27 09:23:23 monitor kernel: Active:289687 inactive:569174 dirty:0 writeback:0 unstable:0
  3374. Sep 27 09:23:23 monitor kernel: free:29133 slab:4681 mapped:20 pagetables:4657 bounce:0
  3375. Sep 27 09:23:23 monitor kernel: DMA free:11028kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  3376. able? yes
  3377. Sep 27 09:23:23 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  3378. Sep 27 09:23:23 monitor kernel: Normal free:105020kB min:3744kB low:4680kB high:5616kB active:395540kB inactive:303692kB present:894080kB pages_sca
  3379. nned:9243165 all_unreclaimable? yes
  3380. Sep 27 09:23:23 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  3381. Sep 27 09:23:23 monitor kernel: HighMem free:484kB min:512kB low:3916kB high:7324kB active:763208kB inactive:1973004kB present:3251200kB pages_scan
  3382. ned:18477092 all_unreclaimable? yes
  3383. Sep 27 09:23:23 monitor kernel: lowmem_reserve[]: 0 0 0 0
  3384. Sep 27 09:23:23 monitor kernel: DMA: 65*4kB 64*8kB 59*16kB 45*32kB 27*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 11028kB
  3385. Sep 27 09:23:23 monitor kernel: Normal: 847*4kB 1784*8kB 1438*16kB 425*32kB 133*64kB 38*128kB 28*256kB 13*512kB 11*1024kB 4*2048kB 1*4096kB = 10502
  3386. 0kB
  3387. Sep 27 09:23:23 monitor kernel: HighMem: 63*4kB 21*8kB 0*16kB 0*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 484kB
  3388. Sep 27 09:23:23 monitor kernel: Swap cache: add 8336371, delete 8336388, find 1543522579/1543703206, race 145+182
  3389. Sep 27 09:23:23 monitor kernel: Free swap = 0kB
  3390. Sep 27 09:23:23 monitor kernel: Total swap = 3911816kB
  3391. Sep 27 09:23:23 monitor kernel: Out of memory: kill process 23091 (pbx_log.pl) score 17270 or a child
  3392. Sep 27 09:23:23 monitor kernel: Killed process 23091 (pbx_log.pl)
  3393. Sep 27 09:42:15 monitor kernel: pbx_log.pl invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  3394. Sep 27 09:42:17 monitor kernel: Pid: 23079, comm: pbx_log.pl Not tainted 2.6.24.3 #3
  3395. Sep 27 09:42:28 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  3396. Sep 27 09:42:28 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  3397. Sep 27 09:42:28 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  3398. Sep 27 09:42:28 monitor kernel: [<c011abaa>] enqueue_task_fair+0x1a/0x30
  3399. Sep 27 09:42:28 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  3400. Sep 27 09:42:28 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  3401. Sep 27 09:42:28 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  3402. Sep 27 09:42:28 monitor kernel: [<c011c0ee>] __wake_up+0x3e/0x60
  3403. Sep 27 09:42:28 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  3404. Sep 27 09:42:28 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  3405. Sep 27 09:42:29 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  3406. Sep 27 09:42:29 monitor kernel: [<c017fe72>] sys_select+0x182/0x1b0
  3407. Sep 27 09:42:29 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  3408. Sep 27 09:42:29 monitor kernel: [<c043a202>] error_code+0x72/0x78
  3409. Sep 27 09:42:29 monitor kernel: [<c0430000>] rpcb_getport_sync+0x0/0x190
  3410. Sep 27 09:42:29 monitor kernel: =======================
  3411. Sep 27 09:42:29 monitor kernel: DMA per-cpu:
  3412. Sep 27 09:42:29 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  3413. Sep 27 09:42:29 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  3414. Sep 27 09:42:29 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  3415. Sep 27 09:42:29 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  3416. Sep 27 09:42:29 monitor kernel: Normal per-cpu:
  3417. Sep 27 09:42:29 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 168 Cold: hi: 62, btch: 15 usd: 50
  3418. Sep 27 09:42:29 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 87 Cold: hi: 62, btch: 15 usd: 61
  3419. Sep 27 09:42:29 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 151 Cold: hi: 62, btch: 15 usd: 46
  3420. Sep 27 09:42:29 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 124 Cold: hi: 62, btch: 15 usd: 48
  3421. Sep 27 09:42:29 monitor kernel: HighMem per-cpu:
  3422. Sep 27 09:42:29 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 6 Cold: hi: 62, btch: 15 usd: 36
  3423. Sep 27 09:42:29 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 25 Cold: hi: 62, btch: 15 usd: 26
  3424. Sep 27 09:42:29 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 44 Cold: hi: 62, btch: 15 usd: 54
  3425. Sep 27 09:42:29 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 30 Cold: hi: 62, btch: 15 usd: 47
  3426. Sep 27 09:42:29 monitor kernel: Active:405225 inactive:454937 dirty:0 writeback:0 unstable:0
  3427. Sep 27 09:42:29 monitor kernel: free:29157 slab:4525 mapped:20 pagetables:4538 bounce:0
  3428. Sep 27 09:42:29 monitor kernel: DMA free:11028kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  3429. able? yes
  3430. Sep 27 09:42:29 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  3431. Sep 27 09:42:29 monitor kernel: Normal free:105144kB min:3744kB low:4680kB high:5616kB active:359500kB inactive:346276kB present:894080kB pages_sca
  3432. nned:1148305 all_unreclaimable? yes
  3433. Sep 27 09:42:29 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  3434. Sep 27 09:42:29 monitor kernel: HighMem free:456kB min:512kB low:3916kB high:7324kB active:1261400kB inactive:1473216kB present:3251200kB pages_sca
  3435. nned:12127883 all_unreclaimable? yes
  3436. Sep 27 09:42:29 monitor kernel: lowmem_reserve[]: 0 0 0 0
  3437. Sep 27 09:42:29 monitor kernel: DMA: 65*4kB 64*8kB 59*16kB 45*32kB 27*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 11028kB
  3438. Sep 27 09:42:29 monitor kernel: Normal: 956*4kB 1653*8kB 1452*16kB 441*32kB 133*64kB 38*128kB 28*256kB 13*512kB 11*1024kB 4*2048kB 1*4096kB = 10514
  3439. 4kB
  3440. Sep 27 09:42:29 monitor kernel: HighMem: 34*4kB 26*8kB 3*16kB 0*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 456kB
  3441. Sep 27 09:42:29 monitor kernel: Swap cache: add 8453474, delete 8453492, find 1543589464/1543774034, race 145+200
  3442. Sep 27 09:42:29 monitor kernel: Free swap = 0kB
  3443. Sep 27 09:42:29 monitor kernel: Total swap = 3911816kB
  3444. Sep 27 09:42:29 monitor kernel: Out of memory: kill process 23190 (pbx_log.pl) score 20479 or a child
  3445. Sep 27 09:42:29 monitor kernel: Killed process 23190 (pbx_log.pl)
  3446. Sep 27 09:42:29 monitor kernel: pbx_log.pl invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
  3447. Sep 27 09:42:29 monitor kernel: Pid: 23079, comm: pbx_log.pl Not tainted 2.6.24.3 #3
  3448. Sep 27 09:42:29 monitor kernel: [<c01527ad>] oom_kill_process+0x11d/0x130
  3449. Sep 27 09:42:29 monitor kernel: [<c0152bb3>] out_of_memory+0x173/0x1b0
  3450. Sep 27 09:42:29 monitor kernel: [<c0154e64>] __alloc_pages+0x264/0x360
  3451. Sep 27 09:42:29 monitor kernel: [<c011abaa>] enqueue_task_fair+0x1a/0x30
  3452. Sep 27 09:42:29 monitor kernel: [<c0156dfa>] __do_page_cache_readahead+0x10a/0x220
  3453. Sep 27 09:42:29 monitor kernel: [<c015730a>] do_page_cache_readahead+0x4a/0x70
  3454. Sep 27 09:42:29 monitor kernel: [<c0151cd1>] filemap_fault+0x2e1/0x420
  3455. Sep 27 09:42:29 monitor kernel: [<c011c0ee>] __wake_up+0x3e/0x60
  3456. Sep 27 09:42:29 monitor kernel: [<c015c843>] __do_fault+0x53/0x370
  3457. Sep 27 09:42:29 monitor kernel: [<c015e80c>] handle_mm_fault+0x17c/0x5b0
  3458. Sep 27 09:42:29 monitor kernel: [<c01186b7>] do_page_fault+0xe7/0x620
  3459. Sep 27 09:42:29 monitor kernel: [<c017fe72>] sys_select+0x182/0x1b0
  3460. Sep 27 09:42:29 monitor kernel: [<c01185d0>] do_page_fault+0x0/0x620
  3461. Sep 27 09:42:29 monitor kernel: [<c043a202>] error_code+0x72/0x78
  3462. Sep 27 09:42:29 monitor kernel: [<c0430000>] rpcb_getport_sync+0x0/0x190
  3463. Sep 27 09:42:29 monitor kernel: =======================
  3464. Sep 27 09:42:29 monitor kernel: DMA per-cpu:
  3465. Sep 27 09:42:29 monitor kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  3466. Sep 27 09:42:29 monitor kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  3467. Sep 27 09:42:29 monitor kernel: CPU 2: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  3468. Sep 27 09:42:29 monitor kernel: CPU 3: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
  3469. Sep 27 09:42:29 monitor kernel: Normal per-cpu:
  3470. Sep 27 09:42:29 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 168 Cold: hi: 62, btch: 15 usd: 50
  3471. Sep 27 09:42:29 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 87 Cold: hi: 62, btch: 15 usd: 61
  3472. Sep 27 09:42:29 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 151 Cold: hi: 62, btch: 15 usd: 46
  3473. Sep 27 09:42:29 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 124 Cold: hi: 62, btch: 15 usd: 48
  3474. Sep 27 09:42:29 monitor kernel: HighMem per-cpu:
  3475. Sep 27 09:42:29 monitor kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 6 Cold: hi: 62, btch: 15 usd: 36
  3476. Sep 27 09:42:29 monitor kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 25 Cold: hi: 62, btch: 15 usd: 26
  3477. Sep 27 09:42:29 monitor kernel: CPU 2: Hot: hi: 186, btch: 31 usd: 44 Cold: hi: 62, btch: 15 usd: 50
  3478. Sep 27 09:42:29 monitor kernel: CPU 3: Hot: hi: 186, btch: 31 usd: 30 Cold: hi: 62, btch: 15 usd: 47
  3479. Sep 27 09:42:29 monitor kernel: Active:529423 inactive:330727 dirty:0 writeback:0 unstable:0
  3480. Sep 27 09:42:29 monitor kernel: free:29157 slab:4527 mapped:20 pagetables:4538 bounce:0
  3481. Sep 27 09:42:29 monitor kernel: DMA free:11028kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaim
  3482. able? yes
  3483. Sep 27 09:42:29 monitor kernel: lowmem_reserve[]: 0 873 4048 4048
  3484. Sep 27 09:42:29 monitor kernel: Normal free:105144kB min:3744kB low:4680kB high:5616kB active:364084kB inactive:341716kB present:894080kB pages_sca
  3485. nned:1155103 all_unreclaimable? yes
  3486. Sep 27 09:42:29 monitor kernel: lowmem_reserve[]: 0 0 25400 25400
  3487. Sep 27 09:42:29 monitor kernel: HighMem free:456kB min:512kB low:3916kB high:7324kB active:1753608kB inactive:981192kB present:3251200kB pages_scan
  3488. ned:12780948 all_unreclaimable? yes
  3489. Sep 27 09:42:29 monitor kernel: lowmem_reserve[]: 0 0 0 0
  3490. Sep 27 09:42:29 monitor kernel: DMA: 65*4kB 64*8kB 59*16kB 45*32kB 27*64kB 12*128kB 4*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 11028kB
  3491. Sep 27 09:42:29 monitor kernel: Normal: 956*4kB 1653*8kB 1452*16kB 441*32kB 133*64kB 38*128kB 28*256kB 13*512kB 11*1024kB 4*2048kB 1*4096kB = 10514
  3492. 4kB
  3493. Sep 27 09:42:29 monitor kernel: HighMem: 34*4kB 26*8kB 3*16kB 0*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 456kB
  3494. Sep 27 09:42:29 monitor kernel: Swap cache: add 8453474, delete 8453492, find 1543589464/1543774034, race 145+200
  3495. Sep 27 09:42:29 monitor kernel: Free swap = 0kB
  3496. Sep 27 09:42:29 monitor kernel: Total swap = 3911816kB
  3497. Sep 27 09:42:29 monitor kernel: Out of memory: kill process 23191 (pbx_log.pl) score 20479 or a child
  3498. Sep 27 09:42:29 monitor kernel: Killed process 23191 (pbx_log.pl)
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!