API tools faq
paste
Advertisement
Guest User

Untitled

a guest
Jun 13th, 2019
207
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 119.55 KB | None | 0 0
raw download clone embed print report
  1. list
  2.  
  3.  
  4. Jun 13 12:37:57 online-node-81-113 kernel: Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.3.4 11/08/2016
  5. Jun 13 12:37:57 online-node-81-113 kernel: Call Trace:
  6. Jun 13 12:37:57 online-node-81-113 kernel: dump_stack+0x63/0x85
  7. Jun 13 12:37:57 online-node-81-113 kernel: dump_header+0x9f/0x234
  8. Jun 13 12:37:57 online-node-81-113 kernel: ? mem_cgroup_scan_tasks+0x96/0xf0
  9. Jun 13 12:37:57 online-node-81-113 kernel: oom_kill_process+0x21c/0x430
  10. Jun 13 12:37:57 online-node-81-113 kernel: out_of_memory+0x114/0x4a0
  11. Jun 13 12:37:57 online-node-81-113 kernel: mem_cgroup_out_of_memory+0x4b/0x80
  12. Jun 13 12:37:57 online-node-81-113 kernel: mem_cgroup_oom_synchronize+0x2f9/0x320
  13. Jun 13 12:37:57 online-node-81-113 kernel: ? get_mctgt_type_thp.isra.30+0xc0/0xc0
  14. Jun 13 12:37:57 online-node-81-113 kernel: pagefault_out_of_memory+0x36/0x7c
  15. Jun 13 12:37:57 online-node-81-113 kernel: mm_fault_error+0x65/0x152
  16. Jun 13 12:37:57 online-node-81-113 kernel: __do_page_fault+0x456/0x4f0
  17. Jun 13 12:37:57 online-node-81-113 kernel: do_page_fault+0x38/0x130
  18. Jun 13 12:37:57 online-node-81-113 kernel: ? page_fault+0x36/0x60
  19. Jun 13 12:37:57 online-node-81-113 kernel: page_fault+0x4c/0x60
  20. Jun 13 12:37:57 online-node-81-113 kernel: RIP: 0033:0x7f07230666bf
  21. Jun 13 12:37:57 online-node-81-113 kernel: RSP: 002b:00007ffceb7f1fd8 EFLAGS: 00010206
  22. Jun 13 12:37:57 online-node-81-113 kernel: RAX: 00007f0716179040 RBX: 0000000001400000 RCX: 0000000000208808
  23. Jun 13 12:37:57 online-node-81-113 kernel: RDX: 0000000001400000 RSI: 00007f0715135000 RDI: 00007f0716535000
  24. Jun 13 12:37:57 online-node-81-113 kernel: RBP: 00007f0723296b50 R08: 0000000003240000 R09: 00007f0717f79048
  25. Jun 13 12:37:57 online-node-81-113 kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 00007f0714d79040
  26. Jun 13 12:37:57 online-node-81-113 kernel: R13: 00007f0717f79040 R14: 0000000003200000 R15: 000055b13d705c60
  27. Jun 13 12:37:57 online-node-81-113 kernel: Task in /docker/aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d killed as a result of limit of /docker/aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d
  28. Jun 13 12:37:57 online-node-81-113 kernel: memory: usage 57344kB, limit 57344kB, failcnt 88
  29. Jun 13 12:37:57 online-node-81-113 kernel: memory+swap: usage 57344kB, limit 114688kB, failcnt 0
  30. Jun 13 12:37:57 online-node-81-113 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
  31. Jun 13 12:37:57 online-node-81-113 kernel: Memory cgroup stats for /docker/aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d: cache:10756KB rss:46588KB rss_huge:22528KB shmem:10716KB mapped_file:10716KB dirty:16KB writeback:0KB swap:0KB inactive_anon:10716KB active_anon:46536KB inactive_file:24KB active_file:16KB unevictable:0KB
  32. Jun 13 12:37:57 online-node-81-113 kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
  33. Jun 13 12:37:57 online-node-81-113 kernel: [22551] 0 22551 21395 4336 47 3 0 0 supervisord
  34. Jun 13 12:37:57 online-node-81-113 kernel: [23356] 0 23356 49846 6761 44 3 0 0 php-fpm
  35. Jun 13 12:37:57 online-node-81-113 kernel: [23357] 0 23357 3480 1069 13 3 0 0 nginx
  36. Jun 13 12:37:57 online-node-81-113 kernel: [23358] 7373 23358 3540 453 11 3 0 0 nginx
  37. Jun 13 12:37:57 online-node-81-113 kernel: [23359] 7373 23359 3540 453 11 3 0 0 nginx
  38. Jun 13 12:37:57 online-node-81-113 kernel: [23362] 7373 23362 62710 8466 48 3 0 0 php-fpm
  39. Jun 13 12:37:57 online-node-81-113 kernel: [23363] 7373 23363 49845 1346 30 3 0 0 php-fpm
  40. Jun 13 12:37:57 online-node-81-113 kernel: [23364] 7373 23364 49845 1346 30 3 0 0 php-fpm
  41. Jun 13 12:37:57 online-node-81-113 kernel: [23365] 7373 23365 49845 1346 30 3 0 0 php-fpm
  42. Jun 13 12:37:57 online-node-81-113 kernel: [23367] 7373 23367 49845 1346 30 3 0 0 php-fpm
  43. Jun 13 12:37:57 online-node-81-113 kernel: [23368] 7373 23368 49845 1346 30 3 0 0 php-fpm
  44. Jun 13 12:37:57 online-node-81-113 kernel: [23369] 7373 23369 49845 1346 30 3 0 0 php-fpm
  45. Jun 13 12:37:57 online-node-81-113 kernel: [23370] 7373 23370 49846 1346 30 3 0 0 php-fpm
  46. Jun 13 12:37:57 online-node-81-113 kernel: [23371] 7373 23371 49846 1346 30 3 0 0 php-fpm
  47. Jun 13 12:37:57 online-node-81-113 kernel: [23372] 7373 23372 49846 1346 30 3 0 0 php-fpm
  48. Jun 13 12:37:57 online-node-81-113 kernel: [23373] 7373 23373 49846 1346 30 3 0 0 php-fpm
  49. Jun 13 12:37:57 online-node-81-113 kernel: [23374] 7373 23374 49846 1347 30 3 0 0 php-fpm
  50. Jun 13 12:37:57 online-node-81-113 kernel: [23375] 7373 23375 49846 1347 30 3 0 0 php-fpm
  51. Jun 13 12:37:57 online-node-81-113 kernel: [23376] 7373 23376 49846 1347 30 3 0 0 php-fpm
  52. Jun 13 12:37:57 online-node-81-113 kernel: [23377] 7373 23377 49846 1347 30 3 0 0 php-fpm
  53. Jun 13 12:37:57 online-node-81-113 kernel: [23378] 7373 23378 49846 1347 30 3 0 0 php-fpm
  54. Jun 13 12:37:57 online-node-81-113 kernel: [23379] 7373 23379 49846 1347 30 3 0 0 php-fpm
  55. Jun 13 12:37:57 online-node-81-113 kernel: [23380] 7373 23380 49846 1347 30 3 0 0 php-fpm
  56. Jun 13 12:37:57 online-node-81-113 kernel: [23381] 7373 23381 49846 1347 30 3 0 0 php-fpm
  57. Jun 13 12:37:57 online-node-81-113 kernel: [26458] 0 26458 393 262 7 3 0 0 sh
  58. Jun 13 12:37:57 online-node-81-113 kernel: [ 499] 7373 499 49846 1347 30 3 0 0 php-fpm
  59. Jun 13 12:37:57 online-node-81-113 kernel: [ 2681] 0 2681 3524 824 13 3 0 0 curl
  60. Jun 13 12:37:57 online-node-81-113 kernel: Memory cgroup out of memory: Kill process 23362 (php-fpm) score 594 or sacrifice child
  61. Jun 13 12:37:57 online-node-81-113 kernel: Killed process 23362 (php-fpm) total-vm:250840kB, anon-rss:29532kB, file-rss:4160kB, shmem-rss:172kB
  62. Jun 13 12:37:57 online-node-81-113 kernel: oom_reaper: reaped process 23362 (php-fpm), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
  63. Jun 13 12:37:58 online-node-81-113 kernel: php-fpm invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=0
  64. Jun 13 12:37:58 online-node-81-113 kernel: php-fpm cpuset=aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d mems_allowed=0-1
  65. Jun 13 12:37:58 online-node-81-113 kernel: CPU: 27 PID: 23363 Comm: php-fpm Not tainted 4.14.15-1.el7.elrepo.x86_64 #1
  66. Jun 13 12:37:58 online-node-81-113 kernel: Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.3.4 11/08/2016
  67. Jun 13 12:37:58 online-node-81-113 kernel: Call Trace:
  68. Jun 13 12:37:58 online-node-81-113 kernel: dump_stack+0x63/0x85
  69. Jun 13 12:37:58 online-node-81-113 kernel: dump_header+0x9f/0x234
  70. Jun 13 12:37:58 online-node-81-113 kernel: ? mem_cgroup_scan_tasks+0x96/0xf0
  71. Jun 13 12:37:58 online-node-81-113 kernel: oom_kill_process+0x21c/0x430
  72. Jun 13 12:37:58 online-node-81-113 kernel: out_of_memory+0x114/0x4a0
  73. Jun 13 12:37:58 online-node-81-113 kernel: mem_cgroup_out_of_memory+0x4b/0x80
  74. Jun 13 12:37:58 online-node-81-113 kernel: mem_cgroup_oom_synchronize+0x2f9/0x320
  75. Jun 13 12:37:58 online-node-81-113 kernel: ? get_mctgt_type_thp.isra.30+0xc0/0xc0
  76. Jun 13 12:37:58 online-node-81-113 kernel: pagefault_out_of_memory+0x36/0x7c
  77. Jun 13 12:37:58 online-node-81-113 kernel: mm_fault_error+0x65/0x152
  78. Jun 13 12:37:58 online-node-81-113 kernel: __do_page_fault+0x456/0x4f0
  79. Jun 13 12:37:58 online-node-81-113 kernel: do_page_fault+0x38/0x130
  80. Jun 13 12:37:58 online-node-81-113 kernel: ? page_fault+0x36/0x60
  81. Jun 13 12:37:58 online-node-81-113 kernel: page_fault+0x4c/0x60
  82. Jun 13 12:37:58 online-node-81-113 kernel: RIP: 0033:0x7f07230666bf
  83. Jun 13 12:37:58 online-node-81-113 kernel: RSP: 002b:00007ffceb7f1fd8 EFLAGS: 00010206
  84. Jun 13 12:37:58 online-node-81-113 kernel: RAX: 00007f0716179040 RBX: 0000000001400000 RCX: 0000000000208e08
  85. Jun 13 12:37:58 online-node-81-113 kernel: RDX: 0000000001400000 RSI: 00007f0715132000 RDI: 00007f0716532000
  86. Jun 13 12:37:58 online-node-81-113 kernel: RBP: 00007f0723296b50 R08: 0000000003240000 R09: 00007f0717f79048
  87. Jun 13 12:37:58 online-node-81-113 kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 00007f0714d79040
  88. Jun 13 12:37:58 online-node-81-113 kernel: R13: 00007f0717f79040 R14: 0000000003200000 R15: 000055b13d705c60
  89. Jun 13 12:37:58 online-node-81-113 kernel: Task in /docker/aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d killed as a result of limit of /docker/aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d
  90. Jun 13 12:37:58 online-node-81-113 kernel: memory: usage 57344kB, limit 57344kB, failcnt 126
  91. Jun 13 12:37:58 online-node-81-113 kernel: memory+swap: usage 57344kB, limit 114688kB, failcnt 0
  92. Jun 13 12:37:58 online-node-81-113 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
  93. Jun 13 12:37:58 online-node-81-113 kernel: Memory cgroup stats for /docker/aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d: cache:10756KB rss:46588KB rss_huge:22528KB shmem:10716KB mapped_file:10716KB dirty:16KB writeback:0KB swap:0KB inactive_anon:10572KB active_anon:46728KB inactive_file:24KB active_file:16KB unevictable:0KB
  94. Jun 13 12:37:58 online-node-81-113 kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
  95. Jun 13 12:37:58 online-node-81-113 kernel: [22551] 0 22551 21395 4336 47 3 0 0 supervisord
  96. Jun 13 12:37:58 online-node-81-113 kernel: [23356] 0 23356 49846 6764 44 3 0 0 php-fpm
  97. Jun 13 12:37:58 online-node-81-113 kernel: [23357] 0 23357 3480 1069 13 3 0 0 nginx
  98. Jun 13 12:37:58 online-node-81-113 kernel: [23358] 7373 23358 3540 453 11 3 0 0 nginx
  99. Jun 13 12:37:58 online-node-81-113 kernel: [23359] 7373 23359 3540 453 11 3 0 0 nginx
  100. Jun 13 12:37:58 online-node-81-113 kernel: [23363] 7373 23363 62710 8467 48 3 0 0 php-fpm
  101. Jun 13 12:37:58 online-node-81-113 kernel: [23364] 7373 23364 49845 1346 30 3 0 0 php-fpm
  102. Jun 13 12:37:58 online-node-81-113 kernel: [23365] 7373 23365 49845 1346 30 3 0 0 php-fpm
  103. Jun 13 12:37:58 online-node-81-113 kernel: [23367] 7373 23367 49845 1346 30 3 0 0 php-fpm
  104. Jun 13 12:37:58 online-node-81-113 kernel: [23368] 7373 23368 49845 1346 30 3 0 0 php-fpm
  105. Jun 13 12:37:58 online-node-81-113 kernel: [23369] 7373 23369 49845 1346 30 3 0 0 php-fpm
  106. Jun 13 12:37:58 online-node-81-113 kernel: [23370] 7373 23370 49846 1346 30 3 0 0 php-fpm
  107. Jun 13 12:37:58 online-node-81-113 kernel: [23371] 7373 23371 49846 1346 30 3 0 0 php-fpm
  108. Jun 13 12:37:58 online-node-81-113 kernel: [23372] 7373 23372 49846 1346 30 3 0 0 php-fpm
  109. Jun 13 12:37:58 online-node-81-113 kernel: [23373] 7373 23373 49846 1346 30 3 0 0 php-fpm
  110. Jun 13 12:37:58 online-node-81-113 kernel: [23374] 7373 23374 49846 1347 30 3 0 0 php-fpm
  111. Jun 13 12:37:58 online-node-81-113 kernel: [23375] 7373 23375 49846 1347 30 3 0 0 php-fpm
  112. Jun 13 12:37:58 online-node-81-113 kernel: [23376] 7373 23376 49846 1347 30 3 0 0 php-fpm
  113. Jun 13 12:37:58 online-node-81-113 kernel: [23377] 7373 23377 49846 1347 30 3 0 0 php-fpm
  114. Jun 13 12:37:58 online-node-81-113 kernel: [23378] 7373 23378 49846 1347 30 3 0 0 php-fpm
  115. Jun 13 12:37:58 online-node-81-113 kernel: [23379] 7373 23379 49846 1347 30 3 0 0 php-fpm
  116. Jun 13 12:37:58 online-node-81-113 kernel: [23380] 7373 23380 49846 1347 30 3 0 0 php-fpm
  117. Jun 13 12:37:58 online-node-81-113 kernel: [23381] 7373 23381 49846 1347 30 3 0 0 php-fpm
  118. Jun 13 12:37:58 online-node-81-113 kernel: [26458] 0 26458 393 262 7 3 0 0 sh
  119. Jun 13 12:37:58 online-node-81-113 kernel: [ 499] 7373 499 49846 1347 30 3 0 0 php-fpm
  120. Jun 13 12:37:58 online-node-81-113 kernel: [ 2710] 7373 2710 49846 1347 30 3 0 0 php-fpm
  121. Jun 13 12:37:58 online-node-81-113 kernel: [ 3308] 0 3308 3524 824 15 3 0 0 curl
  122. Jun 13 12:37:58 online-node-81-113 kernel: Memory cgroup out of memory: Kill process 23363 (php-fpm) score 594 or sacrifice child
  123. Jun 13 12:37:58 online-node-81-113 kernel: Killed process 23363 (php-fpm) total-vm:250840kB, anon-rss:29532kB, file-rss:4160kB, shmem-rss:176kB
  124. Jun 13 12:37:58 online-node-81-113 kernel: oom_reaper: reaped process 23363 (php-fpm), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
  125. Jun 13 12:37:59 online-node-81-113 kernel: php-fpm invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=0
  126. Jun 13 12:37:59 online-node-81-113 kernel: php-fpm cpuset=aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d mems_allowed=0-1
  127. Jun 13 12:37:59 online-node-81-113 kernel: CPU: 9 PID: 23364 Comm: php-fpm Not tainted 4.14.15-1.el7.elrepo.x86_64 #1
  128. Jun 13 12:37:59 online-node-81-113 kernel: Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.3.4 11/08/2016
  129. Jun 13 12:37:59 online-node-81-113 kernel: Call Trace:
  130. Jun 13 12:37:59 online-node-81-113 kernel: dump_stack+0x63/0x85
  131. Jun 13 12:37:59 online-node-81-113 kernel: dump_header+0x9f/0x234
  132. Jun 13 12:37:59 online-node-81-113 kernel: ? mem_cgroup_scan_tasks+0x96/0xf0
  133. Jun 13 12:37:59 online-node-81-113 kernel: oom_kill_process+0x21c/0x430
  134. Jun 13 12:37:59 online-node-81-113 kernel: out_of_memory+0x114/0x4a0
  135. Jun 13 12:37:59 online-node-81-113 kernel: mem_cgroup_out_of_memory+0x4b/0x80
  136. Jun 13 12:37:59 online-node-81-113 kernel: mem_cgroup_oom_synchronize+0x2f9/0x320
  137. Jun 13 12:37:59 online-node-81-113 kernel: ? get_mctgt_type_thp.isra.30+0xc0/0xc0
  138. Jun 13 12:37:59 online-node-81-113 kernel: pagefault_out_of_memory+0x36/0x7c
  139. Jun 13 12:37:59 online-node-81-113 kernel: mm_fault_error+0x65/0x152
  140. Jun 13 12:37:59 online-node-81-113 kernel: __do_page_fault+0x456/0x4f0
  141. Jun 13 12:37:59 online-node-81-113 kernel: do_page_fault+0x38/0x130
  142. Jun 13 12:37:59 online-node-81-113 kernel: ? page_fault+0x36/0x60
  143. Jun 13 12:37:59 online-node-81-113 kernel: page_fault+0x4c/0x60
  144. Jun 13 12:37:59 online-node-81-113 kernel: RIP: 0033:0x7f07230666bf
  145. Jun 13 12:37:59 online-node-81-113 kernel: RSP: 002b:00007ffceb7f1fd8 EFLAGS: 00010206
  146. Jun 13 12:37:59 online-node-81-113 kernel: RAX: 00007f0716179040 RBX: 0000000001400000 RCX: 0000000000208e08
  147. Jun 13 12:37:59 online-node-81-113 kernel: RDX: 0000000001400000 RSI: 00007f0715132000 RDI: 00007f0716532000
  148. Jun 13 12:37:59 online-node-81-113 kernel: RBP: 00007f0723296b50 R08: 0000000003240000 R09: 00007f0717f79048
  149. Jun 13 12:37:59 online-node-81-113 kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 00007f0714d79040
  150. Jun 13 12:37:59 online-node-81-113 kernel: R13: 00007f0717f79040 R14: 0000000003200000 R15: 000055b13d705c60
  151. Jun 13 12:37:59 online-node-81-113 kernel: Task in /docker/aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d killed as a result of limit of /docker/aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d
  152. Jun 13 12:37:59 online-node-81-113 kernel: memory: usage 57344kB, limit 57344kB, failcnt 163
  153. Jun 13 12:37:59 online-node-81-113 kernel: memory+swap: usage 57344kB, limit 114688kB, failcnt 0
  154. Jun 13 12:37:59 online-node-81-113 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
  155. Jun 13 12:37:59 online-node-81-113 kernel: Memory cgroup stats for /docker/aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d: cache:10756KB rss:46588KB rss_huge:22528KB shmem:10716KB mapped_file:10716KB dirty:16KB writeback:0KB swap:0KB inactive_anon:10544KB active_anon:46740KB inactive_file:24KB active_file:16KB unevictable:0KB
  156. Jun 13 12:37:59 online-node-81-113 kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
  157. Jun 13 12:37:59 online-node-81-113 kernel: [22551] 0 22551 21395 4336 47 3 0 0 supervisord
  158. Jun 13 12:37:59 online-node-81-113 kernel: [23356] 0 23356 49846 6764 44 3 0 0 php-fpm
  159. Jun 13 12:37:59 online-node-81-113 kernel: [23357] 0 23357 3480 1069 13 3 0 0 nginx
  160. Jun 13 12:37:59 online-node-81-113 kernel: [23358] 7373 23358 3540 453 11 3 0 0 nginx
  161. Jun 13 12:37:59 online-node-81-113 kernel: [23359] 7373 23359 3540 453 11 3 0 0 nginx
  162. Jun 13 12:37:59 online-node-81-113 kernel: [23364] 7373 23364 62710 8467 48 3 0 0 php-fpm
  163. Jun 13 12:37:59 online-node-81-113 kernel: [23365] 7373 23365 49845 1346 30 3 0 0 php-fpm
  164. Jun 13 12:37:59 online-node-81-113 kernel: [23367] 7373 23367 49845 1346 30 3 0 0 php-fpm
  165. Jun 13 12:37:59 online-node-81-113 kernel: [23368] 7373 23368 49845 1346 30 3 0 0 php-fpm
  166. Jun 13 12:37:59 online-node-81-113 kernel: [23369] 7373 23369 49845 1346 30 3 0 0 php-fpm
  167. Jun 13 12:37:59 online-node-81-113 kernel: [23370] 7373 23370 49846 1346 30 3 0 0 php-fpm
  168. Jun 13 12:37:59 online-node-81-113 kernel: [23371] 7373 23371 49846 1346 30 3 0 0 php-fpm
  169. Jun 13 12:37:59 online-node-81-113 kernel: [23372] 7373 23372 49846 1346 30 3 0 0 php-fpm
  170. Jun 13 12:37:59 online-node-81-113 kernel: [23373] 7373 23373 49846 1346 30 3 0 0 php-fpm
  171. Jun 13 12:37:59 online-node-81-113 kernel: [23374] 7373 23374 49846 1347 30 3 0 0 php-fpm
  172. Jun 13 12:37:59 online-node-81-113 kernel: [23375] 7373 23375 49846 1347 30 3 0 0 php-fpm
  173. Jun 13 12:37:59 online-node-81-113 kernel: [23376] 7373 23376 49846 1347 30 3 0 0 php-fpm
  174. Jun 13 12:37:59 online-node-81-113 kernel: [23377] 7373 23377 49846 1347 30 3 0 0 php-fpm
  175. Jun 13 12:37:59 online-node-81-113 kernel: [23378] 7373 23378 49846 1347 30 3 0 0 php-fpm
  176. Jun 13 12:37:59 online-node-81-113 kernel: [23379] 7373 23379 49846 1347 30 3 0 0 php-fpm
  177. Jun 13 12:37:59 online-node-81-113 kernel: [23380] 7373 23380 49846 1347 30 3 0 0 php-fpm
  178. Jun 13 12:37:59 online-node-81-113 kernel: [23381] 7373 23381 49846 1347 30 3 0 0 php-fpm
  179. Jun 13 12:37:59 online-node-81-113 kernel: [26458] 0 26458 393 262 7 3 0 0 sh
  180. Jun 13 12:37:59 online-node-81-113 kernel: [ 499] 7373 499 49846 1347 30 3 0 0 php-fpm
  181. Jun 13 12:37:59 online-node-81-113 kernel: [ 2710] 7373 2710 49846 1347 30 3 0 0 php-fpm
  182. Jun 13 12:37:59 online-node-81-113 kernel: [ 3329] 7373 3329 49846 1347 30 3 0 0 php-fpm
  183. Jun 13 12:37:59 online-node-81-113 kernel: [ 3807] 0 3807 3524 827 13 3 0 0 curl
  184. Jun 13 12:37:59 online-node-81-113 kernel: Memory cgroup out of memory: Kill process 23364 (php-fpm) score 594 or sacrifice child
  185. Jun 13 12:37:59 online-node-81-113 kernel: Killed process 23364 (php-fpm) total-vm:250840kB, anon-rss:29532kB, file-rss:4160kB, shmem-rss:176kB
  186. Jun 13 12:37:59 online-node-81-113 kernel: oom_reaper: reaped process 23364 (php-fpm), now anon-rss:0kB, file-rss:0kB, shmem-rss:8kB
  187. Jun 13 12:38:00 online-node-81-113 kernel: php-fpm invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=0
  188. Jun 13 12:38:00 online-node-81-113 kernel: php-fpm cpuset=aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d mems_allowed=0-1
  189. Jun 13 12:38:00 online-node-81-113 kernel: CPU: 17 PID: 23365 Comm: php-fpm Not tainted 4.14.15-1.el7.elrepo.x86_64 #1
  190. Jun 13 12:38:00 online-node-81-113 kernel: Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.3.4 11/08/2016
  191. Jun 13 12:38:00 online-node-81-113 kernel: Call Trace:
  192. Jun 13 12:38:00 online-node-81-113 kernel: dump_stack+0x63/0x85
  193. Jun 13 12:38:00 online-node-81-113 kernel: dump_header+0x9f/0x234
  194. Jun 13 12:38:00 online-node-81-113 kernel: ? mem_cgroup_scan_tasks+0x96/0xf0
  195. Jun 13 12:38:00 online-node-81-113 kernel: oom_kill_process+0x21c/0x430
  196. Jun 13 12:38:00 online-node-81-113 kernel: out_of_memory+0x114/0x4a0
  197. Jun 13 12:38:00 online-node-81-113 kernel: mem_cgroup_out_of_memory+0x4b/0x80
  198. Jun 13 12:38:00 online-node-81-113 kernel: mem_cgroup_oom_synchronize+0x2f9/0x320
  199. Jun 13 12:38:00 online-node-81-113 kernel: ? get_mctgt_type_thp.isra.30+0xc0/0xc0
  200. Jun 13 12:38:00 online-node-81-113 kernel: pagefault_out_of_memory+0x36/0x7c
  201. Jun 13 12:38:00 online-node-81-113 kernel: mm_fault_error+0x65/0x152
  202. Jun 13 12:38:00 online-node-81-113 kernel: __do_page_fault+0x456/0x4f0
  203. Jun 13 12:38:00 online-node-81-113 kernel: do_page_fault+0x38/0x130
  204. Jun 13 12:38:00 online-node-81-113 kernel: ? page_fault+0x36/0x60
  205. Jun 13 12:38:00 online-node-81-113 kernel: page_fault+0x4c/0x60
  206. Jun 13 12:38:00 online-node-81-113 kernel: RIP: 0033:0x7f07230666bf
  207. Jun 13 12:38:00 online-node-81-113 kernel: RSP: 002b:00007ffceb7f1fd8 EFLAGS: 00010206
  208. Jun 13 12:38:00 online-node-81-113 kernel: RAX: 00007f0716179040 RBX: 0000000001400000 RCX: 0000000000208a08
  209. Jun 13 12:38:00 online-node-81-113 kernel: RDX: 0000000001400000 RSI: 00007f0715134000 RDI: 00007f0716534000
  210. Jun 13 12:38:00 online-node-81-113 kernel: RBP: 00007f0723296b50 R08: 0000000003240000 R09: 00007f0717f79048
  211. Jun 13 12:38:00 online-node-81-113 kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 00007f0714d79040
  212. Jun 13 12:38:00 online-node-81-113 kernel: R13: 00007f0717f79040 R14: 0000000003200000 R15: 000055b13d705c60
  213. Jun 13 12:38:00 online-node-81-113 kernel: Task in /docker/aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d killed as a result of limit of /docker/aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d
  214. Jun 13 12:38:00 online-node-81-113 kernel: memory: usage 57344kB, limit 57344kB, failcnt 204
  215. Jun 13 12:38:00 online-node-81-113 kernel: memory+swap: usage 57344kB, limit 114688kB, failcnt 0
  216. Jun 13 12:38:00 online-node-81-113 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
  217. Jun 13 12:38:00 online-node-81-113 kernel: Memory cgroup stats for /docker/aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d: cache:10744KB rss:46600KB rss_huge:22528KB shmem:10716KB mapped_file:10716KB dirty:0KB writeback:0KB swap:0KB inactive_anon:10540KB active_anon:46772KB inactive_file:24KB active_file:4KB unevictable:0KB
  218. Jun 13 12:38:00 online-node-81-113 kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
  219. Jun 13 12:38:00 online-node-81-113 kernel: [22551] 0 22551 21395 4336 47 3 0 0 supervisord
  220. Jun 13 12:38:00 online-node-81-113 kernel: [23356] 0 23356 49846 6764 44 3 0 0 php-fpm
  221. Jun 13 12:38:00 online-node-81-113 kernel: [23357] 0 23357 3480 1069 13 3 0 0 nginx
  222. Jun 13 12:38:00 online-node-81-113 kernel: [23358] 7373 23358 3540 453 11 3 0 0 nginx
  223. Jun 13 12:38:00 online-node-81-113 kernel: [23359] 7373 23359 3540 453 11 3 0 0 nginx
  224. Jun 13 12:38:00 online-node-81-113 kernel: [23365] 7373 23365 62710 8467 48 3 0 0 php-fpm
  225. Jun 13 12:38:00 online-node-81-113 kernel: [23367] 7373 23367 49845 1346 30 3 0 0 php-fpm
  226. Jun 13 12:38:00 online-node-81-113 kernel: [23368] 7373 23368 49845 1346 30 3 0 0 php-fpm
  227. Jun 13 12:38:00 online-node-81-113 kernel: [23369] 7373 23369 49845 1346 30 3 0 0 php-fpm
  228. Jun 13 12:38:00 online-node-81-113 kernel: [23370] 7373 23370 49846 1346 30 3 0 0 php-fpm
  229. Jun 13 12:38:00 online-node-81-113 kernel: [23371] 7373 23371 49846 1346 30 3 0 0 php-fpm
  230. Jun 13 12:38:00 online-node-81-113 kernel: [23372] 7373 23372 49846 1346 30 3 0 0 php-fpm
  231. Jun 13 12:38:00 online-node-81-113 kernel: [23373] 7373 23373 49846 1346 30 3 0 0 php-fpm
  232. Jun 13 12:38:00 online-node-81-113 kernel: [23374] 7373 23374 49846 1347 30 3 0 0 php-fpm
  233. Jun 13 12:38:00 online-node-81-113 kernel: [23375] 7373 23375 49846 1347 30 3 0 0 php-fpm
  234. Jun 13 12:38:00 online-node-81-113 kernel: [23376] 7373 23376 49846 1347 30 3 0 0 php-fpm
  235. Jun 13 12:38:00 online-node-81-113 kernel: [23377] 7373 23377 49846 1347 30 3 0 0 php-fpm
  236. Jun 13 12:38:00 online-node-81-113 kernel: [23378] 7373 23378 49846 1347 30 3 0 0 php-fpm
  237. Jun 13 12:38:00 online-node-81-113 kernel: [23379] 7373 23379 49846 1347 30 3 0 0 php-fpm
  238. Jun 13 12:38:00 online-node-81-113 kernel: [23380] 7373 23380 49846 1347 30 3 0 0 php-fpm
  239. Jun 13 12:38:00 online-node-81-113 kernel: [23381] 7373 23381 49846 1347 30 3 0 0 php-fpm
  240. Jun 13 12:38:00 online-node-81-113 kernel: [26458] 0 26458 393 262 7 3 0 0 sh
  241. Jun 13 12:38:00 online-node-81-113 kernel: [ 499] 7373 499 49846 1347 30 3 0 0 php-fpm
  242. Jun 13 12:38:00 online-node-81-113 kernel: [ 2710] 7373 2710 49846 1347 30 3 0 0 php-fpm
  243. Jun 13 12:38:00 online-node-81-113 kernel: [ 3329] 7373 3329 49846 1347 30 3 0 0 php-fpm
  244. Jun 13 12:38:00 online-node-81-113 kernel: [ 3808] 7373 3808 49846 1347 30 3 0 0 php-fpm
  245. Jun 13 12:38:00 online-node-81-113 kernel: [ 3962] 0 3962 3524 799 12 3 0 0 curl
  246. Jun 13 12:38:00 online-node-81-113 kernel: Memory cgroup out of memory: Kill process 23365 (php-fpm) score 594 or sacrifice child
  247. Jun 13 12:38:00 online-node-81-113 kernel: Killed process 23365 (php-fpm) total-vm:250840kB, anon-rss:29532kB, file-rss:4160kB, shmem-rss:176kB
  248. Jun 13 12:38:00 online-node-81-113 kernel: oom_reaper: reaped process 23365 (php-fpm), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
  249. Jun 13 12:38:01 online-node-81-113 kernel: php-fpm invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=0
  250. Jun 13 12:38:01 online-node-81-113 kernel: php-fpm cpuset=aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d mems_allowed=0-1
  251. Jun 13 12:38:01 online-node-81-113 kernel: CPU: 25 PID: 23367 Comm: php-fpm Not tainted 4.14.15-1.el7.elrepo.x86_64 #1
  252. Jun 13 12:38:01 online-node-81-113 kernel: Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.3.4 11/08/2016
  253. Jun 13 12:38:01 online-node-81-113 kernel: Call Trace:
  254. Jun 13 12:38:01 online-node-81-113 kernel: dump_stack+0x63/0x85
  255. Jun 13 12:38:01 online-node-81-113 kernel: dump_header+0x9f/0x234
  256. Jun 13 12:38:01 online-node-81-113 kernel: ? mem_cgroup_scan_tasks+0x96/0xf0
  257. Jun 13 12:38:01 online-node-81-113 kernel: oom_kill_process+0x21c/0x430
  258. Jun 13 12:38:01 online-node-81-113 kernel: out_of_memory+0x114/0x4a0
  259. Jun 13 12:38:01 online-node-81-113 kernel: mem_cgroup_out_of_memory+0x4b/0x80
  260. Jun 13 12:38:01 online-node-81-113 kernel: mem_cgroup_oom_synchronize+0x2f9/0x320
  261. Jun 13 12:38:01 online-node-81-113 kernel: ? get_mctgt_type_thp.isra.30+0xc0/0xc0
  262. Jun 13 12:38:01 online-node-81-113 kernel: pagefault_out_of_memory+0x36/0x7c
  263. Jun 13 12:38:01 online-node-81-113 kernel: mm_fault_error+0x65/0x152
  264. Jun 13 12:38:01 online-node-81-113 kernel: __do_page_fault+0x456/0x4f0
  265. Jun 13 12:38:01 online-node-81-113 kernel: do_page_fault+0x38/0x130
  266. Jun 13 12:38:01 online-node-81-113 kernel: ? page_fault+0x36/0x60
  267. Jun 13 12:38:01 online-node-81-113 kernel: page_fault+0x4c/0x60
  268. Jun 13 12:38:01 online-node-81-113 kernel: RIP: 0033:0x7f07230666bf
  269. Jun 13 12:38:01 online-node-81-113 kernel: RSP: 002b:00007ffceb7f1fd8 EFLAGS: 00010206
  270. Jun 13 12:38:01 online-node-81-113 kernel: RAX: 00007f0716179040 RBX: 0000000001400000 RCX: 0000000000209008
  271. Jun 13 12:38:01 online-node-81-113 kernel: RDX: 0000000001400000 RSI: 00007f0715131000 RDI: 00007f0716531000
  272. Jun 13 12:38:01 online-node-81-113 kernel: RBP: 00007f0723296b50 R08: 0000000003240000 R09: 00007f0717f79048
  273. Jun 13 12:38:01 online-node-81-113 kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 00007f0714d79040
  274. Jun 13 12:38:01 online-node-81-113 kernel: R13: 00007f0717f79040 R14: 0000000003200000 R15: 000055b13d705c60
  275. Jun 13 12:38:01 online-node-81-113 kernel: Task in /docker/aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d killed as a result of limit of /docker/aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d
  276. Jun 13 12:38:01 online-node-81-113 kernel: memory: usage 57344kB, limit 57344kB, failcnt 240
  277. Jun 13 12:38:01 online-node-81-113 kernel: memory+swap: usage 57344kB, limit 114688kB, failcnt 0
  278. Jun 13 12:38:01 online-node-81-113 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
  279. Jun 13 12:38:01 online-node-81-113 kernel: Memory cgroup stats for /docker/aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d: cache:10752KB rss:46592KB rss_huge:22528KB shmem:10716KB mapped_file:10716KB dirty:12KB writeback:0KB swap:0KB inactive_anon:10540KB active_anon:46744KB inactive_file:24KB active_file:12KB unevictable:0KB
  280. Jun 13 12:38:01 online-node-81-113 kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
  281. Jun 13 12:38:01 online-node-81-113 kernel: [22551] 0 22551 21395 4336 47 3 0 0 supervisord
  282. Jun 13 12:38:01 online-node-81-113 kernel: [23356] 0 23356 49846 6764 44 3 0 0 php-fpm
  283. Jun 13 12:38:01 online-node-81-113 kernel: [23357] 0 23357 3480 1069 13 3 0 0 nginx
  284. Jun 13 12:38:01 online-node-81-113 kernel: [23358] 7373 23358 3540 453 11 3 0 0 nginx
  285. Jun 13 12:38:01 online-node-81-113 kernel: [23359] 7373 23359 3540 453 11 3 0 0 nginx
  286. Jun 13 12:38:01 online-node-81-113 kernel: [23367] 7373 23367 62710 8467 48 3 0 0 php-fpm
  287. Jun 13 12:38:01 online-node-81-113 kernel: [23368] 7373 23368 49845 1346 30 3 0 0 php-fpm
  288. Jun 13 12:38:01 online-node-81-113 kernel: [23369] 7373 23369 49845 1346 30 3 0 0 php-fpm
  289. Jun 13 12:38:01 online-node-81-113 kernel: [23370] 7373 23370 49846 1346 30 3 0 0 php-fpm
  290. Jun 13 12:38:01 online-node-81-113 kernel: [23371] 7373 23371 49846 1346 30 3 0 0 php-fpm
  291. Jun 13 12:38:01 online-node-81-113 kernel: [23372] 7373 23372 49846 1346 30 3 0 0 php-fpm
  292. Jun 13 12:38:01 online-node-81-113 kernel: [23373] 7373 23373 49846 1346 30 3 0 0 php-fpm
  293. Jun 13 12:38:01 online-node-81-113 kernel: [23374] 7373 23374 49846 1347 30 3 0 0 php-fpm
  294. Jun 13 12:38:01 online-node-81-113 kernel: [23375] 7373 23375 49846 1347 30 3 0 0 php-fpm
  295. Jun 13 12:38:01 online-node-81-113 kernel: [23376] 7373 23376 49846 1347 30 3 0 0 php-fpm
  296. Jun 13 12:38:01 online-node-81-113 kernel: [23377] 7373 23377 49846 1347 30 3 0 0 php-fpm
  297. Jun 13 12:38:01 online-node-81-113 kernel: [23378] 7373 23378 49846 1347 30 3 0 0 php-fpm
  298. Jun 13 12:38:01 online-node-81-113 kernel: [23379] 7373 23379 49846 1347 30 3 0 0 php-fpm
  299. Jun 13 12:38:01 online-node-81-113 kernel: [23380] 7373 23380 49846 1347 30 3 0 0 php-fpm
  300. Jun 13 12:38:01 online-node-81-113 kernel: [23381] 7373 23381 49846 1347 30 3 0 0 php-fpm
  301. Jun 13 12:38:01 online-node-81-113 kernel: [26458] 0 26458 393 262 7 3 0 0 sh
  302. Jun 13 12:38:01 online-node-81-113 kernel: [ 499] 7373 499 49846 1347 30 3 0 0 php-fpm
  303. Jun 13 12:38:01 online-node-81-113 kernel: [ 2710] 7373 2710 49846 1347 30 3 0 0 php-fpm
  304. Jun 13 12:38:01 online-node-81-113 kernel: [ 3329] 7373 3329 49846 1347 30 3 0 0 php-fpm
  305. Jun 13 12:38:01 online-node-81-113 kernel: [ 3808] 7373 3808 49846 1347 30 3 0 0 php-fpm
  306. Jun 13 12:38:01 online-node-81-113 kernel: [ 3978] 7373 3978 49846 1347 30 3 0 0 php-fpm
  307. Jun 13 12:38:01 online-node-81-113 kernel: [ 4495] 0 4495 3524 818 13 4 0 0 curl
  308. Jun 13 12:38:01 online-node-81-113 kernel: Memory cgroup out of memory: Kill process 23367 (php-fpm) score 594 or sacrifice child
  309. Jun 13 12:38:01 online-node-81-113 kernel: Killed process 23367 (php-fpm) total-vm:250840kB, anon-rss:29532kB, file-rss:4160kB, shmem-rss:176kB
  310. Jun 13 12:38:01 online-node-81-113 kernel: oom_reaper: reaped process 23367 (php-fpm), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
  311. Jun 13 12:38:09 online-node-81-113 kernel: php invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=0
  312. Jun 13 12:38:09 online-node-81-113 kernel: php cpuset=aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d mems_allowed=0-1
  313. Jun 13 12:38:09 online-node-81-113 kernel: CPU: 31 PID: 9743 Comm: php Not tainted 4.14.15-1.el7.elrepo.x86_64 #1
  314. Jun 13 12:38:09 online-node-81-113 kernel: Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.3.4 11/08/2016
  315. Jun 13 12:38:09 online-node-81-113 kernel: Call Trace:
  316. Jun 13 12:38:09 online-node-81-113 kernel: dump_stack+0x63/0x85
  317. Jun 13 12:38:09 online-node-81-113 kernel: dump_header+0x9f/0x234
  318. Jun 13 12:38:09 online-node-81-113 kernel: ? mem_cgroup_scan_tasks+0x96/0xf0
  319. Jun 13 12:38:09 online-node-81-113 kernel: oom_kill_process+0x21c/0x430
  320. Jun 13 12:38:09 online-node-81-113 kernel: out_of_memory+0x114/0x4a0
  321. Jun 13 12:38:09 online-node-81-113 kernel: mem_cgroup_out_of_memory+0x4b/0x80
  322. Jun 13 12:38:09 online-node-81-113 kernel: mem_cgroup_oom_synchronize+0x2f9/0x320
  323. Jun 13 12:38:09 online-node-81-113 kernel: ? get_mctgt_type_thp.isra.30+0xc0/0xc0
  324. Jun 13 12:38:09 online-node-81-113 kernel: pagefault_out_of_memory+0x36/0x7c
  325. Jun 13 12:38:09 online-node-81-113 kernel: mm_fault_error+0x65/0x152
  326. Jun 13 12:38:09 online-node-81-113 kernel: __do_page_fault+0x456/0x4f0
  327. Jun 13 12:38:09 online-node-81-113 kernel: do_page_fault+0x38/0x130
  328. Jun 13 12:38:09 online-node-81-113 kernel: ? page_fault+0x36/0x60
  329. Jun 13 12:38:09 online-node-81-113 kernel: page_fault+0x4c/0x60
  330. Jun 13 12:38:09 online-node-81-113 kernel: RIP: 0033:0x7f5c2c9186bf
  331. Jun 13 12:38:09 online-node-81-113 kernel: RSP: 002b:00007ffe81b456c8 EFLAGS: 00010206
  332. Jun 13 12:38:09 online-node-81-113 kernel: RAX: 00007f5c2702b040 RBX: 0000000000a00000 RCX: 0000000000020a08
  333. Jun 13 12:38:09 online-node-81-113 kernel: RDX: 0000000000a00000 RSI: 00007f5c26f26000 RDI: 00007f5c27926000
  334. Jun 13 12:38:09 online-node-81-113 kernel: RBP: 00007f5c2cb45480 R08: 0000000003240000 R09: 00007f5c2982b048
  335. Jun 13 12:38:09 online-node-81-113 kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 00007f5c2662b040
  336. Jun 13 12:38:09 online-node-81-113 kernel: R13: 00007f5c2982b040 R14: 0000000003200000 R15: 000055f69db1b9e0
  337. Jun 13 12:38:09 online-node-81-113 kernel: Task in /docker/aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d killed as a result of limit of /docker/aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d
  338. Jun 13 12:38:09 online-node-81-113 kernel: memory: usage 57344kB, limit 57344kB, failcnt 288
  339. Jun 13 12:38:09 online-node-81-113 kernel: memory+swap: usage 57344kB, limit 114688kB, failcnt 0
  340. Jun 13 12:38:09 online-node-81-113 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
  341. Jun 13 12:38:09 online-node-81-113 kernel: Memory cgroup stats for /docker/aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d: cache:10760KB rss:46584KB rss_huge:16384KB shmem:10716KB mapped_file:10708KB dirty:16KB writeback:0KB swap:0KB inactive_anon:10540KB active_anon:46708KB inactive_file:24KB active_file:12KB unevictable:0KB
  342. Jun 13 12:38:09 online-node-81-113 kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
  343. Jun 13 12:38:09 online-node-81-113 kernel: [22551] 0 22551 21395 4336 47 3 0 0 supervisord
  344. Jun 13 12:38:09 online-node-81-113 kernel: [23356] 0 23356 49846 6764 44 3 0 0 php-fpm
  345. Jun 13 12:38:09 online-node-81-113 kernel: [23357] 0 23357 3480 1069 13 3 0 0 nginx
  346. Jun 13 12:38:09 online-node-81-113 kernel: [23358] 7373 23358 3540 453 11 3 0 0 nginx
  347. Jun 13 12:38:09 online-node-81-113 kernel: [23359] 7373 23359 3540 453 11 3 0 0 nginx
  348. Jun 13 12:38:09 online-node-81-113 kernel: [23368] 7373 23368 49845 1346 30 3 0 0 php-fpm
  349. Jun 13 12:38:09 online-node-81-113 kernel: [23369] 7373 23369 49845 1346 30 3 0 0 php-fpm
  350. Jun 13 12:38:09 online-node-81-113 kernel: [23370] 7373 23370 49846 1346 30 3 0 0 php-fpm
  351. Jun 13 12:38:09 online-node-81-113 kernel: [23371] 7373 23371 49846 1346 30 3 0 0 php-fpm
  352. Jun 13 12:38:09 online-node-81-113 kernel: [23372] 7373 23372 49846 1346 30 3 0 0 php-fpm
  353. Jun 13 12:38:09 online-node-81-113 kernel: [23373] 7373 23373 49846 1346 30 3 0 0 php-fpm
  354. Jun 13 12:38:09 online-node-81-113 kernel: [23374] 7373 23374 49846 1347 30 3 0 0 php-fpm
  355. Jun 13 12:38:09 online-node-81-113 kernel: [23375] 7373 23375 49846 1347 30 3 0 0 php-fpm
  356. Jun 13 12:38:09 online-node-81-113 kernel: [23376] 7373 23376 49846 1347 30 3 0 0 php-fpm
  357. Jun 13 12:38:09 online-node-81-113 kernel: [23377] 7373 23377 49846 1347 30 3 0 0 php-fpm
  358. Jun 13 12:38:09 online-node-81-113 kernel: [23378] 7373 23378 49846 1347 30 3 0 0 php-fpm
  359. Jun 13 12:38:09 online-node-81-113 kernel: [23379] 7373 23379 49846 1347 30 3 0 0 php-fpm
  360. Jun 13 12:38:09 online-node-81-113 kernel: [23380] 7373 23380 49846 1347 30 3 0 0 php-fpm
  361. Jun 13 12:38:09 online-node-81-113 kernel: [23381] 7373 23381 49846 1347 30 3 0 0 php-fpm
  362. Jun 13 12:38:09 online-node-81-113 kernel: [26458] 0 26458 393 262 7 3 0 0 sh
  363. Jun 13 12:38:09 online-node-81-113 kernel: [ 499] 7373 499 49846 1347 30 3 0 0 php-fpm
  364. Jun 13 12:38:09 online-node-81-113 kernel: [ 2710] 7373 2710 49846 1347 30 3 0 0 php-fpm
  365. Jun 13 12:38:09 online-node-81-113 kernel: [ 3329] 7373 3329 49846 1347 30 3 0 0 php-fpm
  366. Jun 13 12:38:09 online-node-81-113 kernel: [ 3808] 7373 3808 49846 1347 30 3 0 0 php-fpm
  367. Jun 13 12:38:09 online-node-81-113 kernel: [ 3978] 7373 3978 49846 1347 30 3 0 0 php-fpm
  368. Jun 13 12:38:09 online-node-81-113 kernel: [ 4496] 7373 4496 49846 1347 30 3 0 0 php-fpm
  369. Jun 13 12:38:09 online-node-81-113 kernel: [ 9743] 0 9743 29888 8999 43 3 0 0 php
  370. Jun 13 12:38:09 online-node-81-113 kernel: Memory cgroup out of memory: Kill process 9743 (php) score 630 or sacrifice child
  371. Jun 13 12:38:09 online-node-81-113 kernel: Killed process 9743 (php) total-vm:119552kB, anon-rss:24640kB, file-rss:11356kB, shmem-rss:0kB
  372. Jun 13 12:38:09 online-node-81-113 kernel: oom_reaper: reaped process 9743 (php), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
  373. Jun 13 12:38:28 online-node-81-113 kernel: php invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=0
  374. Jun 13 12:38:28 online-node-81-113 kernel: php cpuset=aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d mems_allowed=0-1
  375. Jun 13 12:38:28 online-node-81-113 kernel: CPU: 22 PID: 20967 Comm: php Not tainted 4.14.15-1.el7.elrepo.x86_64 #1
  376. Jun 13 12:38:28 online-node-81-113 kernel: Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.3.4 11/08/2016
  377. Jun 13 12:38:28 online-node-81-113 kernel: Call Trace:
  378. Jun 13 12:38:28 online-node-81-113 kernel: dump_stack+0x63/0x85
  379. Jun 13 12:38:28 online-node-81-113 kernel: dump_header+0x9f/0x234
  380. Jun 13 12:38:28 online-node-81-113 kernel: ? mem_cgroup_scan_tasks+0x96/0xf0
  381. Jun 13 12:38:28 online-node-81-113 kernel: oom_kill_process+0x21c/0x430
  382. Jun 13 12:38:28 online-node-81-113 kernel: out_of_memory+0x114/0x4a0
  383. Jun 13 12:38:28 online-node-81-113 kernel: mem_cgroup_out_of_memory+0x4b/0x80
  384. Jun 13 12:38:28 online-node-81-113 kernel: mem_cgroup_oom_synchronize+0x2f9/0x320
  385. Jun 13 12:38:28 online-node-81-113 kernel: ? get_mctgt_type_thp.isra.30+0xc0/0xc0
  386. Jun 13 12:38:28 online-node-81-113 kernel: pagefault_out_of_memory+0x36/0x7c
  387. Jun 13 12:38:28 online-node-81-113 kernel: mm_fault_error+0x65/0x152
  388. Jun 13 12:38:28 online-node-81-113 kernel: __do_page_fault+0x456/0x4f0
  389. Jun 13 12:38:28 online-node-81-113 kernel: do_page_fault+0x38/0x130
  390. Jun 13 12:38:28 online-node-81-113 kernel: ? page_fault+0x36/0x60
  391. Jun 13 12:38:28 online-node-81-113 kernel: page_fault+0x4c/0x60
  392. Jun 13 12:38:28 online-node-81-113 kernel: RIP: 0033:0x7f163e46b6bf
  393. Jun 13 12:38:28 online-node-81-113 kernel: RSP: 002b:00007ffc40ef1758 EFLAGS: 00010206
  394. Jun 13 12:38:28 online-node-81-113 kernel: RAX: 00007f1638b7e040 RBX: 0000000000a00000 RCX: 0000000000020008
  395. Jun 13 12:38:28 online-node-81-113 kernel: RDX: 0000000000a00000 RSI: 00007f1638a7e000 RDI: 00007f163947e000
  396. Jun 13 12:38:28 online-node-81-113 kernel: RBP: 00007f163e698480 R08: 0000000003240000 R09: 00007f163b37e048
  397. Jun 13 12:38:28 online-node-81-113 kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 00007f163817e040
  398. Jun 13 12:38:28 online-node-81-113 kernel: R13: 00007f163b37e040 R14: 0000000003200000 R15: 000055d9a60529e0
  399. Jun 13 12:38:28 online-node-81-113 kernel: Task in /docker/aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d killed as a result of limit of /docker/aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d
  400. Jun 13 12:38:28 online-node-81-113 kernel: memory: usage 57344kB, limit 57344kB, failcnt 314
  401. Jun 13 12:38:28 online-node-81-113 kernel: memory+swap: usage 57344kB, limit 114688kB, failcnt 0
  402. Jun 13 12:38:28 online-node-81-113 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
  403. Jun 13 12:38:28 online-node-81-113 kernel: Memory cgroup stats for /docker/aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d: cache:10744KB rss:46600KB rss_huge:18432KB shmem:10716KB mapped_file:10708KB dirty:0KB writeback:0KB swap:0KB inactive_anon:10540KB active_anon:46776KB inactive_file:24KB active_file:4KB unevictable:0KB
  404. Jun 13 12:38:28 online-node-81-113 kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
  405. Jun 13 12:38:28 online-node-81-113 kernel: [22551] 0 22551 21395 4336 47 3 0 0 supervisord
  406. Jun 13 12:38:28 online-node-81-113 kernel: [23356] 0 23356 49846 6764 44 3 0 0 php-fpm
  407. Jun 13 12:38:28 online-node-81-113 kernel: [23357] 0 23357 3480 1069 13 3 0 0 nginx
  408. Jun 13 12:38:28 online-node-81-113 kernel: [23358] 7373 23358 3540 453 11 3 0 0 nginx
  409. Jun 13 12:38:28 online-node-81-113 kernel: [23359] 7373 23359 3540 453 11 3 0 0 nginx
  410. Jun 13 12:38:28 online-node-81-113 kernel: [23368] 7373 23368 49845 1346 30 3 0 0 php-fpm
  411. Jun 13 12:38:28 online-node-81-113 kernel: [23369] 7373 23369 49845 1346 30 3 0 0 php-fpm
  412. Jun 13 12:38:28 online-node-81-113 kernel: [23370] 7373 23370 49846 1346 30 3 0 0 php-fpm
  413. Jun 13 12:38:28 online-node-81-113 kernel: [23371] 7373 23371 49846 1346 30 3 0 0 php-fpm
  414. Jun 13 12:38:28 online-node-81-113 kernel: [23372] 7373 23372 49846 1346 30 3 0 0 php-fpm
  415. Jun 13 12:38:28 online-node-81-113 kernel: [23373] 7373 23373 49846 1346 30 3 0 0 php-fpm
  416. Jun 13 12:38:28 online-node-81-113 kernel: [23374] 7373 23374 49846 1347 30 3 0 0 php-fpm
  417. Jun 13 12:38:28 online-node-81-113 kernel: [23375] 7373 23375 49846 1347 30 3 0 0 php-fpm
  418. Jun 13 12:38:28 online-node-81-113 kernel: [23376] 7373 23376 49846 1347 30 3 0 0 php-fpm
  419. Jun 13 12:38:28 online-node-81-113 kernel: [23377] 7373 23377 49846 1347 30 3 0 0 php-fpm
  420. Jun 13 12:38:28 online-node-81-113 kernel: [23378] 7373 23378 49846 1347 30 3 0 0 php-fpm
  421. Jun 13 12:38:28 online-node-81-113 kernel: [23379] 7373 23379 49846 1347 30 3 0 0 php-fpm
  422. Jun 13 12:38:28 online-node-81-113 kernel: [23380] 7373 23380 49846 1347 30 3 0 0 php-fpm
  423. Jun 13 12:38:28 online-node-81-113 kernel: [23381] 7373 23381 49846 1347 30 3 0 0 php-fpm
  424. Jun 13 12:38:28 online-node-81-113 kernel: [26458] 0 26458 393 262 7 3 0 0 sh
  425. Jun 13 12:38:28 online-node-81-113 kernel: [ 499] 7373 499 49846 1347 30 3 0 0 php-fpm
  426. Jun 13 12:38:28 online-node-81-113 kernel: [ 2710] 7373 2710 49846 1347 30 3 0 0 php-fpm
  427. Jun 13 12:38:28 online-node-81-113 kernel: [ 3329] 7373 3329 49846 1347 30 3 0 0 php-fpm
  428. Jun 13 12:38:28 online-node-81-113 kernel: [ 3808] 7373 3808 49846 1347 30 3 0 0 php-fpm
  429. Jun 13 12:38:28 online-node-81-113 kernel: [ 3978] 7373 3978 49846 1347 30 3 0 0 php-fpm
  430. Jun 13 12:38:28 online-node-81-113 kernel: [ 4496] 7373 4496 49846 1347 30 3 0 0 php-fpm
  431. Jun 13 12:38:28 online-node-81-113 kernel: [20967] 0 20967 29888 9041 50 3 0 0 php
  432. Jun 13 12:38:28 online-node-81-113 kernel: Memory cgroup out of memory: Kill process 20967 (php) score 634 or sacrifice child
  433. Jun 13 12:38:28 online-node-81-113 kernel: Killed process 20967 (php) total-vm:119552kB, anon-rss:24828kB, file-rss:11336kB, shmem-rss:0kB
  434. Jun 13 12:38:28 online-node-81-113 kernel: oom_reaper: reaped process 20967 (php), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
  435. Jun 13 12:38:40 online-node-81-113 kernel: php invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=0
  436. Jun 13 12:38:40 online-node-81-113 kernel: php cpuset=aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d mems_allowed=0-1
  437. Jun 13 12:38:40 online-node-81-113 kernel: CPU: 25 PID: 27581 Comm: php Not tainted 4.14.15-1.el7.elrepo.x86_64 #1
  438. Jun 13 12:38:40 online-node-81-113 kernel: Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.3.4 11/08/2016
  439. Jun 13 12:38:40 online-node-81-113 kernel: Call Trace:
  440. Jun 13 12:38:40 online-node-81-113 kernel: dump_stack+0x63/0x85
  441. Jun 13 12:38:40 online-node-81-113 kernel: dump_header+0x9f/0x234
  442. Jun 13 12:38:40 online-node-81-113 kernel: ? mem_cgroup_scan_tasks+0x96/0xf0
  443. Jun 13 12:38:40 online-node-81-113 kernel: oom_kill_process+0x21c/0x430
  444. Jun 13 12:38:40 online-node-81-113 kernel: out_of_memory+0x114/0x4a0
  445. Jun 13 12:38:40 online-node-81-113 kernel: mem_cgroup_out_of_memory+0x4b/0x80
  446. Jun 13 12:38:40 online-node-81-113 kernel: mem_cgroup_oom_synchronize+0x2f9/0x320
  447. Jun 13 12:38:40 online-node-81-113 kernel: ? get_mctgt_type_thp.isra.30+0xc0/0xc0
  448. Jun 13 12:38:40 online-node-81-113 kernel: pagefault_out_of_memory+0x36/0x7c
  449. Jun 13 12:38:40 online-node-81-113 kernel: mm_fault_error+0x65/0x152
  450. Jun 13 12:38:40 online-node-81-113 kernel: __do_page_fault+0x456/0x4f0
  451. Jun 13 12:38:40 online-node-81-113 kernel: do_page_fault+0x38/0x130
  452. Jun 13 12:38:40 online-node-81-113 kernel: ? page_fault+0x36/0x60
  453. Jun 13 12:38:40 online-node-81-113 kernel: page_fault+0x4c/0x60
  454. Jun 13 12:38:40 online-node-81-113 kernel: RIP: 0033:0x7feb18b646bf
  455. Jun 13 12:38:40 online-node-81-113 kernel: RSP: 002b:00007ffd0f0c7098 EFLAGS: 00010206
  456. Jun 13 12:38:40 online-node-81-113 kernel: RAX: 00007feb13277040 RBX: 0000000000a00000 RCX: 0000000000020208
  457. Jun 13 12:38:40 online-node-81-113 kernel: RDX: 0000000000a00000 RSI: 00007feb13176000 RDI: 00007feb13b76000
  458. Jun 13 12:38:40 online-node-81-113 kernel: RBP: 00007feb18d91480 R08: 0000000003240000 R09: 00007feb15a77048
  459. Jun 13 12:38:40 online-node-81-113 kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 00007feb12877040
  460. Jun 13 12:38:40 online-node-81-113 kernel: R13: 00007feb15a77040 R14: 0000000003200000 R15: 00005560c61299e0
  461. Jun 13 12:38:40 online-node-81-113 kernel: Task in /docker/aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d killed as a result of limit of /docker/aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d
  462. Jun 13 12:38:40 online-node-81-113 kernel: memory: usage 57344kB, limit 57344kB, failcnt 365
  463. Jun 13 12:38:40 online-node-81-113 kernel: memory+swap: usage 57344kB, limit 114688kB, failcnt 0
  464. Jun 13 12:38:40 online-node-81-113 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
  465. Jun 13 12:38:40 online-node-81-113 kernel: Memory cgroup stats for /docker/aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d: cache:10744KB rss:46600KB rss_huge:16384KB shmem:10716KB mapped_file:10708KB dirty:0KB writeback:0KB swap:0KB inactive_anon:10540KB active_anon:46772KB inactive_file:24KB active_file:4KB unevictable:0KB
  466. Jun 13 12:38:40 online-node-81-113 kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
  467. Jun 13 12:38:40 online-node-81-113 kernel: [22551] 0 22551 21395 4336 47 3 0 0 supervisord
  468. Jun 13 12:38:40 online-node-81-113 kernel: [23356] 0 23356 49846 6764 44 3 0 0 php-fpm
  469. Jun 13 12:38:40 online-node-81-113 kernel: [23357] 0 23357 3480 1069 13 3 0 0 nginx
  470. Jun 13 12:38:40 online-node-81-113 kernel: [23358] 7373 23358 3540 453 11 3 0 0 nginx
  471. Jun 13 12:38:40 online-node-81-113 kernel: [23359] 7373 23359 3540 453 11 3 0 0 nginx
  472. Jun 13 12:38:40 online-node-81-113 kernel: [23368] 7373 23368 49845 1346 30 3 0 0 php-fpm
  473. Jun 13 12:38:40 online-node-81-113 kernel: [23369] 7373 23369 49845 1346 30 3 0 0 php-fpm
  474. Jun 13 12:38:40 online-node-81-113 kernel: [23370] 7373 23370 49846 1346 30 3 0 0 php-fpm
  475. Jun 13 12:38:40 online-node-81-113 kernel: [23371] 7373 23371 49846 1346 30 3 0 0 php-fpm
  476. Jun 13 12:38:40 online-node-81-113 kernel: [23372] 7373 23372 49846 1346 30 3 0 0 php-fpm
  477. Jun 13 12:38:40 online-node-81-113 kernel: [23373] 7373 23373 49846 1346 30 3 0 0 php-fpm
  478. Jun 13 12:38:40 online-node-81-113 kernel: [23374] 7373 23374 49846 1347 30 3 0 0 php-fpm
  479. Jun 13 12:38:40 online-node-81-113 kernel: [23375] 7373 23375 49846 1347 30 3 0 0 php-fpm
  480. Jun 13 12:38:40 online-node-81-113 kernel: [23376] 7373 23376 49846 1347 30 3 0 0 php-fpm
  481. Jun 13 12:38:40 online-node-81-113 kernel: [23377] 7373 23377 49846 1347 30 3 0 0 php-fpm
  482. Jun 13 12:38:40 online-node-81-113 kernel: [23378] 7373 23378 49846 1347 30 3 0 0 php-fpm
  483. Jun 13 12:38:40 online-node-81-113 kernel: [23379] 7373 23379 49846 1347 30 3 0 0 php-fpm
  484. Jun 13 12:38:40 online-node-81-113 kernel: [23380] 7373 23380 49846 1347 30 3 0 0 php-fpm
  485. Jun 13 12:38:40 online-node-81-113 kernel: [23381] 7373 23381 49846 1347 30 3 0 0 php-fpm
  486. Jun 13 12:38:40 online-node-81-113 kernel: [26458] 0 26458 393 262 7 3 0 0 sh
  487. Jun 13 12:38:40 online-node-81-113 kernel: [ 499] 7373 499 49846 1347 30 3 0 0 php-fpm
  488. Jun 13 12:38:40 online-node-81-113 kernel: [ 2710] 7373 2710 49846 1347 30 3 0 0 php-fpm
  489. Jun 13 12:38:40 online-node-81-113 kernel: [ 3329] 7373 3329 49846 1347 30 3 0 0 php-fpm
  490. Jun 13 12:38:40 online-node-81-113 kernel: [ 3808] 7373 3808 49846 1347 30 3 0 0 php-fpm
  491. Jun 13 12:38:40 online-node-81-113 kernel: [ 3978] 7373 3978 49846 1347 30 3 0 0 php-fpm
  492. Jun 13 12:38:40 online-node-81-113 kernel: [ 4496] 7373 4496 49846 1347 30 3 0 0 php-fpm
  493. Jun 13 12:38:40 online-node-81-113 kernel: [27581] 0 27581 29888 9038 45 3 0 0 php
  494. Jun 13 12:38:40 online-node-81-113 kernel: Memory cgroup out of memory: Kill process 27581 (php) score 633 or sacrifice child
  495. Jun 13 12:38:40 online-node-81-113 kernel: Killed process 27581 (php) total-vm:119552kB, anon-rss:24636kB, file-rss:11516kB, shmem-rss:0kB
  496. Jun 13 12:38:40 online-node-81-113 kernel: oom_reaper: reaped process 27581 (php), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
  497. Jun 13 12:38:42 online-node-81-113 kernel: php invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=0
  498. Jun 13 12:38:42 online-node-81-113 kernel: php cpuset=aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d mems_allowed=0-1
  499. Jun 13 12:38:42 online-node-81-113 kernel: CPU: 37 PID: 28534 Comm: php Not tainted 4.14.15-1.el7.elrepo.x86_64 #1
  500. Jun 13 12:38:42 online-node-81-113 kernel: Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.3.4 11/08/2016
  501. Jun 13 12:38:42 online-node-81-113 kernel: Call Trace:
  502. Jun 13 12:38:42 online-node-81-113 kernel: dump_stack+0x63/0x85
  503. Jun 13 12:38:42 online-node-81-113 kernel: dump_header+0x9f/0x234
  504. Jun 13 12:38:42 online-node-81-113 kernel: ? mem_cgroup_scan_tasks+0x96/0xf0
  505. Jun 13 12:38:42 online-node-81-113 kernel: oom_kill_process+0x21c/0x430
  506. Jun 13 12:38:42 online-node-81-113 kernel: out_of_memory+0x114/0x4a0
  507. Jun 13 12:38:42 online-node-81-113 kernel: mem_cgroup_out_of_memory+0x4b/0x80
  508. Jun 13 12:38:42 online-node-81-113 kernel: mem_cgroup_oom_synchronize+0x2f9/0x320
  509. Jun 13 12:38:42 online-node-81-113 kernel: ? get_mctgt_type_thp.isra.30+0xc0/0xc0
  510. Jun 13 12:38:42 online-node-81-113 kernel: pagefault_out_of_memory+0x36/0x7c
  511. Jun 13 12:38:42 online-node-81-113 kernel: mm_fault_error+0x65/0x152
  512. Jun 13 12:38:42 online-node-81-113 kernel: __do_page_fault+0x456/0x4f0
  513. Jun 13 12:38:42 online-node-81-113 kernel: do_page_fault+0x38/0x130
  514. Jun 13 12:38:42 online-node-81-113 kernel: ? page_fault+0x36/0x60
  515. Jun 13 12:38:42 online-node-81-113 kernel: page_fault+0x4c/0x60
  516. Jun 13 12:38:42 online-node-81-113 kernel: RIP: 0033:0x7f0c4be486bf
  517. Jun 13 12:38:42 online-node-81-113 kernel: RSP: 002b:00007fff9e9efac8 EFLAGS: 00010206
  518. Jun 13 12:38:42 online-node-81-113 kernel: RAX: 00007f0c4655b040 RBX: 0000000000a00000 RCX: 0000000000020008
  519. Jun 13 12:38:42 online-node-81-113 kernel: RDX: 0000000000a00000 RSI: 00007f0c4645b000 RDI: 00007f0c46e5b000
  520. Jun 13 12:38:42 online-node-81-113 kernel: RBP: 00007f0c4c075480 R08: 0000000003240000 R09: 00007f0c48d5b048
  521. Jun 13 12:38:42 online-node-81-113 kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 00007f0c45b5b040
  522. Jun 13 12:38:42 online-node-81-113 kernel: R13: 00007f0c48d5b040 R14: 0000000003200000 R15: 000055656fb9a9e0
  523. Jun 13 12:38:42 online-node-81-113 kernel: Task in /docker/aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d killed as a result of limit of /docker/aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d
  524. Jun 13 12:38:42 online-node-81-113 kernel: memory: usage 57344kB, limit 57344kB, failcnt 384
  525. Jun 13 12:38:42 online-node-81-113 kernel: memory+swap: usage 57344kB, limit 114688kB, failcnt 0
  526. Jun 13 12:38:42 online-node-81-113 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
  527. Jun 13 12:38:42 online-node-81-113 kernel: Memory cgroup stats for /docker/aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d: cache:10744KB rss:46600KB rss_huge:18432KB shmem:10716KB mapped_file:10708KB dirty:0KB writeback:0KB swap:0KB inactive_anon:10540KB active_anon:46752KB inactive_file:24KB active_file:4KB unevictable:0KB
  528. Jun 13 12:38:42 online-node-81-113 kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
  529. Jun 13 12:38:42 online-node-81-113 kernel: [22551] 0 22551 21395 4336 47 3 0 0 supervisord
  530. Jun 13 12:38:42 online-node-81-113 kernel: [23356] 0 23356 49846 6764 44 3 0 0 php-fpm
  531. Jun 13 12:38:42 online-node-81-113 kernel: [23357] 0 23357 3480 1069 13 3 0 0 nginx
  532. Jun 13 12:38:42 online-node-81-113 kernel: [23358] 7373 23358 3540 453 11 3 0 0 nginx
  533. Jun 13 12:38:42 online-node-81-113 kernel: [23359] 7373 23359 3540 453 11 3 0 0 nginx
  534. Jun 13 12:38:42 online-node-81-113 kernel: [23368] 7373 23368 49845 1346 30 3 0 0 php-fpm
  535. Jun 13 12:38:42 online-node-81-113 kernel: [23369] 7373 23369 49845 1346 30 3 0 0 php-fpm
  536. Jun 13 12:38:42 online-node-81-113 kernel: [23370] 7373 23370 49846 1346 30 3 0 0 php-fpm
  537. Jun 13 12:38:42 online-node-81-113 kernel: [23371] 7373 23371 49846 1346 30 3 0 0 php-fpm
  538. Jun 13 12:38:42 online-node-81-113 kernel: [23372] 7373 23372 49846 1346 30 3 0 0 php-fpm
  539. Jun 13 12:38:42 online-node-81-113 kernel: [23373] 7373 23373 49846 1346 30 3 0 0 php-fpm
  540. Jun 13 12:38:42 online-node-81-113 kernel: [23374] 7373 23374 49846 1347 30 3 0 0 php-fpm
  541. Jun 13 12:38:42 online-node-81-113 kernel: [23375] 7373 23375 49846 1347 30 3 0 0 php-fpm
  542. Jun 13 12:38:42 online-node-81-113 kernel: [23376] 7373 23376 49846 1347 30 3 0 0 php-fpm
  543. Jun 13 12:38:42 online-node-81-113 kernel: [23377] 7373 23377 49846 1347 30 3 0 0 php-fpm
  544. Jun 13 12:38:42 online-node-81-113 kernel: [23378] 7373 23378 49846 1347 30 3 0 0 php-fpm
  545. Jun 13 12:38:42 online-node-81-113 kernel: [23379] 7373 23379 49846 1347 30 3 0 0 php-fpm
  546. Jun 13 12:38:42 online-node-81-113 kernel: [23380] 7373 23380 49846 1347 30 3 0 0 php-fpm
  547. Jun 13 12:38:42 online-node-81-113 kernel: [23381] 7373 23381 49846 1347 30 3 0 0 php-fpm
  548. Jun 13 12:38:42 online-node-81-113 kernel: [26458] 0 26458 393 262 7 3 0 0 sh
  549. Jun 13 12:38:42 online-node-81-113 kernel: [ 499] 7373 499 49846 1347 30 3 0 0 php-fpm
  550. Jun 13 12:38:42 online-node-81-113 kernel: [ 2710] 7373 2710 49846 1347 30 3 0 0 php-fpm
  551. Jun 13 12:38:42 online-node-81-113 kernel: [ 3329] 7373 3329 49846 1347 30 3 0 0 php-fpm
  552. Jun 13 12:38:42 online-node-81-113 kernel: [ 3808] 7373 3808 49846 1347 30 3 0 0 php-fpm
  553. Jun 13 12:38:42 online-node-81-113 kernel: [ 3978] 7373 3978 49846 1347 30 3 0 0 php-fpm
  554. Jun 13 12:38:42 online-node-81-113 kernel: [ 4496] 7373 4496 49846 1347 30 3 0 0 php-fpm
  555. Jun 13 12:38:42 online-node-81-113 kernel: [28534] 0 28534 29888 9046 50 3 0 0 php
  556. Jun 13 12:38:42 online-node-81-113 kernel: Memory cgroup out of memory: Kill process 28534 (php) score 634 or sacrifice child
  557. Jun 13 12:38:42 online-node-81-113 kernel: Killed process 28534 (php) total-vm:119552kB, anon-rss:24832kB, file-rss:11352kB, shmem-rss:0kB
  558. Jun 13 12:38:42 online-node-81-113 kernel: oom_reaper: reaped process 28534 (php), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
  559. Jun 13 12:38:43 online-node-81-113 kernel: php invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=0
  560. Jun 13 12:38:43 online-node-81-113 kernel: php cpuset=aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d mems_allowed=0-1
  561. Jun 13 12:38:43 online-node-81-113 kernel: CPU: 3 PID: 29416 Comm: php Not tainted 4.14.15-1.el7.elrepo.x86_64 #1
  562. Jun 13 12:38:43 online-node-81-113 kernel: Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.3.4 11/08/2016
  563. Jun 13 12:38:43 online-node-81-113 kernel: Call Trace:
  564. Jun 13 12:38:43 online-node-81-113 kernel: dump_stack+0x63/0x85
  565. Jun 13 12:38:43 online-node-81-113 kernel: dump_header+0x9f/0x234
  566. Jun 13 12:38:43 online-node-81-113 kernel: ? mem_cgroup_scan_tasks+0x96/0xf0
  567. Jun 13 12:38:43 online-node-81-113 kernel: oom_kill_process+0x21c/0x430
  568. Jun 13 12:38:43 online-node-81-113 kernel: out_of_memory+0x114/0x4a0
  569. Jun 13 12:38:43 online-node-81-113 kernel: mem_cgroup_out_of_memory+0x4b/0x80
  570. Jun 13 12:38:43 online-node-81-113 kernel: mem_cgroup_oom_synchronize+0x2f9/0x320
  571. Jun 13 12:38:43 online-node-81-113 kernel: ? get_mctgt_type_thp.isra.30+0xc0/0xc0
  572. Jun 13 12:38:43 online-node-81-113 kernel: pagefault_out_of_memory+0x36/0x7c
  573. Jun 13 12:38:43 online-node-81-113 kernel: mm_fault_error+0x65/0x152
  574. Jun 13 12:38:43 online-node-81-113 kernel: __do_page_fault+0x456/0x4f0
  575. Jun 13 12:38:43 online-node-81-113 kernel: do_page_fault+0x38/0x130
  576. Jun 13 12:38:43 online-node-81-113 kernel: ? page_fault+0x36/0x60
  577. Jun 13 12:38:43 online-node-81-113 kernel: page_fault+0x4c/0x60
  578. Jun 13 12:38:43 online-node-81-113 kernel: RIP: 0033:0x7fbfef3596bf
  579. Jun 13 12:38:43 online-node-81-113 kernel: RSP: 002b:00007fffe1e9d128 EFLAGS: 00010206
  580. Jun 13 12:38:43 online-node-81-113 kernel: RAX: 00007fbfe9a6c040 RBX: 0000000000a00000 RCX: 0000000000020208
  581. Jun 13 12:38:43 online-node-81-113 kernel: RDX: 0000000000a00000 RSI: 00007fbfe996b000 RDI: 00007fbfea36b000
  582. Jun 13 12:38:43 online-node-81-113 kernel: RBP: 00007fbfef586480 R08: 0000000003240000 R09: 00007fbfec26c048
  583. Jun 13 12:38:43 online-node-81-113 kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 00007fbfe906c040
  584. Jun 13 12:38:43 online-node-81-113 kernel: R13: 00007fbfec26c040 R14: 0000000003200000 R15: 00005566c23799e0
  585. Jun 13 12:38:43 online-node-81-113 kernel: Task in /docker/aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d killed as a result of limit of /docker/aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d
  586. Jun 13 12:38:43 online-node-81-113 kernel: memory: usage 57344kB, limit 57344kB, failcnt 436
  587. Jun 13 12:38:43 online-node-81-113 kernel: memory+swap: usage 57344kB, limit 114688kB, failcnt 0
  588. Jun 13 12:38:43 online-node-81-113 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
  589. Jun 13 12:38:43 online-node-81-113 kernel: Memory cgroup stats for /docker/aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d: cache:10744KB rss:46600KB rss_huge:16384KB shmem:10716KB mapped_file:10708KB dirty:0KB writeback:0KB swap:0KB inactive_anon:10540KB active_anon:46748KB inactive_file:24KB active_file:4KB unevictable:0KB
  590. Jun 13 12:38:43 online-node-81-113 kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
  591. Jun 13 12:38:43 online-node-81-113 kernel: [22551] 0 22551 21395 4336 47 3 0 0 supervisord
  592. Jun 13 12:38:43 online-node-81-113 kernel: [23356] 0 23356 49846 6764 44 3 0 0 php-fpm
  593. Jun 13 12:38:43 online-node-81-113 kernel: [23357] 0 23357 3480 1069 13 3 0 0 nginx
  594. Jun 13 12:38:43 online-node-81-113 kernel: [23358] 7373 23358 3540 453 11 3 0 0 nginx
  595. Jun 13 12:38:43 online-node-81-113 kernel: [23359] 7373 23359 3540 453 11 3 0 0 nginx
  596. Jun 13 12:38:43 online-node-81-113 kernel: [23368] 7373 23368 49845 1346 30 3 0 0 php-fpm
  597. Jun 13 12:38:43 online-node-81-113 kernel: [23369] 7373 23369 49845 1346 30 3 0 0 php-fpm
  598. Jun 13 12:38:43 online-node-81-113 kernel: [23370] 7373 23370 49846 1346 30 3 0 0 php-fpm
  599. Jun 13 12:38:43 online-node-81-113 kernel: [23371] 7373 23371 49846 1346 30 3 0 0 php-fpm
  600. Jun 13 12:38:43 online-node-81-113 kernel: [23372] 7373 23372 49846 1346 30 3 0 0 php-fpm
  601. Jun 13 12:38:43 online-node-81-113 kernel: [23373] 7373 23373 49846 1346 30 3 0 0 php-fpm
  602. Jun 13 12:38:43 online-node-81-113 kernel: [23374] 7373 23374 49846 1347 30 3 0 0 php-fpm
  603. Jun 13 12:38:43 online-node-81-113 kernel: [23375] 7373 23375 49846 1347 30 3 0 0 php-fpm
  604. Jun 13 12:38:43 online-node-81-113 kernel: [23376] 7373 23376 49846 1347 30 3 0 0 php-fpm
  605. Jun 13 12:38:43 online-node-81-113 kernel: [23377] 7373 23377 49846 1347 30 3 0 0 php-fpm
  606. Jun 13 12:38:43 online-node-81-113 kernel: [23378] 7373 23378 49846 1347 30 3 0 0 php-fpm
  607. Jun 13 12:38:43 online-node-81-113 kernel: [23379] 7373 23379 49846 1347 30 3 0 0 php-fpm
  608. Jun 13 12:38:43 online-node-81-113 kernel: [23380] 7373 23380 49846 1347 30 3 0 0 php-fpm
  609. Jun 13 12:38:43 online-node-81-113 kernel: [23381] 7373 23381 49846 1347 30 3 0 0 php-fpm
  610. Jun 13 12:38:43 online-node-81-113 kernel: [26458] 0 26458 393 262 7 3 0 0 sh
  611. Jun 13 12:38:43 online-node-81-113 kernel: [ 499] 7373 499 49846 1347 30 3 0 0 php-fpm
  612. Jun 13 12:38:43 online-node-81-113 kernel: [ 2710] 7373 2710 49846 1347 30 3 0 0 php-fpm
  613. Jun 13 12:38:43 online-node-81-113 kernel: [ 3329] 7373 3329 49846 1347 30 3 0 0 php-fpm
  614. Jun 13 12:38:43 online-node-81-113 kernel: [ 3808] 7373 3808 49846 1347 30 3 0 0 php-fpm
  615. Jun 13 12:38:43 online-node-81-113 kernel: [ 3978] 7373 3978 49846 1347 30 3 0 0 php-fpm
  616. Jun 13 12:38:43 online-node-81-113 kernel: [ 4496] 7373 4496 49846 1347 30 3 0 0 php-fpm
  617. Jun 13 12:38:43 online-node-81-113 kernel: [29416] 0 29416 29888 9002 46 3 0 0 php
  618. Jun 13 12:38:43 online-node-81-113 kernel: Memory cgroup out of memory: Kill process 29416 (php) score 631 or sacrifice child
  619. Jun 13 12:38:43 online-node-81-113 kernel: Killed process 29416 (php) total-vm:119552kB, anon-rss:24636kB, file-rss:11372kB, shmem-rss:0kB
  620. Jun 13 12:38:43 online-node-81-113 kernel: oom_reaper: reaped process 29416 (php), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
  621. Jun 13 12:38:44 online-node-81-113 kernel: php invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=0
  622. Jun 13 12:38:44 online-node-81-113 kernel: php cpuset=aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d mems_allowed=0-1
  623. Jun 13 12:38:44 online-node-81-113 kernel: CPU: 5 PID: 29971 Comm: php Not tainted 4.14.15-1.el7.elrepo.x86_64 #1
  624. Jun 13 12:38:44 online-node-81-113 kernel: Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.3.4 11/08/2016
  625. Jun 13 12:38:44 online-node-81-113 kernel: Call Trace:
  626. Jun 13 12:38:44 online-node-81-113 kernel: dump_stack+0x63/0x85
  627. Jun 13 12:38:44 online-node-81-113 kernel: dump_header+0x9f/0x234
  628. Jun 13 12:38:44 online-node-81-113 kernel: ? mem_cgroup_scan_tasks+0x96/0xf0
  629. Jun 13 12:38:44 online-node-81-113 kernel: oom_kill_process+0x21c/0x430
  630. Jun 13 12:38:44 online-node-81-113 kernel: out_of_memory+0x114/0x4a0
  631. Jun 13 12:38:44 online-node-81-113 kernel: mem_cgroup_out_of_memory+0x4b/0x80
  632. Jun 13 12:38:44 online-node-81-113 kernel: mem_cgroup_oom_synchronize+0x2f9/0x320
  633. Jun 13 12:38:44 online-node-81-113 kernel: ? get_mctgt_type_thp.isra.30+0xc0/0xc0
  634. Jun 13 12:38:44 online-node-81-113 kernel: pagefault_out_of_memory+0x36/0x7c
  635. Jun 13 12:38:44 online-node-81-113 kernel: mm_fault_error+0x65/0x152
  636. Jun 13 12:38:44 online-node-81-113 kernel: __do_page_fault+0x456/0x4f0
  637. Jun 13 12:38:44 online-node-81-113 kernel: do_page_fault+0x38/0x130
  638. Jun 13 12:38:44 online-node-81-113 kernel: ? page_fault+0x36/0x60
  639. Jun 13 12:38:44 online-node-81-113 kernel: page_fault+0x4c/0x60
  640. Jun 13 12:38:44 online-node-81-113 kernel: RIP: 0033:0x7f48faf706bf
  641. Jun 13 12:38:44 online-node-81-113 kernel: RSP: 002b:00007ffeb3414d98 EFLAGS: 00010206
  642. Jun 13 12:38:44 online-node-81-113 kernel: RAX: 00007f48f5683040 RBX: 0000000000a00000 RCX: 000000000001fe08
  643. Jun 13 12:38:44 online-node-81-113 kernel: RDX: 0000000000a00000 RSI: 00007f48f5584000 RDI: 00007f48f5f84000
  644. Jun 13 12:38:44 online-node-81-113 kernel: RBP: 00007f48fb19d480 R08: 0000000003240000 R09: 00007f48f7e83048
  645. Jun 13 12:38:44 online-node-81-113 kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 00007f48f4c83040
  646. Jun 13 12:38:44 online-node-81-113 kernel: R13: 00007f48f7e83040 R14: 0000000003200000 R15: 00005585a658b9e0
  647. Jun 13 12:38:44 online-node-81-113 kernel: Task in /docker/aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d killed as a result of limit of /docker/aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d
  648. Jun 13 12:38:44 online-node-81-113 kernel: memory: usage 57344kB, limit 57344kB, failcnt 461
  649. Jun 13 12:38:44 online-node-81-113 kernel: memory+swap: usage 57344kB, limit 114688kB, failcnt 0
  650. Jun 13 12:38:44 online-node-81-113 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
  651. Jun 13 12:38:44 online-node-81-113 kernel: Memory cgroup stats for /docker/aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d: cache:10744KB rss:46600KB rss_huge:16384KB shmem:10716KB mapped_file:10708KB dirty:0KB writeback:0KB swap:0KB inactive_anon:10540KB active_anon:46776KB inactive_file:24KB active_file:4KB unevictable:0KB
  652. Jun 13 12:38:44 online-node-81-113 kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
  653. Jun 13 12:38:44 online-node-81-113 kernel: [22551] 0 22551 21395 4336 47 3 0 0 supervisord
  654. Jun 13 12:38:44 online-node-81-113 kernel: [23356] 0 23356 49846 6764 44 3 0 0 php-fpm
  655. Jun 13 12:38:44 online-node-81-113 kernel: [23357] 0 23357 3480 1069 13 3 0 0 nginx
  656. Jun 13 12:38:44 online-node-81-113 kernel: [23358] 7373 23358 3540 453 11 3 0 0 nginx
  657. Jun 13 12:38:44 online-node-81-113 kernel: [23359] 7373 23359 3540 453 11 3 0 0 nginx
  658. Jun 13 12:38:44 online-node-81-113 kernel: [23368] 7373 23368 49845 1346 30 3 0 0 php-fpm
  659. Jun 13 12:38:44 online-node-81-113 kernel: [23369] 7373 23369 49845 1346 30 3 0 0 php-fpm
  660. Jun 13 12:38:44 online-node-81-113 kernel: [23370] 7373 23370 49846 1346 30 3 0 0 php-fpm
  661. Jun 13 12:38:44 online-node-81-113 kernel: [23371] 7373 23371 49846 1346 30 3 0 0 php-fpm
  662. Jun 13 12:38:44 online-node-81-113 kernel: [23372] 7373 23372 49846 1346 30 3 0 0 php-fpm
  663. Jun 13 12:38:44 online-node-81-113 kernel: [23373] 7373 23373 49846 1346 30 3 0 0 php-fpm
  664. Jun 13 12:38:44 online-node-81-113 kernel: [23374] 7373 23374 49846 1347 30 3 0 0 php-fpm
  665. Jun 13 12:38:44 online-node-81-113 kernel: [23375] 7373 23375 49846 1347 30 3 0 0 php-fpm
  666. Jun 13 12:38:44 online-node-81-113 kernel: [23376] 7373 23376 49846 1347 30 3 0 0 php-fpm
  667. Jun 13 12:38:44 online-node-81-113 kernel: [23377] 7373 23377 49846 1347 30 3 0 0 php-fpm
  668. Jun 13 12:38:44 online-node-81-113 kernel: [23378] 7373 23378 49846 1347 30 3 0 0 php-fpm
  669. Jun 13 12:38:44 online-node-81-113 kernel: [23379] 7373 23379 49846 1347 30 3 0 0 php-fpm
  670. Jun 13 12:38:44 online-node-81-113 kernel: [23380] 7373 23380 49846 1347 30 3 0 0 php-fpm
  671. Jun 13 12:38:44 online-node-81-113 kernel: [23381] 7373 23381 49846 1347 30 3 0 0 php-fpm
  672. Jun 13 12:38:44 online-node-81-113 kernel: [26458] 0 26458 393 262 7 3 0 0 sh
  673. Jun 13 12:38:44 online-node-81-113 kernel: [ 499] 7373 499 49846 1347 30 3 0 0 php-fpm
  674. Jun 13 12:38:44 online-node-81-113 kernel: [ 2710] 7373 2710 49846 1347 30 3 0 0 php-fpm
  675. Jun 13 12:38:44 online-node-81-113 kernel: [ 3329] 7373 3329 49846 1347 30 3 0 0 php-fpm
  676. Jun 13 12:38:44 online-node-81-113 kernel: [ 3808] 7373 3808 49846 1347 30 3 0 0 php-fpm
  677. Jun 13 12:38:44 online-node-81-113 kernel: [ 3978] 7373 3978 49846 1347 30 3 0 0 php-fpm
  678. Jun 13 12:38:44 online-node-81-113 kernel: [ 4496] 7373 4496 49846 1347 30 3 0 0 php-fpm
  679. Jun 13 12:38:44 online-node-81-113 kernel: [29971] 0 29971 29888 8997 47 3 0 0 php
  680. Jun 13 12:38:44 online-node-81-113 kernel: Memory cgroup out of memory: Kill process 29971 (php) score 631 or sacrifice child
  681. Jun 13 12:38:44 online-node-81-113 kernel: Killed process 29971 (php) total-vm:119552kB, anon-rss:24632kB, file-rss:11356kB, shmem-rss:0kB
  682. Jun 13 12:38:44 online-node-81-113 kernel: oom_reaper: reaped process 29971 (php), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
  683. Jun 13 12:38:45 online-node-81-113 kernel: php invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=0
  684. Jun 13 12:38:45 online-node-81-113 kernel: php cpuset=aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d mems_allowed=0-1
  685. Jun 13 12:38:45 online-node-81-113 kernel: CPU: 17 PID: 30334 Comm: php Not tainted 4.14.15-1.el7.elrepo.x86_64 #1
  686. Jun 13 12:38:45 online-node-81-113 kernel: Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.3.4 11/08/2016
  687. Jun 13 12:38:45 online-node-81-113 kernel: Call Trace:
  688. Jun 13 12:38:45 online-node-81-113 kernel: dump_stack+0x63/0x85
  689. Jun 13 12:38:45 online-node-81-113 kernel: dump_header+0x9f/0x234
  690. Jun 13 12:38:45 online-node-81-113 kernel: ? mem_cgroup_scan_tasks+0x96/0xf0
  691. Jun 13 12:38:45 online-node-81-113 kernel: oom_kill_process+0x21c/0x430
  692. Jun 13 12:38:45 online-node-81-113 kernel: out_of_memory+0x114/0x4a0
  693. Jun 13 12:38:45 online-node-81-113 kernel: mem_cgroup_out_of_memory+0x4b/0x80
  694. Jun 13 12:38:45 online-node-81-113 kernel: mem_cgroup_oom_synchronize+0x2f9/0x320
  695. Jun 13 12:38:45 online-node-81-113 kernel: ? get_mctgt_type_thp.isra.30+0xc0/0xc0
  696. Jun 13 12:38:45 online-node-81-113 kernel: pagefault_out_of_memory+0x36/0x7c
  697. Jun 13 12:38:45 online-node-81-113 kernel: mm_fault_error+0x65/0x152
  698. Jun 13 12:38:45 online-node-81-113 kernel: __do_page_fault+0x456/0x4f0
  699. Jun 13 12:38:45 online-node-81-113 kernel: do_page_fault+0x38/0x130
  700. Jun 13 12:38:45 online-node-81-113 kernel: ? page_fault+0x36/0x60
  701. Jun 13 12:38:45 online-node-81-113 kernel: page_fault+0x4c/0x60
  702. Jun 13 12:38:45 online-node-81-113 kernel: RIP: 0033:0x7f3180dac6bf
  703. Jun 13 12:38:45 online-node-81-113 kernel: RSP: 002b:00007ffc4e655118 EFLAGS: 00010206
  704. Jun 13 12:38:45 online-node-81-113 kernel: RAX: 00007f317b4bf040 RBX: 0000000000a00000 RCX: 0000000000020008
  705. Jun 13 12:38:45 online-node-81-113 kernel: RDX: 0000000000a00000 RSI: 00007f317b3bf000 RDI: 00007f317bdbf000
  706. Jun 13 12:38:45 online-node-81-113 kernel: RBP: 00007f3180fd9480 R08: 0000000003240000 R09: 00007f317dcbf048
  707. Jun 13 12:38:45 online-node-81-113 kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 00007f317aabf040
  708. Jun 13 12:38:45 online-node-81-113 kernel: R13: 00007f317dcbf040 R14: 0000000003200000 R15: 000055bba8f709e0
  709. Jun 13 12:38:45 online-node-81-113 kernel: Task in /docker/aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d killed as a result of limit of /docker/aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d
  710. Jun 13 12:38:45 online-node-81-113 kernel: memory: usage 57344kB, limit 57344kB, failcnt 517
  711. Jun 13 12:38:45 online-node-81-113 kernel: memory+swap: usage 57344kB, limit 114688kB, failcnt 0
  712. Jun 13 12:38:45 online-node-81-113 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
  713. Jun 13 12:38:45 online-node-81-113 kernel: Memory cgroup stats for /docker/aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d: cache:10744KB rss:46600KB rss_huge:16384KB shmem:10716KB mapped_file:10708KB dirty:0KB writeback:0KB swap:0KB inactive_anon:10540KB active_anon:46752KB inactive_file:24KB active_file:4KB unevictable:0KB
  714. Jun 13 12:38:45 online-node-81-113 kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
  715. Jun 13 12:38:45 online-node-81-113 kernel: [22551] 0 22551 21395 4336 47 3 0 0 supervisord
  716. Jun 13 12:38:45 online-node-81-113 kernel: [23356] 0 23356 49846 6764 44 3 0 0 php-fpm
  717. Jun 13 12:38:45 online-node-81-113 kernel: [23357] 0 23357 3480 1069 13 3 0 0 nginx
  718. Jun 13 12:38:45 online-node-81-113 kernel: [23358] 7373 23358 3540 453 11 3 0 0 nginx
  719. Jun 13 12:38:45 online-node-81-113 kernel: [23359] 7373 23359 3540 453 11 3 0 0 nginx
  720. Jun 13 12:38:45 online-node-81-113 kernel: [23368] 7373 23368 49845 1346 30 3 0 0 php-fpm
  721. Jun 13 12:38:45 online-node-81-113 kernel: [23369] 7373 23369 49845 1346 30 3 0 0 php-fpm
  722. Jun 13 12:38:45 online-node-81-113 kernel: [23370] 7373 23370 49846 1346 30 3 0 0 php-fpm
  723. Jun 13 12:38:45 online-node-81-113 kernel: [23371] 7373 23371 49846 1346 30 3 0 0 php-fpm
  724. Jun 13 12:38:45 online-node-81-113 kernel: [23372] 7373 23372 49846 1346 30 3 0 0 php-fpm
  725. Jun 13 12:38:45 online-node-81-113 kernel: [23373] 7373 23373 49846 1346 30 3 0 0 php-fpm
  726. Jun 13 12:38:45 online-node-81-113 kernel: [23374] 7373 23374 49846 1347 30 3 0 0 php-fpm
  727. Jun 13 12:38:45 online-node-81-113 kernel: [23375] 7373 23375 49846 1347 30 3 0 0 php-fpm
  728. Jun 13 12:38:45 online-node-81-113 kernel: [23376] 7373 23376 49846 1347 30 3 0 0 php-fpm
  729. Jun 13 12:38:45 online-node-81-113 kernel: [23377] 7373 23377 49846 1347 30 3 0 0 php-fpm
  730. Jun 13 12:38:45 online-node-81-113 kernel: [23378] 7373 23378 49846 1347 30 3 0 0 php-fpm
  731. Jun 13 12:38:45 online-node-81-113 kernel: [23379] 7373 23379 49846 1347 30 3 0 0 php-fpm
  732. Jun 13 12:38:45 online-node-81-113 kernel: [23380] 7373 23380 49846 1347 30 3 0 0 php-fpm
  733. Jun 13 12:38:45 online-node-81-113 kernel: [23381] 7373 23381 49846 1347 30 3 0 0 php-fpm
  734. Jun 13 12:38:45 online-node-81-113 kernel: [26458] 0 26458 393 262 7 3 0 0 sh
  735. Jun 13 12:38:45 online-node-81-113 kernel: [ 499] 7373 499 49846 1347 30 3 0 0 php-fpm
  736. Jun 13 12:38:45 online-node-81-113 kernel: [ 2710] 7373 2710 49846 1347 30 3 0 0 php-fpm
  737. Jun 13 12:38:45 online-node-81-113 kernel: [ 3329] 7373 3329 49846 1347 30 3 0 0 php-fpm
  738. Jun 13 12:38:45 online-node-81-113 kernel: [ 3808] 7373 3808 49846 1347 30 3 0 0 php-fpm
  739. Jun 13 12:38:45 online-node-81-113 kernel: [ 3978] 7373 3978 49846 1347 30 3 0 0 php-fpm
  740. Jun 13 12:38:45 online-node-81-113 kernel: [ 4496] 7373 4496 49846 1347 30 3 0 0 php-fpm
  741. Jun 13 12:38:45 online-node-81-113 kernel: [30334] 0 30334 29888 8992 45 4 0 0 php
  742. Jun 13 12:38:45 online-node-81-113 kernel: Memory cgroup out of memory: Kill process 30334 (php) score 630 or sacrifice child
  743. Jun 13 12:38:45 online-node-81-113 kernel: Killed process 30334 (php) total-vm:119552kB, anon-rss:24640kB, file-rss:11328kB, shmem-rss:0kB
  744. Jun 13 12:38:45 online-node-81-113 kernel: oom_reaper: reaped process 30334 (php), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
  745. Jun 13 12:38:45 online-node-81-113 kernel: php invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=0
  746. Jun 13 12:38:45 online-node-81-113 kernel: php cpuset=aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d mems_allowed=0-1
  747. Jun 13 12:38:45 online-node-81-113 kernel: CPU: 1 PID: 30681 Comm: php Not tainted 4.14.15-1.el7.elrepo.x86_64 #1
  748. Jun 13 12:38:45 online-node-81-113 kernel: Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.3.4 11/08/2016
  749. Jun 13 12:38:45 online-node-81-113 kernel: Call Trace:
  750. Jun 13 12:38:45 online-node-81-113 kernel: dump_stack+0x63/0x85
  751. Jun 13 12:38:45 online-node-81-113 kernel: dump_header+0x9f/0x234
  752. Jun 13 12:38:45 online-node-81-113 kernel: ? mem_cgroup_scan_tasks+0x96/0xf0
  753. Jun 13 12:38:45 online-node-81-113 kernel: oom_kill_process+0x21c/0x430
  754. Jun 13 12:38:45 online-node-81-113 kernel: out_of_memory+0x114/0x4a0
  755. Jun 13 12:38:45 online-node-81-113 kernel: mem_cgroup_out_of_memory+0x4b/0x80
  756. Jun 13 12:38:45 online-node-81-113 kernel: mem_cgroup_oom_synchronize+0x2f9/0x320
  757. Jun 13 12:38:45 online-node-81-113 kernel: ? get_mctgt_type_thp.isra.30+0xc0/0xc0
  758. Jun 13 12:38:45 online-node-81-113 kernel: pagefault_out_of_memory+0x36/0x7c
  759. Jun 13 12:38:45 online-node-81-113 kernel: mm_fault_error+0x65/0x152
  760. Jun 13 12:38:45 online-node-81-113 kernel: __do_page_fault+0x456/0x4f0
  761. Jun 13 12:38:45 online-node-81-113 kernel: do_page_fault+0x38/0x130
  762. Jun 13 12:38:45 online-node-81-113 kernel: ? page_fault+0x36/0x60
  763. Jun 13 12:38:45 online-node-81-113 kernel: page_fault+0x4c/0x60
  764. Jun 13 12:38:45 online-node-81-113 kernel: RIP: 0033:0x7f13279236bf
  765. Jun 13 12:38:45 online-node-81-113 kernel: RSP: 002b:00007ffcdaa96438 EFLAGS: 00010206
  766. Jun 13 12:38:45 online-node-81-113 kernel: RAX: 00007f1322036040 RBX: 0000000000a00000 RCX: 0000000000020208
  767. Jun 13 12:38:45 online-node-81-113 kernel: RDX: 0000000000a00000 RSI: 00007f1321f35000 RDI: 00007f1322935000
  768. Jun 13 12:38:45 online-node-81-113 kernel: RBP: 00007f1327b50480 R08: 0000000003240000 R09: 00007f1324836048
  769. Jun 13 12:38:45 online-node-81-113 kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 00007f1321636040
  770. Jun 13 12:38:45 online-node-81-113 kernel: R13: 00007f1324836040 R14: 0000000003200000 R15: 00005570021359e0
  771. Jun 13 12:38:45 online-node-81-113 kernel: Task in /docker/aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d killed as a result of limit of /docker/aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d
  772. Jun 13 12:38:45 online-node-81-113 kernel: memory: usage 57344kB, limit 57344kB, failcnt 570
  773. Jun 13 12:38:45 online-node-81-113 kernel: memory+swap: usage 57344kB, limit 114688kB, failcnt 0
  774. Jun 13 12:38:45 online-node-81-113 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
  775. Jun 13 12:38:45 online-node-81-113 kernel: Memory cgroup stats for /docker/aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d: cache:10744KB rss:46600KB rss_huge:16384KB shmem:10716KB mapped_file:10708KB dirty:0KB writeback:0KB swap:0KB inactive_anon:10540KB active_anon:46752KB inactive_file:24KB active_file:4KB unevictable:0KB
  776. Jun 13 12:38:45 online-node-81-113 kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
  777. Jun 13 12:38:45 online-node-81-113 kernel: [22551] 0 22551 21395 4336 47 3 0 0 supervisord
  778. Jun 13 12:38:45 online-node-81-113 kernel: [23356] 0 23356 49846 6764 44 3 0 0 php-fpm
  779. Jun 13 12:38:45 online-node-81-113 kernel: [23357] 0 23357 3480 1069 13 3 0 0 nginx
  780. Jun 13 12:38:45 online-node-81-113 kernel: [23358] 7373 23358 3540 453 11 3 0 0 nginx
  781. Jun 13 12:38:45 online-node-81-113 kernel: [23359] 7373 23359 3540 453 11 3 0 0 nginx
  782. Jun 13 12:38:45 online-node-81-113 kernel: [23368] 7373 23368 49845 1346 30 3 0 0 php-fpm
  783. Jun 13 12:38:45 online-node-81-113 kernel: [23369] 7373 23369 49845 1346 30 3 0 0 php-fpm
  784. Jun 13 12:38:45 online-node-81-113 kernel: [23370] 7373 23370 49846 1346 30 3 0 0 php-fpm
  785. Jun 13 12:38:45 online-node-81-113 kernel: [23371] 7373 23371 49846 1346 30 3 0 0 php-fpm
  786. Jun 13 12:38:45 online-node-81-113 kernel: [23372] 7373 23372 49846 1346 30 3 0 0 php-fpm
  787. Jun 13 12:38:45 online-node-81-113 kernel: [23373] 7373 23373 49846 1346 30 3 0 0 php-fpm
  788. Jun 13 12:38:45 online-node-81-113 kernel: [23374] 7373 23374 49846 1347 30 3 0 0 php-fpm
  789. Jun 13 12:38:45 online-node-81-113 kernel: [23375] 7373 23375 49846 1347 30 3 0 0 php-fpm
  790. Jun 13 12:38:45 online-node-81-113 kernel: [23376] 7373 23376 49846 1347 30 3 0 0 php-fpm
  791. Jun 13 12:38:45 online-node-81-113 kernel: [23377] 7373 23377 49846 1347 30 3 0 0 php-fpm
  792. Jun 13 12:38:45 online-node-81-113 kernel: [23378] 7373 23378 49846 1347 30 3 0 0 php-fpm
  793. Jun 13 12:38:45 online-node-81-113 kernel: [23379] 7373 23379 49846 1347 30 3 0 0 php-fpm
  794. Jun 13 12:38:45 online-node-81-113 kernel: [23380] 7373 23380 49846 1347 30 3 0 0 php-fpm
  795. Jun 13 12:38:45 online-node-81-113 kernel: [23381] 7373 23381 49846 1347 30 3 0 0 php-fpm
  796. Jun 13 12:38:45 online-node-81-113 kernel: [26458] 0 26458 393 262 7 3 0 0 sh
  797. Jun 13 12:38:45 online-node-81-113 kernel: [ 499] 7373 499 49846 1347 30 3 0 0 php-fpm
  798. Jun 13 12:38:45 online-node-81-113 kernel: [ 2710] 7373 2710 49846 1347 30 3 0 0 php-fpm
  799. Jun 13 12:38:45 online-node-81-113 kernel: [ 3329] 7373 3329 49846 1347 30 3 0 0 php-fpm
  800. Jun 13 12:38:45 online-node-81-113 kernel: [ 3808] 7373 3808 49846 1347 30 3 0 0 php-fpm
  801. Jun 13 12:38:45 online-node-81-113 kernel: [ 3978] 7373 3978 49846 1347 30 3 0 0 php-fpm
  802. Jun 13 12:38:45 online-node-81-113 kernel: [ 4496] 7373 4496 49846 1347 30 3 0 0 php-fpm
  803. Jun 13 12:38:45 online-node-81-113 kernel: [30681] 0 30681 29888 8962 44 3 0 0 php
  804. Jun 13 12:38:45 online-node-81-113 kernel: Memory cgroup out of memory: Kill process 30681 (php) score 628 or sacrifice child
  805. Jun 13 12:38:45 online-node-81-113 kernel: Killed process 30681 (php) total-vm:119552kB, anon-rss:24640kB, file-rss:11208kB, shmem-rss:0kB
  806. Jun 13 12:38:46 online-node-81-113 kernel: oom_reaper: reaped process 30681 (php), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
  807. Jun 13 12:38:55 online-node-81-113 kernel: php invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=0
  808. Jun 13 12:38:55 online-node-81-113 kernel: php cpuset=aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d mems_allowed=0-1
  809. Jun 13 12:38:55 online-node-81-113 kernel: CPU: 35 PID: 36371 Comm: php Not tainted 4.14.15-1.el7.elrepo.x86_64 #1
  810. Jun 13 12:38:55 online-node-81-113 kernel: Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.3.4 11/08/2016
  811. Jun 13 12:38:55 online-node-81-113 kernel: Call Trace:
  812. Jun 13 12:38:55 online-node-81-113 kernel: dump_stack+0x63/0x85
  813. Jun 13 12:38:55 online-node-81-113 kernel: dump_header+0x9f/0x234
  814. Jun 13 12:38:55 online-node-81-113 kernel: ? mem_cgroup_scan_tasks+0x96/0xf0
  815. Jun 13 12:38:55 online-node-81-113 kernel: oom_kill_process+0x21c/0x430
  816. Jun 13 12:38:55 online-node-81-113 kernel: out_of_memory+0x114/0x4a0
  817. Jun 13 12:38:55 online-node-81-113 kernel: mem_cgroup_out_of_memory+0x4b/0x80
  818. Jun 13 12:38:55 online-node-81-113 kernel: mem_cgroup_oom_synchronize+0x2f9/0x320
  819. Jun 13 12:38:55 online-node-81-113 kernel: ? get_mctgt_type_thp.isra.30+0xc0/0xc0
  820. Jun 13 12:38:55 online-node-81-113 kernel: pagefault_out_of_memory+0x36/0x7c
  821. Jun 13 12:38:55 online-node-81-113 kernel: mm_fault_error+0x65/0x152
  822. Jun 13 12:38:55 online-node-81-113 kernel: __do_page_fault+0x456/0x4f0
  823. Jun 13 12:38:55 online-node-81-113 kernel: do_page_fault+0x38/0x130
  824. Jun 13 12:38:55 online-node-81-113 kernel: ? page_fault+0x36/0x60
  825. Jun 13 12:38:55 online-node-81-113 kernel: page_fault+0x4c/0x60
  826. Jun 13 12:38:55 online-node-81-113 kernel: RIP: 0033:0x7fd9621eb6bf
  827. Jun 13 12:38:55 online-node-81-113 kernel: RSP: 002b:00007ffc4d1f3468 EFLAGS: 00010206
  828. Jun 13 12:38:55 online-node-81-113 kernel: RAX: 00007fd95c8fe040 RBX: 0000000000a00000 RCX: 0000000000020208
  829. Jun 13 12:38:55 online-node-81-113 kernel: RDX: 0000000000a00000 RSI: 00007fd95c7fd000 RDI: 00007fd95d1fd000
  830. Jun 13 12:38:55 online-node-81-113 kernel: RBP: 00007fd962418480 R08: 0000000003240000 R09: 00007fd95f0fe048
  831. Jun 13 12:38:55 online-node-81-113 kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 00007fd95befe040
  832. Jun 13 12:38:55 online-node-81-113 kernel: R13: 00007fd95f0fe040 R14: 0000000003200000 R15: 000055b45210a9e0
  833. Jun 13 12:38:55 online-node-81-113 kernel: Task in /docker/aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d killed as a result of limit of /docker/aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d
  834. Jun 13 12:38:55 online-node-81-113 kernel: memory: usage 57344kB, limit 57344kB, failcnt 624
  835. Jun 13 12:38:55 online-node-81-113 kernel: memory+swap: usage 57344kB, limit 114688kB, failcnt 0
  836. Jun 13 12:38:55 online-node-81-113 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
  837. Jun 13 12:38:55 online-node-81-113 kernel: Memory cgroup stats for /docker/aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d: cache:10744KB rss:46600KB rss_huge:16384KB shmem:10716KB mapped_file:10708KB dirty:0KB writeback:0KB swap:0KB inactive_anon:10540KB active_anon:46756KB inactive_file:24KB active_file:4KB unevictable:0KB
  838. Jun 13 12:38:55 online-node-81-113 kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
  839. Jun 13 12:38:55 online-node-81-113 kernel: [22551] 0 22551 21395 4336 47 3 0 0 supervisord
  840. Jun 13 12:38:55 online-node-81-113 kernel: [23356] 0 23356 49846 6764 44 3 0 0 php-fpm
  841. Jun 13 12:38:55 online-node-81-113 kernel: [23357] 0 23357 3480 1069 13 3 0 0 nginx
  842. Jun 13 12:38:55 online-node-81-113 kernel: [23358] 7373 23358 3540 453 11 3 0 0 nginx
  843. Jun 13 12:38:55 online-node-81-113 kernel: [23359] 7373 23359 3540 453 11 3 0 0 nginx
  844. Jun 13 12:38:55 online-node-81-113 kernel: [23368] 7373 23368 49845 1346 30 3 0 0 php-fpm
  845. Jun 13 12:38:55 online-node-81-113 kernel: [23369] 7373 23369 49845 1346 30 3 0 0 php-fpm
  846. Jun 13 12:38:55 online-node-81-113 kernel: [23370] 7373 23370 49846 1346 30 3 0 0 php-fpm
  847. Jun 13 12:38:55 online-node-81-113 kernel: [23371] 7373 23371 49846 1346 30 3 0 0 php-fpm
  848. Jun 13 12:38:55 online-node-81-113 kernel: [23372] 7373 23372 49846 1346 30 3 0 0 php-fpm
  849. Jun 13 12:38:55 online-node-81-113 kernel: [23373] 7373 23373 49846 1346 30 3 0 0 php-fpm
  850. Jun 13 12:38:55 online-node-81-113 kernel: [23374] 7373 23374 49846 1347 30 3 0 0 php-fpm
  851. Jun 13 12:38:55 online-node-81-113 kernel: [23375] 7373 23375 49846 1347 30 3 0 0 php-fpm
  852. Jun 13 12:38:55 online-node-81-113 kernel: [23376] 7373 23376 49846 1347 30 3 0 0 php-fpm
  853. Jun 13 12:38:55 online-node-81-113 kernel: [23377] 7373 23377 49846 1347 30 3 0 0 php-fpm
  854. Jun 13 12:38:55 online-node-81-113 kernel: [23378] 7373 23378 49846 1347 30 3 0 0 php-fpm
  855. Jun 13 12:38:55 online-node-81-113 kernel: [23379] 7373 23379 49846 1347 30 3 0 0 php-fpm
  856. Jun 13 12:38:55 online-node-81-113 kernel: [23380] 7373 23380 49846 1347 30 3 0 0 php-fpm
  857. Jun 13 12:38:55 online-node-81-113 kernel: [23381] 7373 23381 49846 1347 30 3 0 0 php-fpm
  858. Jun 13 12:38:55 online-node-81-113 kernel: [26458] 0 26458 393 262 7 3 0 0 sh
  859. Jun 13 12:38:55 online-node-81-113 kernel: [ 499] 7373 499 49846 1347 30 3 0 0 php-fpm
  860. Jun 13 12:38:55 online-node-81-113 kernel: [ 2710] 7373 2710 49846 1347 30 3 0 0 php-fpm
  861. Jun 13 12:38:55 online-node-81-113 kernel: [ 3329] 7373 3329 49846 1347 30 3 0 0 php-fpm
  862. Jun 13 12:38:55 online-node-81-113 kernel: [ 3808] 7373 3808 49846 1347 30 3 0 0 php-fpm
  863. Jun 13 12:38:55 online-node-81-113 kernel: [ 3978] 7373 3978 49846 1347 30 3 0 0 php-fpm
  864. Jun 13 12:38:55 online-node-81-113 kernel: [ 4496] 7373 4496 49846 1347 30 3 0 0 php-fpm
  865. Jun 13 12:38:55 online-node-81-113 kernel: [36371] 0 36371 29888 9006 47 3 0 0 php
  866. Jun 13 12:38:55 online-node-81-113 kernel: Memory cgroup out of memory: Kill process 36371 (php) score 631 or sacrifice child
  867. Jun 13 12:38:55 online-node-81-113 kernel: Killed process 36371 (php) total-vm:119552kB, anon-rss:24636kB, file-rss:11388kB, shmem-rss:0kB
  868. Jun 13 12:38:55 online-node-81-113 kernel: oom_reaper: reaped process 36371 (php), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
  869. Jun 13 12:40:22 online-node-81-113 kernel: php invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=0
  870. Jun 13 12:40:22 online-node-81-113 kernel: php cpuset=aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d mems_allowed=0-1
  871. Jun 13 12:40:22 online-node-81-113 kernel: CPU: 37 PID: 7768 Comm: php Not tainted 4.14.15-1.el7.elrepo.x86_64 #1
  872. Jun 13 12:40:22 online-node-81-113 kernel: Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.3.4 11/08/2016
  873. Jun 13 12:40:22 online-node-81-113 kernel: Call Trace:
  874. Jun 13 12:40:22 online-node-81-113 kernel: dump_stack+0x63/0x85
  875. Jun 13 12:40:22 online-node-81-113 kernel: dump_header+0x9f/0x234
  876. Jun 13 12:40:22 online-node-81-113 kernel: ? mem_cgroup_scan_tasks+0x96/0xf0
  877. Jun 13 12:40:22 online-node-81-113 kernel: oom_kill_process+0x21c/0x430
  878. Jun 13 12:40:22 online-node-81-113 kernel: out_of_memory+0x114/0x4a0
  879. Jun 13 12:40:22 online-node-81-113 kernel: mem_cgroup_out_of_memory+0x4b/0x80
  880. Jun 13 12:40:22 online-node-81-113 kernel: mem_cgroup_oom_synchronize+0x2f9/0x320
  881. Jun 13 12:40:22 online-node-81-113 kernel: ? get_mctgt_type_thp.isra.30+0xc0/0xc0
  882. Jun 13 12:40:22 online-node-81-113 kernel: pagefault_out_of_memory+0x36/0x7c
  883. Jun 13 12:40:22 online-node-81-113 kernel: mm_fault_error+0x65/0x152
  884. Jun 13 12:40:22 online-node-81-113 kernel: __do_page_fault+0x456/0x4f0
  885. Jun 13 12:40:22 online-node-81-113 kernel: do_page_fault+0x38/0x130
  886. Jun 13 12:40:22 online-node-81-113 kernel: ? page_fault+0x36/0x60
  887. Jun 13 12:40:22 online-node-81-113 kernel: page_fault+0x4c/0x60
  888. Jun 13 12:40:22 online-node-81-113 kernel: RIP: 0033:0x7fd7c1e8f6bf
  889. Jun 13 12:40:22 online-node-81-113 kernel: RSP: 002b:00007fff92f0ca68 EFLAGS: 00010206
  890. Jun 13 12:40:22 online-node-81-113 kernel: RAX: 00007fd7bc5a2040 RBX: 0000000000a00000 RCX: 0000000000021808
  891. Jun 13 12:40:22 online-node-81-113 kernel: RDX: 0000000000a00000 RSI: 00007fd7bc496000 RDI: 00007fd7bce96000
  892. Jun 13 12:40:22 online-node-81-113 kernel: RBP: 00007fd7c20bc480 R08: 0000000003240000 R09: 00007fd7beda2048
  893. Jun 13 12:40:22 online-node-81-113 kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 00007fd7bbba2040
  894. Jun 13 12:40:22 online-node-81-113 kernel: R13: 00007fd7beda2040 R14: 0000000003200000 R15: 0000557b755e19e0
  895. Jun 13 12:40:22 online-node-81-113 kernel: Task in /docker/aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d killed as a result of limit of /docker/aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d
  896. Jun 13 12:40:22 online-node-81-113 kernel: memory: usage 57344kB, limit 57344kB, failcnt 665
  897. Jun 13 12:40:22 online-node-81-113 kernel: memory+swap: usage 57344kB, limit 114688kB, failcnt 0
  898. Jun 13 12:40:22 online-node-81-113 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
  899. Jun 13 12:40:22 online-node-81-113 kernel: Memory cgroup stats for /docker/aaaf61d798fa04813f49d0365ef17902b20ff1d8485553622f20f1fdc0f97b7d: cache:10744KB rss:46600KB rss_huge:18432KB shmem:10716KB mapped_file:10708KB dirty:4KB writeback:0KB swap:0KB inactive_anon:10540KB active_anon:46776KB inactive_file:28KB active_file:0KB unevictable:0KB
  900. Jun 13 12:40:22 online-node-81-113 kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
  901. Jun 13 12:40:22 online-node-81-113 kernel: [22551] 0 22551 21395 4336 47 3 0 0 supervisord
  902. Jun 13 12:40:22 online-node-81-113 kernel: [23356] 0 23356 49846 6764 44 3 0 0 php-fpm
  903. Jun 13 12:40:22 online-node-81-113 kernel: [23357] 0 23357 3480 1069 13 3 0 0 nginx
  904. Jun 13 12:40:22 online-node-81-113 kernel: [23358] 7373 23358 3540 453 11 3 0 0 nginx
  905. Jun 13 12:40:22 online-node-81-113 kernel: [23359] 7373 23359 3540 453 11 3 0 0 nginx
  906. Jun 13 12:40:22 online-node-81-113 kernel: [23368] 7373 23368 49845 1346 30 3 0 0 php-fpm
  907. Jun 13 12:40:22 online-node-81-113 kernel: [23369] 7373 23369 49845 1346 30 3 0 0 php-fpm
  908. Jun 13 12:40:22 online-node-81-113 kernel: [23370] 7373 23370 49846 1346 30 3 0 0 php-fpm
  909. Jun 13 12:40:22 online-node-81-113 kernel: [23371] 7373 23371 49846 1346 30 3 0 0 php-fpm
  910. Jun 13 12:40:22 online-node-81-113 kernel: [23372] 7373 23372 49846 1346 30 3 0 0 php-fpm
  911. Jun 13 12:40:22 online-node-81-113 kernel: [23373] 7373 23373 49846 1346 30 3 0 0 php-fpm
  912. Jun 13 12:40:22 online-node-81-113 kernel: [23374] 7373 23374 49846 1347 30 3 0 0 php-fpm
  913. Jun 13 12:40:22 online-node-81-113 kernel: [23375] 7373 23375 49846 1347 30 3 0 0 php-fpm
  914. Jun 13 12:40:22 online-node-81-113 kernel: [23376] 7373 23376 49846 1347 30 3 0 0 php-fpm
  915. Jun 13 12:40:22 online-node-81-113 kernel: [23377] 7373 23377 49846 1347 30 3 0 0 php-fpm
  916. Jun 13 12:40:22 online-node-81-113 kernel: [23378] 7373 23378 49846 1347 30 3 0 0 php-fpm
  917. Jun 13 12:40:22 online-node-81-113 kernel: [23379] 7373 23379 49846 1347 30 3 0 0 php-fpm
  918. Jun 13 12:40:22 online-node-81-113 kernel: [23380] 7373 23380 49846 1347 30 3 0 0 php-fpm
  919. Jun 13 12:40:22 online-node-81-113 kernel: [23381] 7373 23381 49846 1347 30 3 0 0 php-fpm
  920. Jun 13 12:40:22 online-node-81-113 kernel: [ 499] 7373 499 49846 1347 30 3 0 0 php-fpm
  921. Jun 13 12:40:22 online-node-81-113 kernel: [ 2710] 7373 2710 49846 1347 30 3 0 0 php-fpm
  922. Jun 13 12:40:22 online-node-81-113 kernel: [ 3329] 7373 3329 49846 1347 30 3 0 0 php-fpm
  923. Jun 13 12:40:22 online-node-81-113 kernel: [ 3808] 7373 3808 49846 1347 30 3 0 0 php-fpm
  924. Jun 13 12:40:22 online-node-81-113 kernel: [ 3978] 7373 3978 49846 1347 30 3 0 0 php-fpm
  925. Jun 13 12:40:22 online-node-81-113 kernel: [ 4496] 7373 4496 49846 1347 30 3 0 0 php-fpm
  926. Jun 13 12:40:22 online-node-81-113 kernel: [ 3736] 0 3736 393 261 7 3 0 0 sh
  927. Jun 13 12:40:22 online-node-81-113 kernel: [ 7768] 0 7768 29888 8980 50 4 0 0 php
  928. Jun 13 12:40:22 online-node-81-113 kernel: Memory cgroup out of memory: Kill process 7768 (php) score 630 or sacrifice child
  929. Jun 13 12:40:22 online-node-81-113 kernel: Killed process 7768 (php) total-vm:119552kB, anon-rss:24564kB, file-rss:11356kB, shmem-rss:0kB
  930. Jun 13 12:40:22 online-node-81-113 kernel: oom_reaper: reaped process 7768 (php), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
  931. Jun 13 12:41:54 online-node-81-113 kernel: thrash invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=0
  932. Jun 13 12:41:54 online-node-81-113 kernel: thrash cpuset=f47120aab1019f857761657b8271185c1cb88372b620c4efdd718570980c24c3 mems_allowed=0-1
  933. Jun 13 12:41:54 online-node-81-113 kernel: CPU: 3 PID: 21921 Comm: thrash Not tainted 4.14.15-1.el7.elrepo.x86_64 #1
  934. Jun 13 12:41:54 online-node-81-113 kernel: Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.3.4 11/08/2016
  935. Jun 13 12:41:54 online-node-81-113 kernel: Call Trace:
  936. Jun 13 12:41:54 online-node-81-113 kernel: dump_stack+0x63/0x85
  937. Jun 13 12:41:54 online-node-81-113 kernel: dump_header+0x9f/0x234
  938. Jun 13 12:41:54 online-node-81-113 kernel: ? mem_cgroup_scan_tasks+0x96/0xf0
  939. Jun 13 12:41:54 online-node-81-113 kernel: oom_kill_process+0x21c/0x430
  940. Jun 13 12:41:54 online-node-81-113 kernel: out_of_memory+0x114/0x4a0
  941. Jun 13 12:41:54 online-node-81-113 kernel: mem_cgroup_out_of_memory+0x4b/0x80
  942. Jun 13 12:41:54 online-node-81-113 kernel: mem_cgroup_oom_synchronize+0x2f9/0x320
  943. Jun 13 12:41:54 online-node-81-113 kernel: ? get_mctgt_type_thp.isra.30+0xc0/0xc0
  944. Jun 13 12:41:54 online-node-81-113 kernel: pagefault_out_of_memory+0x36/0x7c
  945. Jun 13 12:41:54 online-node-81-113 kernel: mm_fault_error+0x65/0x152
  946. Jun 13 12:41:54 online-node-81-113 kernel: __do_page_fault+0x456/0x4f0
  947. Jun 13 12:41:54 online-node-81-113 kernel: do_page_fault+0x38/0x130
  948. Jun 13 12:41:54 online-node-81-113 kernel: ? page_fault+0x36/0x60
  949. Jun 13 12:41:54 online-node-81-113 kernel: page_fault+0x4c/0x60
  950. Jun 13 12:41:54 online-node-81-113 kernel: RIP: 0033:0x400728
  951. Jun 13 12:41:54 online-node-81-113 kernel: RSP: 002b:00007fffe4898830 EFLAGS: 00010206
  952. Jun 13 12:41:54 online-node-81-113 kernel: RAX: 0000000020e67000 RBX: 0000000000000000 RCX: 0000000000000012
  953. Jun 13 12:41:54 online-node-81-113 kernel: RDX: 0000000000000000 RSI: 000000007fffffed RDI: 0000000000000000
  954. Jun 13 12:41:54 online-node-81-113 kernel: RBP: 00007fffe4898870 R08: 0000000000000000 R09: 00007f4d4e81d14d
  955. Jun 13 12:41:54 online-node-81-113 kernel: R10: 0000000000000022 R11: 0000000000000000 R12: 0000000000400530
  956. Jun 13 12:41:54 online-node-81-113 kernel: R13: 00007fffe4898970 R14: 0000000000000000 R15: 0000000000000000
  957. Jun 13 12:41:54 online-node-81-113 kernel: Task in /docker/f47120aab1019f857761657b8271185c1cb88372b620c4efdd718570980c24c3 killed as a result of limit of /docker/f47120aab1019f857761657b8271185c1cb88372b620c4efdd718570980c24c3
  958. Jun 13 12:41:54 online-node-81-113 kernel: memory: usage 524288kB, limit 524288kB, failcnt 0
  959. Jun 13 12:41:54 online-node-81-113 kernel: memory+swap: usage 524288kB, limit 524288kB, failcnt 40
  960. Jun 13 12:41:54 online-node-81-113 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
  961. Jun 13 12:41:54 online-node-81-113 kernel: Memory cgroup stats for /docker/f47120aab1019f857761657b8271185c1cb88372b620c4efdd718570980c24c3: cache:0KB rss:524288KB rss_huge:522240KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:524268KB inactive_file:0KB active_file:0KB unevictable:0KB
  962. Jun 13 12:41:54 online-node-81-113 kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
  963. Jun 13 12:41:54 online-node-81-113 kernel: [21921] 0 21921 263199 131345 264 3 0 0 thrash
  964. Jun 13 12:41:54 online-node-81-113 kernel: Memory cgroup out of memory: Kill process 21921 (thrash) score 1004 or sacrifice child
  965. Jun 13 12:41:54 online-node-81-113 kernel: Killed process 21921 (thrash) total-vm:1052796kB, anon-rss:524148kB, file-rss:1232kB, shmem-rss:0kB
  966. Jun 13 12:41:54 online-node-81-113 kernel: oom_reaper: reaped process 21921 (thrash), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
  967. Jun 13 12:43:39 online-node-81-113 kernel: thrash invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=0
  968. Jun 13 12:43:39 online-node-81-113 kernel: thrash cpuset=ce6ecaf88b8a28ff97fde0ee02442402ea4aea93c7fc176cf1048da31919c382 mems_allowed=0-1
  969. Jun 13 12:43:39 online-node-81-113 kernel: CPU: 1 PID: 4173 Comm: thrash Not tainted 4.14.15-1.el7.elrepo.x86_64 #1
  970. Jun 13 12:43:39 online-node-81-113 kernel: Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.3.4 11/08/2016
  971. Jun 13 12:43:39 online-node-81-113 kernel: Call Trace:
  972. Jun 13 12:43:39 online-node-81-113 kernel: dump_stack+0x63/0x85
  973. Jun 13 12:43:39 online-node-81-113 kernel: dump_header+0x9f/0x234
  974. Jun 13 12:43:39 online-node-81-113 kernel: ? mem_cgroup_scan_tasks+0x96/0xf0
  975. Jun 13 12:43:39 online-node-81-113 kernel: oom_kill_process+0x21c/0x430
  976. Jun 13 12:43:39 online-node-81-113 kernel: out_of_memory+0x114/0x4a0
  977. Jun 13 12:43:39 online-node-81-113 kernel: mem_cgroup_out_of_memory+0x4b/0x80
  978. Jun 13 12:43:39 online-node-81-113 kernel: mem_cgroup_oom_synchronize+0x2f9/0x320
  979. Jun 13 12:43:39 online-node-81-113 kernel: ? get_mctgt_type_thp.isra.30+0xc0/0xc0
  980. Jun 13 12:43:39 online-node-81-113 kernel: pagefault_out_of_memory+0x36/0x7c
  981. Jun 13 12:43:39 online-node-81-113 kernel: mm_fault_error+0x65/0x152
  982. Jun 13 12:43:39 online-node-81-113 kernel: __do_page_fault+0x456/0x4f0
  983. Jun 13 12:43:39 online-node-81-113 kernel: do_page_fault+0x38/0x130
  984. Jun 13 12:43:39 online-node-81-113 kernel: ? page_fault+0x36/0x60
  985. Jun 13 12:43:39 online-node-81-113 kernel: page_fault+0x4c/0x60
  986. Jun 13 12:43:39 online-node-81-113 kernel: RIP: 0033:0x400728
  987. Jun 13 12:43:39 online-node-81-113 kernel: RSP: 002b:00007fff626fe530 EFLAGS: 00010206
  988. Jun 13 12:43:39 online-node-81-113 kernel: RAX: 000000002106a000 RBX: 0000000000000000 RCX: 0000000000000012
  989. Jun 13 12:43:39 online-node-81-113 kernel: RDX: 0000000000000000 RSI: 000000007fffffed RDI: 0000000000000000
  990. Jun 13 12:43:39 online-node-81-113 kernel: RBP: 00007fff626fe570 R08: 0000000000000000 R09: 00007fe9d7a5614d
  991. Jun 13 12:43:39 online-node-81-113 kernel: R10: 0000000000000022 R11: 0000000000000000 R12: 0000000000400530
  992. Jun 13 12:43:39 online-node-81-113 kernel: R13: 00007fff626fe670 R14: 0000000000000000 R15: 0000000000000000
  993. Jun 13 12:43:39 online-node-81-113 kernel: Task in /docker/ce6ecaf88b8a28ff97fde0ee02442402ea4aea93c7fc176cf1048da31919c382 killed as a result of limit of /docker/ce6ecaf88b8a28ff97fde0ee02442402ea4aea93c7fc176cf1048da31919c382
  994. Jun 13 12:43:39 online-node-81-113 kernel: memory: usage 524288kB, limit 524288kB, failcnt 0
  995. Jun 13 12:43:39 online-node-81-113 kernel: memory+swap: usage 524288kB, limit 524288kB, failcnt 28
  996. Jun 13 12:43:39 online-node-81-113 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
  997. Jun 13 12:43:39 online-node-81-113 kernel: Memory cgroup stats for /docker/ce6ecaf88b8a28ff97fde0ee02442402ea4aea93c7fc176cf1048da31919c382: cache:0KB rss:524288KB rss_huge:522240KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:524256KB inactive_file:0KB active_file:0KB unevictable:0KB
  998. Jun 13 12:43:39 online-node-81-113 kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
  999. Jun 13 12:43:39 online-node-81-113 kernel: [ 4173] 0 4173 263199 131337 263 3 0 0 thrash
  1000. Jun 13 12:43:39 online-node-81-113 kernel: Memory cgroup out of memory: Kill process 4173 (thrash) score 1004 or sacrifice child
  1001. Jun 13 12:43:39 online-node-81-113 kernel: Killed process 4173 (thrash) total-vm:1052796kB, anon-rss:524152kB, file-rss:1196kB, shmem-rss:0kB
  1002. Jun 13 12:43:39 online-node-81-113 kernel: oom_reaper: reaped process 4173 (thrash), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
  1003. Jun 13 12:57:17 online-node-81-113 kernel: php invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=0
  1004. Jun 13 12:57:17 online-node-81-113 kernel: php cpuset=f2c0cfcbd7b21c4d713c942ba85eb4885ffa8cfd42394f03537053279e425584 mems_allowed=0-1
  1005. Jun 13 12:57:17 online-node-81-113 kernel: CPU: 31 PID: 6406 Comm: php Not tainted 4.14.15-1.el7.elrepo.x86_64 #1
  1006. Jun 13 12:57:17 online-node-81-113 kernel: Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.3.4 11/08/2016
  1007. Jun 13 12:57:17 online-node-81-113 kernel: Call Trace:
  1008. Jun 13 12:57:17 online-node-81-113 kernel: dump_stack+0x63/0x85
  1009. Jun 13 12:57:17 online-node-81-113 kernel: dump_header+0x9f/0x234
  1010. Jun 13 12:57:17 online-node-81-113 kernel: ? mem_cgroup_scan_tasks+0x96/0xf0
  1011. Jun 13 12:57:17 online-node-81-113 kernel: oom_kill_process+0x21c/0x430
  1012. Jun 13 12:57:17 online-node-81-113 kernel: out_of_memory+0x114/0x4a0
  1013. Jun 13 12:57:17 online-node-81-113 kernel: mem_cgroup_out_of_memory+0x4b/0x80
  1014. Jun 13 12:57:17 online-node-81-113 kernel: mem_cgroup_oom_synchronize+0x2f9/0x320
  1015. Jun 13 12:57:17 online-node-81-113 kernel: ? get_mctgt_type_thp.isra.30+0xc0/0xc0
  1016. Jun 13 12:57:17 online-node-81-113 kernel: pagefault_out_of_memory+0x36/0x7c
  1017. Jun 13 12:57:17 online-node-81-113 kernel: mm_fault_error+0x65/0x152
  1018. Jun 13 12:57:17 online-node-81-113 kernel: __do_page_fault+0x456/0x4f0
  1019. Jun 13 12:57:17 online-node-81-113 kernel: do_page_fault+0x38/0x130
  1020. Jun 13 12:57:17 online-node-81-113 kernel: ? page_fault+0x36/0x60
  1021. Jun 13 12:57:17 online-node-81-113 kernel: page_fault+0x4c/0x60
  1022. Jun 13 12:57:17 online-node-81-113 kernel: RIP: 0033:0x7f302fee96bf
  1023. Jun 13 12:57:17 online-node-81-113 kernel: RSP: 002b:00007ffd68db7748 EFLAGS: 00010206
  1024. Jun 13 12:57:17 online-node-81-113 kernel: RAX: 00007f30291fc040 RBX: 0000000002800000 RCX: 00000000003e9008
  1025. Jun 13 12:57:17 online-node-81-113 kernel: RDX: 0000000002800000 RSI: 00007f30272b4000 RDI: 00007f3029ab4000
  1026. Jun 13 12:57:17 online-node-81-113 kernel: RBP: 00007f3030114420 R08: 0000000006440000 R09: 00007f302cdfc048
  1027. Jun 13 12:57:17 online-node-81-113 kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 00007f30269fc040
  1028. Jun 13 12:57:17 online-node-81-113 kernel: R13: 00007f302cdfc040 R14: 0000000006400000 R15: 00005582b3c939e0
  1029. Jun 13 12:57:17 online-node-81-113 kernel: Task in /docker/f2c0cfcbd7b21c4d713c942ba85eb4885ffa8cfd42394f03537053279e425584 killed as a result of limit of /docker/f2c0cfcbd7b21c4d713c942ba85eb4885ffa8cfd42394f03537053279e425584
  1030. Jun 13 12:57:17 online-node-81-113 kernel: memory: usage 57344kB, limit 57344kB, failcnt 25
  1031. Jun 13 12:57:17 online-node-81-113 kernel: memory+swap: usage 57344kB, limit 114688kB, failcnt 0
  1032. Jun 13 12:57:17 online-node-81-113 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
  1033. Jun 13 12:57:17 online-node-81-113 kernel: Memory cgroup stats for /docker/f2c0cfcbd7b21c4d713c942ba85eb4885ffa8cfd42394f03537053279e425584: cache:0KB rss:57344KB rss_huge:51200KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:57296KB inactive_file:0KB active_file:0KB unevictable:0KB
  1034. Jun 13 12:57:17 online-node-81-113 kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
  1035. Jun 13 12:57:17 online-node-81-113 kernel: [ 6406] 0 6406 42688 17037 65 3 0 0 php
  1036. Jun 13 12:57:17 online-node-81-113 kernel: Memory cgroup out of memory: Kill process 6406 (php) score 1193 or sacrifice child
  1037. Jun 13 12:57:17 online-node-81-113 kernel: Killed process 6406 (php) total-vm:170752kB, anon-rss:57096kB, file-rss:11052kB, shmem-rss:0kB
  1038. Jun 13 12:57:17 online-node-81-113 kernel: oom_reaper: reaped process 6406 (php), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
  1039. Jun 13 13:28:28 online-node-81-113 kernel: php invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=0
  1040. Jun 13 13:28:28 online-node-81-113 kernel: php cpuset=4ed8f6475d229008a46f46d7fd1e33d7ff591f176c293cdfaa4ec240619ecddc mems_allowed=0-1
  1041. Jun 13 13:28:28 online-node-81-113 kernel: CPU: 28 PID: 25345 Comm: php Not tainted 4.14.15-1.el7.elrepo.x86_64 #1
  1042. Jun 13 13:28:28 online-node-81-113 kernel: Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.3.4 11/08/2016
  1043. Jun 13 13:28:28 online-node-81-113 kernel: Call Trace:
  1044. Jun 13 13:28:28 online-node-81-113 kernel: dump_stack+0x63/0x85
  1045. Jun 13 13:28:28 online-node-81-113 kernel: dump_header+0x9f/0x234
  1046. Jun 13 13:28:28 online-node-81-113 kernel: ? mem_cgroup_scan_tasks+0x96/0xf0
  1047. Jun 13 13:28:28 online-node-81-113 kernel: oom_kill_process+0x21c/0x430
  1048. Jun 13 13:28:28 online-node-81-113 kernel: out_of_memory+0x114/0x4a0
  1049. Jun 13 13:28:28 online-node-81-113 kernel: mem_cgroup_out_of_memory+0x4b/0x80
  1050. Jun 13 13:28:28 online-node-81-113 kernel: mem_cgroup_oom_synchronize+0x2f9/0x320
  1051. Jun 13 13:28:28 online-node-81-113 kernel: ? get_mctgt_type_thp.isra.30+0xc0/0xc0
  1052. Jun 13 13:28:28 online-node-81-113 kernel: pagefault_out_of_memory+0x36/0x7c
  1053. Jun 13 13:28:28 online-node-81-113 kernel: mm_fault_error+0x65/0x152
  1054. Jun 13 13:28:28 online-node-81-113 kernel: __do_page_fault+0x456/0x4f0
  1055. Jun 13 13:28:28 online-node-81-113 kernel: do_page_fault+0x38/0x130
  1056. Jun 13 13:28:28 online-node-81-113 kernel: ? page_fault+0x36/0x60
  1057. Jun 13 13:28:28 online-node-81-113 kernel: page_fault+0x4c/0x60
  1058. Jun 13 13:28:28 online-node-81-113 kernel: RIP: 0033:0x7f4861d086bf
  1059. Jun 13 13:28:28 online-node-81-113 kernel: RSP: 002b:00007ffc7c9fa7d8 EFLAGS: 00010206
  1060. Jun 13 13:28:28 online-node-81-113 kernel: RAX: 00007f485b01b040 RBX: 0000000002800000 RCX: 00000000003a9208
  1061. Jun 13 13:28:28 online-node-81-113 kernel: RDX: 0000000002800000 RSI: 00007f48592d2000 RDI: 00007f485bad2000
  1062. Jun 13 13:28:28 online-node-81-113 kernel: RBP: 00007f4861f33420 R08: 0000000006440000 R09: 00007f485ec1b048
  1063. Jun 13 13:28:28 online-node-81-113 kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 00007f485881b040
  1064. Jun 13 13:28:28 online-node-81-113 kernel: R13: 00007f485ec1b040 R14: 0000000006400000 R15: 000055e9863099e0
  1065. Jun 13 13:28:28 online-node-81-113 kernel: Task in /docker/4ed8f6475d229008a46f46d7fd1e33d7ff591f176c293cdfaa4ec240619ecddc killed as a result of limit of /docker/4ed8f6475d229008a46f46d7fd1e33d7ff591f176c293cdfaa4ec240619ecddc
  1066. Jun 13 13:28:28 online-node-81-113 kernel: memory: usage 57344kB, limit 57344kB, failcnt 28
  1067. Jun 13 13:28:28 online-node-81-113 kernel: memory+swap: usage 57344kB, limit 114688kB, failcnt 0
  1068. Jun 13 13:28:28 online-node-81-113 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
  1069. Jun 13 13:28:28 online-node-81-113 kernel: Memory cgroup stats for /docker/4ed8f6475d229008a46f46d7fd1e33d7ff591f176c293cdfaa4ec240619ecddc: cache:0KB rss:57344KB rss_huge:49152KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:57344KB inactive_file:0KB active_file:0KB unevictable:0KB
  1070. Jun 13 13:28:28 online-node-81-113 kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
  1071. Jun 13 13:28:28 online-node-81-113 kernel: [25345] 0 25345 42688 17068 63 3 0 0 php
  1072. Jun 13 13:28:28 online-node-81-113 kernel: Memory cgroup out of memory: Kill process 25345 (php) score 1195 or sacrifice child
  1073. Jun 13 13:28:28 online-node-81-113 kernel: Killed process 25345 (php) total-vm:170752kB, anon-rss:57176kB, file-rss:11096kB, shmem-rss:0kB
  1074. Jun 13 13:28:28 online-node-81-113 kernel: oom_reaper: reaped process 25345 (php), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!