API tools faq
paste
Advertisement
Guest User

system hang on busy NFS server

a guest
Aug 23rd, 2013
161
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 108.56 KB | None | 0 0
raw download clone embed print report
  1. Aug 23 11:10:20 row1r05s03 kernel: swapper: page allocation failure. order:0, mode:0x20
  2. Aug 23 11:10:20 row1r05s03 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.19.1.el6.x86_64 #1
  3. Aug 23 11:10:20 row1r05s03 kernel: Call Trace:
  4. Aug 23 11:10:20 row1r05s03 kernel: <IRQ> [<ffffffff811231ff>] ? __alloc_pages_nodemask+0x77f/0x940
  5. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115d1a2>] ? kmem_getpages+0x62/0x170
  6. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ddba>] ? fallback_alloc+0x1ba/0x270
  7. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  8. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115db39>] ? ____cache_alloc_node+0x99/0x160
  9. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ed00>] ? kmem_cache_alloc_node_trace+0x90/0x200
  10. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ef1d>] ? __kmalloc_node+0x4d/0x60
  11. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8141ea1d>] ? __alloc_skb+0x6d/0x190
  12. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8141eba4>] ? __netdev_alloc_skb+0x24/0x50
  13. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffffa013b86c>] ? bnx2_poll_work+0x85c/0x1280 [bnx2]
  14. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8127ddb0>] ? swiotlb_map_page+0x0/0x100
  15. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffffa013c2cd>] ? bnx2_poll_msix+0x3d/0xc0 [bnx2]
  16. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff813fb7e8>] ? dma_issue_pending_all+0x68/0xa0
  17. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8142d7c3>] ? net_rx_action+0x103/0x2f0
  18. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81094cd3>] ? hrtimer_get_next_event+0xc3/0x100
  19. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81072991>] ? __do_softirq+0xc1/0x1e0
  20. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff810d8b40>] ? handle_IRQ_event+0x60/0x170
  21. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
  22. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
  23. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81072775>] ? irq_exit+0x85/0x90
  24. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff814f1eb5>] ? do_IRQ+0x75/0xf0
  25. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
  26. Aug 23 11:10:20 row1r05s03 kernel: <EOI> [<ffffffff812c2d4e>] ? intel_idle+0xde/0x170
  27. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff812c2d31>] ? intel_idle+0xc1/0x170
  28. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff813f6c67>] ? cpuidle_idle_call+0xa7/0x140
  29. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
  30. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff814d109a>] ? rest_init+0x7a/0x80
  31. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
  32. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
  33. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
  34. Aug 23 11:10:20 row1r05s03 kernel: swapper: page allocation failure. order:0, mode:0x20
  35. Aug 23 11:10:20 row1r05s03 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.19.1.el6.x86_64 #1
  36. Aug 23 11:10:20 row1r05s03 kernel: Call Trace:
  37. Aug 23 11:10:20 row1r05s03 kernel: <IRQ> [<ffffffff811231ff>] ? __alloc_pages_nodemask+0x77f/0x940
  38. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115d1a2>] ? kmem_getpages+0x62/0x170
  39. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ddba>] ? fallback_alloc+0x1ba/0x270
  40. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  41. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115db39>] ? ____cache_alloc_node+0x99/0x160
  42. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ed00>] ? kmem_cache_alloc_node_trace+0x90/0x200
  43. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ef1d>] ? __kmalloc_node+0x4d/0x60
  44. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8141ea1d>] ? __alloc_skb+0x6d/0x190
  45. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8141eba4>] ? __netdev_alloc_skb+0x24/0x50
  46. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffffa013b86c>] ? bnx2_poll_work+0x85c/0x1280 [bnx2]
  47. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8127ddb0>] ? swiotlb_map_page+0x0/0x100
  48. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffffa013c2cd>] ? bnx2_poll_msix+0x3d/0xc0 [bnx2]
  49. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff813fb7e8>] ? dma_issue_pending_all+0x68/0xa0
  50. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8142d7c3>] ? net_rx_action+0x103/0x2f0
  51. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81094cd3>] ? hrtimer_get_next_event+0xc3/0x100
  52. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81072991>] ? __do_softirq+0xc1/0x1e0
  53. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff810d8b40>] ? handle_IRQ_event+0x60/0x170
  54. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
  55. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
  56. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81072775>] ? irq_exit+0x85/0x90
  57. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff814f1eb5>] ? do_IRQ+0x75/0xf0
  58. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
  59. Aug 23 11:10:20 row1r05s03 kernel: <EOI> [<ffffffff812c2d4e>] ? intel_idle+0xde/0x170
  60. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff812c2d31>] ? intel_idle+0xc1/0x170
  61. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff813f6c67>] ? cpuidle_idle_call+0xa7/0x140
  62. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
  63. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff814d109a>] ? rest_init+0x7a/0x80
  64. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
  65. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
  66. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
  67. Aug 23 11:10:20 row1r05s03 kernel: swapper: page allocation failure. order:0, mode:0x20
  68. Aug 23 11:10:20 row1r05s03 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.19.1.el6.x86_64 #1
  69. Aug 23 11:10:20 row1r05s03 kernel: Call Trace:
  70. Aug 23 11:10:20 row1r05s03 kernel: <IRQ> [<ffffffff811231ff>] ? __alloc_pages_nodemask+0x77f/0x940
  71. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115d1a2>] ? kmem_getpages+0x62/0x170
  72. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ddba>] ? fallback_alloc+0x1ba/0x270
  73. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  74. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115db39>] ? ____cache_alloc_node+0x99/0x160
  75. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ed00>] ? kmem_cache_alloc_node_trace+0x90/0x200
  76. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ef1d>] ? __kmalloc_node+0x4d/0x60
  77. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8141ea1d>] ? __alloc_skb+0x6d/0x190
  78. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8141eba4>] ? __netdev_alloc_skb+0x24/0x50
  79. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffffa013b86c>] ? bnx2_poll_work+0x85c/0x1280 [bnx2]
  80. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8127ddb0>] ? swiotlb_map_page+0x0/0x100
  81. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffffa013c2cd>] ? bnx2_poll_msix+0x3d/0xc0 [bnx2]
  82. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff813fb7e8>] ? dma_issue_pending_all+0x68/0xa0
  83. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8142d7c3>] ? net_rx_action+0x103/0x2f0
  84. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81094cd3>] ? hrtimer_get_next_event+0xc3/0x100
  85. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81072991>] ? __do_softirq+0xc1/0x1e0
  86. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff810d8b40>] ? handle_IRQ_event+0x60/0x170
  87. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
  88. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
  89. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81072775>] ? irq_exit+0x85/0x90
  90. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff814f1eb5>] ? do_IRQ+0x75/0xf0
  91. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
  92. Aug 23 11:10:20 row1r05s03 kernel: <EOI> [<ffffffff812c2d4e>] ? intel_idle+0xde/0x170
  93. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff812c2d31>] ? intel_idle+0xc1/0x170
  94. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff813f6c67>] ? cpuidle_idle_call+0xa7/0x140
  95. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
  96. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff814d109a>] ? rest_init+0x7a/0x80
  97. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
  98. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
  99. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
  100. Aug 23 11:10:20 row1r05s03 kernel: swapper: page allocation failure. order:0, mode:0x20
  101. Aug 23 11:10:20 row1r05s03 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.19.1.el6.x86_64 #1
  102. Aug 23 11:10:20 row1r05s03 kernel: Call Trace:
  103. Aug 23 11:10:20 row1r05s03 kernel: <IRQ> [<ffffffff811231ff>] ? __alloc_pages_nodemask+0x77f/0x940
  104. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115d1a2>] ? kmem_getpages+0x62/0x170
  105. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ddba>] ? fallback_alloc+0x1ba/0x270
  106. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  107. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115db39>] ? ____cache_alloc_node+0x99/0x160
  108. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ed00>] ? kmem_cache_alloc_node_trace+0x90/0x200
  109. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ef1d>] ? __kmalloc_node+0x4d/0x60
  110. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8141ea1d>] ? __alloc_skb+0x6d/0x190
  111. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8141eba4>] ? __netdev_alloc_skb+0x24/0x50
  112. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffffa013b86c>] ? bnx2_poll_work+0x85c/0x1280 [bnx2]
  113. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8127ddb0>] ? swiotlb_map_page+0x0/0x100
  114. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffffa013c2cd>] ? bnx2_poll_msix+0x3d/0xc0 [bnx2]
  115. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff813fb7e8>] ? dma_issue_pending_all+0x68/0xa0
  116. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8142d7c3>] ? net_rx_action+0x103/0x2f0
  117. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81094cd3>] ? hrtimer_get_next_event+0xc3/0x100
  118. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81072991>] ? __do_softirq+0xc1/0x1e0
  119. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff810d8b40>] ? handle_IRQ_event+0x60/0x170
  120. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
  121. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
  122. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81072775>] ? irq_exit+0x85/0x90
  123. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff814f1eb5>] ? do_IRQ+0x75/0xf0
  124. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
  125. Aug 23 11:10:20 row1r05s03 kernel: <EOI> [<ffffffff812c2d4e>] ? intel_idle+0xde/0x170
  126. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff812c2d31>] ? intel_idle+0xc1/0x170
  127. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff813f6c67>] ? cpuidle_idle_call+0xa7/0x140
  128. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
  129. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff814d109a>] ? rest_init+0x7a/0x80
  130. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
  131. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
  132. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
  133. Aug 23 11:10:20 row1r05s03 kernel: swapper: page allocation failure. order:0, mode:0x20
  134. Aug 23 11:10:20 row1r05s03 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.19.1.el6.x86_64 #1
  135. Aug 23 11:10:20 row1r05s03 kernel: Call Trace:
  136. Aug 23 11:10:20 row1r05s03 kernel: <IRQ> [<ffffffff811231ff>] ? __alloc_pages_nodemask+0x77f/0x940
  137. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115d1a2>] ? kmem_getpages+0x62/0x170
  138. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ddba>] ? fallback_alloc+0x1ba/0x270
  139. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  140. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115db39>] ? ____cache_alloc_node+0x99/0x160
  141. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ed00>] ? kmem_cache_alloc_node_trace+0x90/0x200
  142. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ef1d>] ? __kmalloc_node+0x4d/0x60
  143. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8141ea1d>] ? __alloc_skb+0x6d/0x190
  144. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8141eba4>] ? __netdev_alloc_skb+0x24/0x50
  145. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffffa013b86c>] ? bnx2_poll_work+0x85c/0x1280 [bnx2]
  146. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8127ddb0>] ? swiotlb_map_page+0x0/0x100
  147. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffffa013c2cd>] ? bnx2_poll_msix+0x3d/0xc0 [bnx2]
  148. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff813fb7e8>] ? dma_issue_pending_all+0x68/0xa0
  149. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8142d7c3>] ? net_rx_action+0x103/0x2f0
  150. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81094cd3>] ? hrtimer_get_next_event+0xc3/0x100
  151. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81072991>] ? __do_softirq+0xc1/0x1e0
  152. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff810d8b40>] ? handle_IRQ_event+0x60/0x170
  153. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
  154. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
  155. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81072775>] ? irq_exit+0x85/0x90
  156. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff814f1eb5>] ? do_IRQ+0x75/0xf0
  157. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
  158. Aug 23 11:10:20 row1r05s03 kernel: <EOI> [<ffffffff812c2d4e>] ? intel_idle+0xde/0x170
  159. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff812c2d31>] ? intel_idle+0xc1/0x170
  160. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff813f6c67>] ? cpuidle_idle_call+0xa7/0x140
  161. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
  162. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff814d109a>] ? rest_init+0x7a/0x80
  163. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
  164. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
  165. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
  166. Aug 23 11:10:20 row1r05s03 kernel: swapper: page allocation failure. order:0, mode:0x20
  167. Aug 23 11:10:20 row1r05s03 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.19.1.el6.x86_64 #1
  168. Aug 23 11:10:20 row1r05s03 kernel: Call Trace:
  169. Aug 23 11:10:20 row1r05s03 kernel: <IRQ> [<ffffffff811231ff>] ? __alloc_pages_nodemask+0x77f/0x940
  170. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115d1a2>] ? kmem_getpages+0x62/0x170
  171. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ddba>] ? fallback_alloc+0x1ba/0x270
  172. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  173. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115db39>] ? ____cache_alloc_node+0x99/0x160
  174. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ed00>] ? kmem_cache_alloc_node_trace+0x90/0x200
  175. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ef1d>] ? __kmalloc_node+0x4d/0x60
  176. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8141ea1d>] ? __alloc_skb+0x6d/0x190
  177. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8141eba4>] ? __netdev_alloc_skb+0x24/0x50
  178. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffffa013b86c>] ? bnx2_poll_work+0x85c/0x1280 [bnx2]
  179. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8127ddb0>] ? swiotlb_map_page+0x0/0x100
  180. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffffa013c2cd>] ? bnx2_poll_msix+0x3d/0xc0 [bnx2]
  181. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff813fb7e8>] ? dma_issue_pending_all+0x68/0xa0
  182. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8142d7c3>] ? net_rx_action+0x103/0x2f0
  183. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81094cd3>] ? hrtimer_get_next_event+0xc3/0x100
  184. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81072991>] ? __do_softirq+0xc1/0x1e0
  185. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff810d8b40>] ? handle_IRQ_event+0x60/0x170
  186. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
  187. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
  188. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81072775>] ? irq_exit+0x85/0x90
  189. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff814f1eb5>] ? do_IRQ+0x75/0xf0
  190. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
  191. Aug 23 11:10:20 row1r05s03 kernel: <EOI> [<ffffffff812c2d4e>] ? intel_idle+0xde/0x170
  192. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff812c2d31>] ? intel_idle+0xc1/0x170
  193. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff813f6c67>] ? cpuidle_idle_call+0xa7/0x140
  194. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
  195. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff814d109a>] ? rest_init+0x7a/0x80
  196. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
  197. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
  198. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
  199. Aug 23 11:10:20 row1r05s03 kernel: swapper: page allocation failure. order:0, mode:0x20
  200. Aug 23 11:10:20 row1r05s03 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.19.1.el6.x86_64 #1
  201. Aug 23 11:10:20 row1r05s03 kernel: Call Trace:
  202. Aug 23 11:10:20 row1r05s03 kernel: <IRQ> [<ffffffff811231ff>] ? __alloc_pages_nodemask+0x77f/0x940
  203. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115d1a2>] ? kmem_getpages+0x62/0x170
  204. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ddba>] ? fallback_alloc+0x1ba/0x270
  205. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  206. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115db39>] ? ____cache_alloc_node+0x99/0x160
  207. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ed00>] ? kmem_cache_alloc_node_trace+0x90/0x200
  208. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ef1d>] ? __kmalloc_node+0x4d/0x60
  209. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8141ea1d>] ? __alloc_skb+0x6d/0x190
  210. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8141eba4>] ? __netdev_alloc_skb+0x24/0x50
  211. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffffa013b86c>] ? bnx2_poll_work+0x85c/0x1280 [bnx2]
  212. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8127ddb0>] ? swiotlb_map_page+0x0/0x100
  213. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffffa013c2cd>] ? bnx2_poll_msix+0x3d/0xc0 [bnx2]
  214. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff813fb7e8>] ? dma_issue_pending_all+0x68/0xa0
  215. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8142d7c3>] ? net_rx_action+0x103/0x2f0
  216. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81094cd3>] ? hrtimer_get_next_event+0xc3/0x100
  217. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81072991>] ? __do_softirq+0xc1/0x1e0
  218. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff810d8b40>] ? handle_IRQ_event+0x60/0x170
  219. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
  220. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
  221. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81072775>] ? irq_exit+0x85/0x90
  222. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff814f1eb5>] ? do_IRQ+0x75/0xf0
  223. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
  224. Aug 23 11:10:20 row1r05s03 kernel: <EOI> [<ffffffff812c2d4e>] ? intel_idle+0xde/0x170
  225. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff812c2d31>] ? intel_idle+0xc1/0x170
  226. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff813f6c67>] ? cpuidle_idle_call+0xa7/0x140
  227. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
  228. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff814d109a>] ? rest_init+0x7a/0x80
  229. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
  230. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
  231. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
  232. Aug 23 11:10:20 row1r05s03 kernel: swapper: page allocation failure. order:0, mode:0x20
  233. Aug 23 11:10:20 row1r05s03 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.19.1.el6.x86_64 #1
  234. Aug 23 11:10:20 row1r05s03 kernel: Call Trace:
  235. Aug 23 11:10:20 row1r05s03 kernel: <IRQ> [<ffffffff811231ff>] ? __alloc_pages_nodemask+0x77f/0x940
  236. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115d1a2>] ? kmem_getpages+0x62/0x170
  237. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ddba>] ? fallback_alloc+0x1ba/0x270
  238. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  239. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115db39>] ? ____cache_alloc_node+0x99/0x160
  240. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ed00>] ? kmem_cache_alloc_node_trace+0x90/0x200
  241. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ef1d>] ? __kmalloc_node+0x4d/0x60
  242. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8141ea1d>] ? __alloc_skb+0x6d/0x190
  243. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8141eba4>] ? __netdev_alloc_skb+0x24/0x50
  244. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffffa013b86c>] ? bnx2_poll_work+0x85c/0x1280 [bnx2]
  245. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8127ddb0>] ? swiotlb_map_page+0x0/0x100
  246. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffffa013c2cd>] ? bnx2_poll_msix+0x3d/0xc0 [bnx2]
  247. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff813fb7e8>] ? dma_issue_pending_all+0x68/0xa0
  248. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8142d7c3>] ? net_rx_action+0x103/0x2f0
  249. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81094cd3>] ? hrtimer_get_next_event+0xc3/0x100
  250. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81072991>] ? __do_softirq+0xc1/0x1e0
  251. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff810d8b40>] ? handle_IRQ_event+0x60/0x170
  252. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
  253. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
  254. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81072775>] ? irq_exit+0x85/0x90
  255. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff814f1eb5>] ? do_IRQ+0x75/0xf0
  256. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
  257. Aug 23 11:10:20 row1r05s03 kernel: <EOI> [<ffffffff812c2d4e>] ? intel_idle+0xde/0x170
  258. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff812c2d31>] ? intel_idle+0xc1/0x170
  259. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff813f6c67>] ? cpuidle_idle_call+0xa7/0x140
  260. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
  261. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff814d109a>] ? rest_init+0x7a/0x80
  262. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
  263. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
  264. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
  265. Aug 23 11:10:20 row1r05s03 kernel: swapper: page allocation failure. order:0, mode:0x20
  266. Aug 23 11:10:20 row1r05s03 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.19.1.el6.x86_64 #1
  267. Aug 23 11:10:20 row1r05s03 kernel: Call Trace:
  268. Aug 23 11:10:20 row1r05s03 kernel: <IRQ> [<ffffffff811231ff>] ? __alloc_pages_nodemask+0x77f/0x940
  269. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115d1a2>] ? kmem_getpages+0x62/0x170
  270. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ddba>] ? fallback_alloc+0x1ba/0x270
  271. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  272. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115db39>] ? ____cache_alloc_node+0x99/0x160
  273. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ed00>] ? kmem_cache_alloc_node_trace+0x90/0x200
  274. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ef1d>] ? __kmalloc_node+0x4d/0x60
  275. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8141ea1d>] ? __alloc_skb+0x6d/0x190
  276. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8141eba4>] ? __netdev_alloc_skb+0x24/0x50
  277. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffffa013b86c>] ? bnx2_poll_work+0x85c/0x1280 [bnx2]
  278. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8127ddb0>] ? swiotlb_map_page+0x0/0x100
  279. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffffa013c2cd>] ? bnx2_poll_msix+0x3d/0xc0 [bnx2]
  280. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff813fb7e8>] ? dma_issue_pending_all+0x68/0xa0
  281. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8142d7c3>] ? net_rx_action+0x103/0x2f0
  282. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81094cd3>] ? hrtimer_get_next_event+0xc3/0x100
  283. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81072991>] ? __do_softirq+0xc1/0x1e0
  284. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff810d8b40>] ? handle_IRQ_event+0x60/0x170
  285. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
  286. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
  287. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81072775>] ? irq_exit+0x85/0x90
  288. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff814f1eb5>] ? do_IRQ+0x75/0xf0
  289. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
  290. Aug 23 11:10:20 row1r05s03 kernel: <EOI> [<ffffffff812c2d4e>] ? intel_idle+0xde/0x170
  291. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff812c2d31>] ? intel_idle+0xc1/0x170
  292. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff813f6c67>] ? cpuidle_idle_call+0xa7/0x140
  293. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
  294. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff814d109a>] ? rest_init+0x7a/0x80
  295. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
  296. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
  297. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
  298. Aug 23 11:10:20 row1r05s03 kernel: swapper: page allocation failure. order:0, mode:0x20
  299. Aug 23 11:10:20 row1r05s03 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.19.1.el6.x86_64 #1
  300. Aug 23 11:10:20 row1r05s03 kernel: Call Trace:
  301. Aug 23 11:10:20 row1r05s03 kernel: <IRQ> [<ffffffff811231ff>] ? __alloc_pages_nodemask+0x77f/0x940
  302. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115d1a2>] ? kmem_getpages+0x62/0x170
  303. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ddba>] ? fallback_alloc+0x1ba/0x270
  304. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  305. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115db39>] ? ____cache_alloc_node+0x99/0x160
  306. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ed00>] ? kmem_cache_alloc_node_trace+0x90/0x200
  307. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ef1d>] ? __kmalloc_node+0x4d/0x60
  308. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8141ea1d>] ? __alloc_skb+0x6d/0x190
  309. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8141eba4>] ? __netdev_alloc_skb+0x24/0x50
  310. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffffa013b86c>] ? bnx2_poll_work+0x85c/0x1280 [bnx2]
  311. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8127ddb0>] ? swiotlb_map_page+0x0/0x100
  312. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffffa013c2cd>] ? bnx2_poll_msix+0x3d/0xc0 [bnx2]
  313. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff813fb7e8>] ? dma_issue_pending_all+0x68/0xa0
  314. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8142d7c3>] ? net_rx_action+0x103/0x2f0
  315. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81094cd3>] ? hrtimer_get_next_event+0xc3/0x100
  316. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81072991>] ? __do_softirq+0xc1/0x1e0
  317. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff810d8b40>] ? handle_IRQ_event+0x60/0x170
  318. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
  319. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
  320. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81072775>] ? irq_exit+0x85/0x90
  321. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff814f1eb5>] ? do_IRQ+0x75/0xf0
  322. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
  323. Aug 23 11:10:20 row1r05s03 kernel: <EOI> [<ffffffff812c2d4e>] ? intel_idle+0xde/0x170
  324. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff812c2d31>] ? intel_idle+0xc1/0x170
  325. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff813f6c67>] ? cpuidle_idle_call+0xa7/0x140
  326. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
  327. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff814d109a>] ? rest_init+0x7a/0x80
  328. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
  329. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
  330. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
  331. Aug 23 11:10:35 row1r05s03 kernel: INFO: task kswapd0:98 blocked for more than 120 seconds.
  332. Aug 23 11:10:35 row1r05s03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  333. Aug 23 11:10:35 row1r05s03 kernel: kswapd0 D 0000000000000005 0 98 2 0x00000000
  334. Aug 23 11:10:35 row1r05s03 kernel: ffff8802262ad530 0000000000000046 ffff8802262ad4f8 ffff8802262ad4f4
  335. Aug 23 11:10:35 row1r05s03 kernel: ffff8802262ad4d0 ffff88022fc28e00 ffff88002f816680 0000000000000400
  336. Aug 23 11:10:35 row1r05s03 kernel: ffff8802262abaf8 ffff8802262adfd8 000000000000fb88 ffff8802262abaf8
  337. Aug 23 11:10:35 row1r05s03 kernel: Call Trace:
  338. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff814eaac5>] schedule_timeout+0x215/0x2e0
  339. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff810966e7>] ? down_trylock+0x37/0x50
  340. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026b67f>] ? xfs_buf_cond_lock+0x2f/0xc0 [xfs]
  341. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff814eb9e2>] __down+0x72/0xb0
  342. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cae2>] ? _xfs_buf_find+0x102/0x280 [xfs]
  343. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff810967f1>] down+0x41/0x50
  344. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026c923>] xfs_buf_lock+0x53/0x110 [xfs]
  345. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cae2>] _xfs_buf_find+0x102/0x280 [xfs]
  346. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cccb>] xfs_buf_get+0x6b/0x1a0 [xfs]
  347. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026d33c>] xfs_buf_read+0x2c/0x100 [xfs]
  348. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0262f88>] xfs_trans_read_buf+0x1f8/0x400 [xfs]
  349. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0219086>] xfs_read_agf+0x66/0x1a0 [xfs]
  350. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02191f9>] xfs_alloc_read_agf+0x39/0xd0 [xfs]
  351. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa021bc01>] xfs_alloc_fix_freelist+0x411/0x480 [xfs]
  352. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0260103>] ? xfs_trans_free_item_desc+0x33/0x40 [xfs]
  353. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0260247>] ? xfs_trans_free_items+0x87/0xb0 [xfs]
  354. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa025ccf8>] ? xfs_perag_get+0x38/0xb0 [xfs]
  355. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa021bd18>] xfs_free_extent+0xa8/0x130 [xfs]
  356. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02253bd>] xfs_bmap_finish+0x15d/0x1a0 [xfs]
  357. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa024aaa7>] xfs_itruncate_finish+0x167/0x330 [xfs]
  358. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0266028>] xfs_free_eofblocks+0x1b8/0x280 [xfs]
  359. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02665c0>] xfs_inactive+0xc0/0x460 [xfs]
  360. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0276a94>] ? xfs_inode_set_reclaim_tag+0x84/0xa0 [xfs]
  361. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0273d80>] xfs_fs_clear_inode+0xa0/0xd0 [xfs]
  362. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81190c3c>] clear_inode+0xac/0x140
  363. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81190d10>] dispose_list+0x40/0x120
  364. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81191064>] shrink_icache_memory+0x274/0x2e0
  365. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81128c7a>] shrink_slab+0x12a/0x1a0
  366. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8112bb4d>] balance_pgdat+0x57d/0x7e0
  367. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8112c160>] ? isolate_pages_global+0x0/0x350
  368. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8112035d>] ? zone_watermark_ok_safe+0xad/0xc0
  369. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8112bee6>] kswapd+0x136/0x3b0
  370. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81090990>] ? autoremove_wake_function+0x0/0x40
  371. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8112bdb0>] ? kswapd+0x0/0x3b0
  372. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81090626>] kthread+0x96/0xa0
  373. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
  374. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81090590>] ? kthread+0x0/0xa0
  375. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
  376. Aug 23 11:10:35 row1r05s03 kernel: INFO: task nfsd:2687 blocked for more than 120 seconds.
  377. Aug 23 11:10:35 row1r05s03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  378. Aug 23 11:10:35 row1r05s03 kernel: nfsd D 0000000000000005 0 2687 2 0x00000080
  379. Aug 23 11:10:35 row1r05s03 kernel: ffff8802171d6960 0000000000000046 0000000000000000 0000000000000000
  380. Aug 23 11:10:35 row1r05s03 kernel: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
  381. Aug 23 11:10:35 row1r05s03 kernel: ffff8802277e7098 ffff8802171d7fd8 000000000000fb88 ffff8802277e7098
  382. Aug 23 11:10:35 row1r05s03 kernel: Call Trace:
  383. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81132ad9>] ? zone_statistics+0x99/0xc0
  384. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff814eaac5>] schedule_timeout+0x215/0x2e0
  385. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff810966e7>] ? down_trylock+0x37/0x50
  386. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026b67f>] ? xfs_buf_cond_lock+0x2f/0xc0 [xfs]
  387. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff814eb9e2>] __down+0x72/0xb0
  388. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cae2>] ? _xfs_buf_find+0x102/0x280 [xfs]
  389. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff810967f1>] down+0x41/0x50
  390. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026c923>] xfs_buf_lock+0x53/0x110 [xfs]
  391. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cae2>] _xfs_buf_find+0x102/0x280 [xfs]
  392. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cccb>] xfs_buf_get+0x6b/0x1a0 [xfs]
  393. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026d33c>] xfs_buf_read+0x2c/0x100 [xfs]
  394. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0262f88>] xfs_trans_read_buf+0x1f8/0x400 [xfs]
  395. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0219086>] xfs_read_agf+0x66/0x1a0 [xfs]
  396. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02191f9>] xfs_alloc_read_agf+0x39/0xd0 [xfs]
  397. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa021bc01>] xfs_alloc_fix_freelist+0x411/0x480 [xfs]
  398. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8115d9bb>] ? cache_alloc_refill+0x15b/0x240
  399. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa025ccf8>] ? xfs_perag_get+0x38/0xb0 [xfs]
  400. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa021bd18>] xfs_free_extent+0xa8/0x130 [xfs]
  401. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02253bd>] xfs_bmap_finish+0x15d/0x1a0 [xfs]
  402. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa024aaa7>] xfs_itruncate_finish+0x167/0x330 [xfs]
  403. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0266028>] xfs_free_eofblocks+0x1b8/0x280 [xfs]
  404. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02665c0>] xfs_inactive+0xc0/0x460 [xfs]
  405. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0276a94>] ? xfs_inode_set_reclaim_tag+0x84/0xa0 [xfs]
  406. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0273d80>] xfs_fs_clear_inode+0xa0/0xd0 [xfs]
  407. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81190c3c>] clear_inode+0xac/0x140
  408. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81190d10>] dispose_list+0x40/0x120
  409. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81191064>] shrink_icache_memory+0x274/0x2e0
  410. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81128c7a>] shrink_slab+0x12a/0x1a0
  411. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8112abff>] do_try_to_free_pages+0x2ff/0x520
  412. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81113e8f>] ? zone_watermark_ok+0x1f/0x30
  413. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8112b00d>] try_to_free_pages+0x9d/0x130
  414. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81131976>] ? next_online_pgdat+0x26/0x50
  415. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8112c160>] ? isolate_pages_global+0x0/0x350
  416. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81122e8d>] __alloc_pages_nodemask+0x40d/0x940
  417. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8115d1a2>] kmem_getpages+0x62/0x170
  418. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8115ddba>] fallback_alloc+0x1ba/0x270
  419. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  420. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8115db39>] ____cache_alloc_node+0x99/0x160
  421. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8115ed00>] kmem_cache_alloc_node_trace+0x90/0x200
  422. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8114967f>] vm_map_ram+0x1cf/0x470
  423. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026b67f>] ? xfs_buf_cond_lock+0x2f/0xc0 [xfs]
  424. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026bbb1>] _xfs_buf_map_pages+0xc1/0x120 [xfs]
  425. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cd63>] xfs_buf_get+0x103/0x1a0 [xfs]
  426. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026d33c>] xfs_buf_read+0x2c/0x100 [xfs]
  427. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0262f88>] xfs_trans_read_buf+0x1f8/0x400 [xfs]
  428. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa023048e>] xfs_btree_read_buf_block+0x5e/0xd0 [xfs]
  429. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa022e8cc>] ? xfs_btree_ptr_offset+0x4c/0x90 [xfs]
  430. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02310c2>] xfs_btree_increment+0x202/0x2a0 [xfs]
  431. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa021ac91>] xfs_alloc_ag_vextent_near+0x231/0xb20 [xfs]
  432. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa021bbdd>] ? xfs_alloc_fix_freelist+0x3ed/0x480 [xfs]
  433. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa021b655>] xfs_alloc_ag_vextent+0xd5/0x100 [xfs]
  434. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa021c216>] xfs_alloc_vextent+0x476/0x610 [xfs]
  435. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0246ca3>] xfs_ialloc_ag_alloc+0x3a3/0x4b0 [xfs]
  436. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0246e8f>] xfs_dialloc+0xdf/0x850 [xfs]
  437. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8109672f>] ? up+0x2f/0x50
  438. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026355b>] ? xfs_trans_brelse+0xdb/0x150 [xfs]
  439. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81221eda>] ? symhash+0x1a/0x50
  440. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81221eb4>] ? symcmp+0x14/0x20
  441. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81221ace>] ? hashtab_search+0x7e/0x90
  442. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa024aed0>] xfs_ialloc+0x60/0x6f0 [xfs]
  443. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0263b14>] xfs_dir_ialloc+0x74/0x2b0 [xfs]
  444. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0265b60>] xfs_create+0x440/0x640 [xfs]
  445. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0272b3d>] xfs_vn_mknod+0xad/0x1c0 [xfs]
  446. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0272c80>] xfs_vn_create+0x10/0x20 [xfs]
  447. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81183b04>] vfs_create+0xb4/0xe0
  448. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03cf6c6>] nfsd_create_v3+0x366/0x4c0 [nfsd]
  449. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03d7413>] nfsd3_proc_create+0x123/0x1b0 [nfsd]
  450. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03c843e>] nfsd_dispatch+0xfe/0x240 [nfsd]
  451. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02fc794>] svc_process_common+0x344/0x640 [sunrpc]
  452. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8105fa40>] ? default_wake_function+0x0/0x20
  453. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02fcdd0>] svc_process+0x110/0x160 [sunrpc]
  454. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03c8b62>] nfsd+0xc2/0x160 [nfsd]
  455. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03c8aa0>] ? nfsd+0x0/0x160 [nfsd]
  456. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81090626>] kthread+0x96/0xa0
  457. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
  458. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81090590>] ? kthread+0x0/0xa0
  459. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
  460. Aug 23 11:10:35 row1r05s03 kernel: INFO: task nfsd:2688 blocked for more than 120 seconds.
  461. Aug 23 11:10:35 row1r05s03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  462. Aug 23 11:10:35 row1r05s03 kernel: nfsd D 0000000000000000 0 2688 2 0x00000080
  463. Aug 23 11:10:35 row1r05s03 kernel: ffff8802172d1730 0000000000000046 0000000000000000 0000000000000003
  464. Aug 23 11:10:35 row1r05s03 kernel: ffff8802288f8020 ffff880224440900 0000000000000002 00000000814bfedc
  465. Aug 23 11:10:35 row1r05s03 kernel: ffff8802240485f8 ffff8802172d1fd8 000000000000fb88 ffff8802240485f8
  466. Aug 23 11:10:35 row1r05s03 kernel: Call Trace:
  467. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff814ec1b5>] rwsem_down_failed_common+0x95/0x1d0
  468. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff814ec313>] rwsem_down_write_failed+0x23/0x30
  469. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81275213>] call_rwsem_down_write_failed+0x13/0x20
  470. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff814eb812>] ? down_write+0x32/0x40
  471. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0247cfe>] xfs_ilock+0x7e/0xd0 [xfs]
  472. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026f1f5>] xfs_file_aio_write_checks+0x45/0x1d0 [xfs]
  473. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026f6b1>] xfs_file_buffered_aio_write+0x81/0x1a0 [xfs]
  474. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026fbdb>] xfs_file_aio_write+0x1bb/0x2e0 [xfs]
  475. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0248493>] ? xfs_iget+0x473/0x6f0 [xfs]
  476. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026fa20>] ? xfs_file_aio_write+0x0/0x2e0 [xfs]
  477. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8117614b>] do_sync_readv_writev+0xfb/0x140
  478. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81090990>] ? autoremove_wake_function+0x0/0x40
  479. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81219b4b>] ? selinux_file_permission+0xfb/0x150
  480. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8120ca26>] ? security_file_permission+0x16/0x20
  481. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff811770d6>] do_readv_writev+0xd6/0x1f0
  482. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81215ec4>] ? inode_has_perm+0x54/0xa0
  483. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81177236>] vfs_writev+0x46/0x60
  484. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03cd555>] nfsd_vfs_write+0x105/0x430 [nfsd]
  485. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81173f32>] ? dentry_open+0x52/0xc0
  486. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03cf9fb>] ? nfsd_open+0x1db/0x230 [nfsd]
  487. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03cfe17>] nfsd_write+0xe7/0x100 [nfsd]
  488. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03d754f>] nfsd3_proc_write+0xaf/0x140 [nfsd]
  489. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03c843e>] nfsd_dispatch+0xfe/0x240 [nfsd]
  490. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02fc794>] svc_process_common+0x344/0x640 [sunrpc]
  491. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8105fa40>] ? default_wake_function+0x0/0x20
  492. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02fcdd0>] svc_process+0x110/0x160 [sunrpc]
  493. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03c8b62>] nfsd+0xc2/0x160 [nfsd]
  494. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03c8aa0>] ? nfsd+0x0/0x160 [nfsd]
  495. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81090626>] kthread+0x96/0xa0
  496. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
  497. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81090590>] ? kthread+0x0/0xa0
  498. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
  499. Aug 23 11:10:35 row1r05s03 kernel: INFO: task nfsd:2689 blocked for more than 120 seconds.
  500. Aug 23 11:10:35 row1r05s03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  501. Aug 23 11:10:35 row1r05s03 kernel: nfsd D 0000000000000004 0 2689 2 0x00000080
  502. Aug 23 11:10:35 row1r05s03 kernel: ffff8802173cd5e0 0000000000000046 ffff8802173cd5c0 ffffffffa013aa51
  503. Aug 23 11:10:35 row1r05s03 kernel: 00000001f530e0ce 0000000000000001 ffff8802288f8020 ffff880224440800
  504. Aug 23 11:10:35 row1r05s03 kernel: ffff880227309ab8 ffff8802173cdfd8 000000000000fb88 ffff880227309ab8
  505. Aug 23 11:10:35 row1r05s03 kernel: Call Trace:
  506. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa013aa51>] ? bnx2_start_xmit+0x241/0x800 [bnx2]
  507. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff814eaac5>] schedule_timeout+0x215/0x2e0
  508. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81446ff8>] ? sch_direct_xmit+0x78/0x1c0
  509. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff814eb9e2>] __down+0x72/0xb0
  510. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cae2>] ? _xfs_buf_find+0x102/0x280 [xfs]
  511. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff810967f1>] down+0x41/0x50
  512. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026c923>] xfs_buf_lock+0x53/0x110 [xfs]
  513. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cae2>] _xfs_buf_find+0x102/0x280 [xfs]
  514. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cccb>] xfs_buf_get+0x6b/0x1a0 [xfs]
  515. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026d33c>] xfs_buf_read+0x2c/0x100 [xfs]
  516. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0262f88>] xfs_trans_read_buf+0x1f8/0x400 [xfs]
  517. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0245774>] xfs_read_agi+0x74/0x100 [xfs]
  518. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa024582e>] xfs_ialloc_read_agi+0x2e/0x90 [xfs]
  519. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0245b13>] xfs_ialloc_ag_select+0x133/0x270 [xfs]
  520. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0247187>] xfs_dialloc+0x3d7/0x850 [xfs]
  521. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81419fde>] ? release_sock+0xce/0xe0
  522. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81221eda>] ? symhash+0x1a/0x50
  523. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81221eb4>] ? symcmp+0x14/0x20
  524. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81221ace>] ? hashtab_search+0x7e/0x90
  525. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa024aed0>] xfs_ialloc+0x60/0x6f0 [xfs]
  526. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0263b14>] xfs_dir_ialloc+0x74/0x2b0 [xfs]
  527. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0265b60>] xfs_create+0x440/0x640 [xfs]
  528. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0272b3d>] xfs_vn_mknod+0xad/0x1c0 [xfs]
  529. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0272c80>] xfs_vn_create+0x10/0x20 [xfs]
  530. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81183b04>] vfs_create+0xb4/0xe0
  531. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03cf6c6>] nfsd_create_v3+0x366/0x4c0 [nfsd]
  532. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03d7413>] nfsd3_proc_create+0x123/0x1b0 [nfsd]
  533. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03c843e>] nfsd_dispatch+0xfe/0x240 [nfsd]
  534. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02fc794>] svc_process_common+0x344/0x640 [sunrpc]
  535. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8105fa40>] ? default_wake_function+0x0/0x20
  536. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02fcdd0>] svc_process+0x110/0x160 [sunrpc]
  537. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03c8b62>] nfsd+0xc2/0x160 [nfsd]
  538. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03c8aa0>] ? nfsd+0x0/0x160 [nfsd]
  539. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81090626>] kthread+0x96/0xa0
  540. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
  541. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81090590>] ? kthread+0x0/0xa0
  542. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
  543. Aug 23 11:10:35 row1r05s03 kernel: INFO: task nfsd:2690 blocked for more than 120 seconds.
  544. Aug 23 11:10:35 row1r05s03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  545. Aug 23 11:10:35 row1r05s03 kernel: nfsd D 0000000000000001 0 2690 2 0x00000080
  546. Aug 23 11:10:35 row1r05s03 kernel: ffff880216ce88c0 0000000000000046 0000000000000000 ffff880000023de8
  547. Aug 23 11:10:35 row1r05s03 kernel: ffff880216ce88c0 ffffffff8111fb93 ffffea00069887b8 0000000000000058
  548. Aug 23 11:10:35 row1r05s03 kernel: ffff88021e399af8 ffff880216ce9fd8 000000000000fb88 ffff88021e399af8
  549. Aug 23 11:10:35 row1r05s03 kernel: Call Trace:
  550. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8111fb93>] ? __rmqueue+0x143/0x490
  551. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff814eaac5>] schedule_timeout+0x215/0x2e0
  552. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff810966e7>] ? down_trylock+0x37/0x50
  553. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026b67f>] ? xfs_buf_cond_lock+0x2f/0xc0 [xfs]
  554. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff814eb9e2>] __down+0x72/0xb0
  555. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cae2>] ? _xfs_buf_find+0x102/0x280 [xfs]
  556. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff810967f1>] down+0x41/0x50
  557. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026c923>] xfs_buf_lock+0x53/0x110 [xfs]
  558. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cae2>] _xfs_buf_find+0x102/0x280 [xfs]
  559. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cccb>] xfs_buf_get+0x6b/0x1a0 [xfs]
  560. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026d33c>] xfs_buf_read+0x2c/0x100 [xfs]
  561. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0262f88>] xfs_trans_read_buf+0x1f8/0x400 [xfs]
  562. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0219086>] xfs_read_agf+0x66/0x1a0 [xfs]
  563. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02191f9>] xfs_alloc_read_agf+0x39/0xd0 [xfs]
  564. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa021bc01>] xfs_alloc_fix_freelist+0x411/0x480 [xfs]
  565. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0260103>] ? xfs_trans_free_item_desc+0x33/0x40 [xfs]
  566. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0260247>] ? xfs_trans_free_items+0x87/0xb0 [xfs]
  567. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa025ccf8>] ? xfs_perag_get+0x38/0xb0 [xfs]
  568. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa021bd18>] xfs_free_extent+0xa8/0x130 [xfs]
  569. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02253bd>] xfs_bmap_finish+0x15d/0x1a0 [xfs]
  570. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa024aaa7>] xfs_itruncate_finish+0x167/0x330 [xfs]
  571. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0266028>] xfs_free_eofblocks+0x1b8/0x280 [xfs]
  572. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02665c0>] xfs_inactive+0xc0/0x460 [xfs]
  573. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0276a94>] ? xfs_inode_set_reclaim_tag+0x84/0xa0 [xfs]
  574. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0273d80>] xfs_fs_clear_inode+0xa0/0xd0 [xfs]
  575. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81190c3c>] clear_inode+0xac/0x140
  576. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81190d10>] dispose_list+0x40/0x120
  577. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81191064>] shrink_icache_memory+0x274/0x2e0
  578. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81128c7a>] shrink_slab+0x12a/0x1a0
  579. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8112abff>] do_try_to_free_pages+0x2ff/0x520
  580. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81113e8f>] ? zone_watermark_ok+0x1f/0x30
  581. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8112b00d>] try_to_free_pages+0x9d/0x130
  582. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81131976>] ? next_online_pgdat+0x26/0x50
  583. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8112c160>] ? isolate_pages_global+0x0/0x350
  584. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81122e8d>] __alloc_pages_nodemask+0x40d/0x940
  585. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8115d1a2>] kmem_getpages+0x62/0x170
  586. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8115ddba>] fallback_alloc+0x1ba/0x270
  587. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  588. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8115db39>] ____cache_alloc_node+0x99/0x160
  589. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8115ed00>] kmem_cache_alloc_node_trace+0x90/0x200
  590. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8114872b>] alloc_vmap_area+0x5b/0x310
  591. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff811496b7>] vm_map_ram+0x207/0x470
  592. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026b67f>] ? xfs_buf_cond_lock+0x2f/0xc0 [xfs]
  593. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026bbb1>] _xfs_buf_map_pages+0xc1/0x120 [xfs]
  594. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cd63>] xfs_buf_get+0x103/0x1a0 [xfs]
  595. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026d33c>] xfs_buf_read+0x2c/0x100 [xfs]
  596. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0262f88>] xfs_trans_read_buf+0x1f8/0x400 [xfs]
  597. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa023048e>] xfs_btree_read_buf_block+0x5e/0xd0 [xfs]
  598. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa022e8cc>] ? xfs_btree_ptr_offset+0x4c/0x90 [xfs]
  599. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02310c2>] xfs_btree_increment+0x202/0x2a0 [xfs]
  600. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa021ac91>] xfs_alloc_ag_vextent_near+0x231/0xb20 [xfs]
  601. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa021bbdd>] ? xfs_alloc_fix_freelist+0x3ed/0x480 [xfs]
  602. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa021b655>] xfs_alloc_ag_vextent+0xd5/0x100 [xfs]
  603. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa021c216>] xfs_alloc_vextent+0x476/0x610 [xfs]
  604. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0246ca3>] xfs_ialloc_ag_alloc+0x3a3/0x4b0 [xfs]
  605. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0246e8f>] xfs_dialloc+0xdf/0x850 [xfs]
  606. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81419fde>] ? release_sock+0xce/0xe0
  607. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81221eda>] ? symhash+0x1a/0x50
  608. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81221eb4>] ? symcmp+0x14/0x20
  609. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81221ace>] ? hashtab_search+0x7e/0x90
  610. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa024aed0>] xfs_ialloc+0x60/0x6f0 [xfs]
  611. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0263b14>] xfs_dir_ialloc+0x74/0x2b0 [xfs]
  612. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0265b60>] xfs_create+0x440/0x640 [xfs]
  613. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0272b3d>] xfs_vn_mknod+0xad/0x1c0 [xfs]
  614. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0272c80>] xfs_vn_create+0x10/0x20 [xfs]
  615. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81183b04>] vfs_create+0xb4/0xe0
  616. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03cf6c6>] nfsd_create_v3+0x366/0x4c0 [nfsd]
  617. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03d7413>] nfsd3_proc_create+0x123/0x1b0 [nfsd]
  618. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03c843e>] nfsd_dispatch+0xfe/0x240 [nfsd]
  619. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02fc794>] svc_process_common+0x344/0x640 [sunrpc]
  620. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8105fa40>] ? default_wake_function+0x0/0x20
  621. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02fcdd0>] svc_process+0x110/0x160 [sunrpc]
  622. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03c8b62>] nfsd+0xc2/0x160 [nfsd]
  623. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03c8aa0>] ? nfsd+0x0/0x160 [nfsd]
  624. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81090626>] kthread+0x96/0xa0
  625. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
  626. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81090590>] ? kthread+0x0/0xa0
  627. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
  628. Aug 23 11:10:35 row1r05s03 kernel: INFO: task nfsd:2691 blocked for more than 120 seconds.
  629. Aug 23 11:10:35 row1r05s03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  630. Aug 23 11:10:35 row1r05s03 kernel: nfsd D 0000000000000000 0 2691 2 0x00000080
  631. Aug 23 11:10:35 row1r05s03 kernel: ffff880216de55e0 0000000000000046 ffff880216de55c0 ffffffffa013aa51
  632. Aug 23 11:10:35 row1r05s03 kernel: 00000002171e94c2 0000000000000001 ffff8802288f8020 ffff880224440800
  633. Aug 23 11:10:35 row1r05s03 kernel: ffff8802277e7af8 ffff880216de5fd8 000000000000fb88 ffff8802277e7af8
  634. Aug 23 11:10:35 row1r05s03 kernel: Call Trace:
  635. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa013aa51>] ? bnx2_start_xmit+0x241/0x800 [bnx2]
  636. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff814eaac5>] schedule_timeout+0x215/0x2e0
  637. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8100b9ce>] ? common_interrupt+0xe/0x13
  638. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8100b9ce>] ? common_interrupt+0xe/0x13
  639. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff814eb9e2>] __down+0x72/0xb0
  640. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cae2>] ? _xfs_buf_find+0x102/0x280 [xfs]
  641. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff810967f1>] down+0x41/0x50
  642. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026c923>] xfs_buf_lock+0x53/0x110 [xfs]
  643. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cae2>] _xfs_buf_find+0x102/0x280 [xfs]
  644. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cccb>] xfs_buf_get+0x6b/0x1a0 [xfs]
  645. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026d33c>] xfs_buf_read+0x2c/0x100 [xfs]
  646. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0262f88>] xfs_trans_read_buf+0x1f8/0x400 [xfs]
  647. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0245774>] xfs_read_agi+0x74/0x100 [xfs]
  648. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa024582e>] xfs_ialloc_read_agi+0x2e/0x90 [xfs]
  649. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0245b13>] xfs_ialloc_ag_select+0x133/0x270 [xfs]
  650. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0247187>] xfs_dialloc+0x3d7/0x850 [xfs]
  651. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff814232b9>] ? skb_copy_datagram_iovec+0x1e9/0x2c0
  652. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8141a09c>] ? lock_sock_nested+0xac/0xc0
  653. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81419fde>] ? release_sock+0xce/0xe0
  654. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81221eda>] ? symhash+0x1a/0x50
  655. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81221eb4>] ? symcmp+0x14/0x20
  656. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81221ace>] ? hashtab_search+0x7e/0x90
  657. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa024aed0>] xfs_ialloc+0x60/0x6f0 [xfs]
  658. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0263b14>] xfs_dir_ialloc+0x74/0x2b0 [xfs]
  659. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0265b60>] xfs_create+0x440/0x640 [xfs]
  660. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0272b3d>] xfs_vn_mknod+0xad/0x1c0 [xfs]
  661. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0272c80>] xfs_vn_create+0x10/0x20 [xfs]
  662. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81183b04>] vfs_create+0xb4/0xe0
  663. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03cf6c6>] nfsd_create_v3+0x366/0x4c0 [nfsd]
  664. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03d7413>] nfsd3_proc_create+0x123/0x1b0 [nfsd]
  665. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03c843e>] nfsd_dispatch+0xfe/0x240 [nfsd]
  666. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02fc794>] svc_process_common+0x344/0x640 [sunrpc]
  667. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8105fa40>] ? default_wake_function+0x0/0x20
  668. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02fcdd0>] svc_process+0x110/0x160 [sunrpc]
  669. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03c8b62>] nfsd+0xc2/0x160 [nfsd]
  670. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03c8aa0>] ? nfsd+0x0/0x160 [nfsd]
  671. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81090626>] kthread+0x96/0xa0
  672. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
  673. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81090590>] ? kthread+0x0/0xa0
  674. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
  675. Aug 23 11:10:35 row1r05s03 kernel: INFO: task nfsd:2692 blocked for more than 120 seconds.
  676. Aug 23 11:10:35 row1r05s03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  677. Aug 23 11:10:35 row1r05s03 kernel: nfsd D 0000000000000002 0 2692 2 0x00000080
  678. Aug 23 11:10:35 row1r05s03 kernel: ffff880216eff670 0000000000000046 ffff880216eff620 ffffffff81429d7c
  679. Aug 23 11:10:35 row1r05s03 kernel: ffff880227253800 0000000400000000 ffff880227309458 ffffffffa013e0e0
  680. Aug 23 11:10:35 row1r05s03 kernel: ffff880227309058 ffff880216efffd8 000000000000fb88 ffff880227309058
  681. Aug 23 11:10:35 row1r05s03 kernel: Call Trace:
  682. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81429d7c>] ? dev_hard_start_xmit+0x2bc/0x3f0
  683. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81446ff8>] ? sch_direct_xmit+0x78/0x1c0
  684. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff814eaac5>] schedule_timeout+0x215/0x2e0
  685. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8127ddb0>] ? swiotlb_map_page+0x0/0x100
  686. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff814eb9e2>] __down+0x72/0xb0
  687. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cae2>] ? _xfs_buf_find+0x102/0x280 [xfs]
  688. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff810967f1>] down+0x41/0x50
  689. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026c923>] xfs_buf_lock+0x53/0x110 [xfs]
  690. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cae2>] _xfs_buf_find+0x102/0x280 [xfs]
  691. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cccb>] xfs_buf_get+0x6b/0x1a0 [xfs]
  692. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026d33c>] xfs_buf_read+0x2c/0x100 [xfs]
  693. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0262f88>] xfs_trans_read_buf+0x1f8/0x400 [xfs]
  694. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0219086>] xfs_read_agf+0x66/0x1a0 [xfs]
  695. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02191f9>] xfs_alloc_read_agf+0x39/0xd0 [xfs]
  696. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa021bc01>] xfs_alloc_fix_freelist+0x411/0x480 [xfs]
  697. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0260103>] ? xfs_trans_free_item_desc+0x33/0x40 [xfs]
  698. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0260247>] ? xfs_trans_free_items+0x87/0xb0 [xfs]
  699. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa025ccf8>] ? xfs_perag_get+0x38/0xb0 [xfs]
  700. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa021bd18>] xfs_free_extent+0xa8/0x130 [xfs]
  701. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02253bd>] xfs_bmap_finish+0x15d/0x1a0 [xfs]
  702. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa024aaa7>] xfs_itruncate_finish+0x167/0x330 [xfs]
  703. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0267243>] xfs_setattr+0x373/0xab0 [xfs]
  704. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8121649b>] ? dentry_has_perm+0x5b/0x80
  705. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02730eb>] xfs_vn_setattr+0x1b/0x20 [xfs]
  706. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81192398>] notify_change+0x168/0x340
  707. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03cdcdc>] nfsd_setattr+0x32c/0x410 [nfsd]
  708. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03d6be6>] nfsd3_proc_setattr+0x76/0xc0 [nfsd]
  709. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03c843e>] nfsd_dispatch+0xfe/0x240 [nfsd]
  710. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02fc794>] svc_process_common+0x344/0x640 [sunrpc]
  711. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8105fa40>] ? default_wake_function+0x0/0x20
  712. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02fcdd0>] svc_process+0x110/0x160 [sunrpc]
  713. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03c8b62>] nfsd+0xc2/0x160 [nfsd]
  714. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03c8aa0>] ? nfsd+0x0/0x160 [nfsd]
  715. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81090626>] kthread+0x96/0xa0
  716. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
  717. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81090590>] ? kthread+0x0/0xa0
  718. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
  719. Aug 23 11:10:35 row1r05s03 kernel: INFO: task flush-253:2:2945 blocked for more than 120 seconds.
  720. Aug 23 11:10:35 row1r05s03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  721. Aug 23 11:10:35 row1r05s03 kernel: flush-253:2 D 0000000000000002 0 2945 2 0x00000080
  722. Aug 23 11:10:35 row1r05s03 kernel: ffff8802166d9120 0000000000000046 0000000000000000 ffffffff8124cf3e
  723. Aug 23 11:10:35 row1r05s03 kernel: ffffea0006d8bd18 0000000000000286 ffff8802166d90b0 ffffffff810966e7
  724. Aug 23 11:10:35 row1r05s03 kernel: ffff880225681058 ffff8802166d9fd8 000000000000fb88 ffff880225681058
  725. Aug 23 11:10:35 row1r05s03 kernel: Call Trace:
  726. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8124cf3e>] ? generic_unplug_device+0x2e/0x50
  727. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff810966e7>] ? down_trylock+0x37/0x50
  728. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff814eaac5>] schedule_timeout+0x215/0x2e0
  729. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0260597>] ? xfs_trans_add_item+0x57/0x70 [xfs]
  730. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff814eb9e2>] __down+0x72/0xb0
  731. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cae2>] ? _xfs_buf_find+0x102/0x280 [xfs]
  732. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff810967f1>] down+0x41/0x50
  733. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026c923>] xfs_buf_lock+0x53/0x110 [xfs]
  734. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cae2>] _xfs_buf_find+0x102/0x280 [xfs]
  735. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cccb>] xfs_buf_get+0x6b/0x1a0 [xfs]
  736. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026d33c>] xfs_buf_read+0x2c/0x100 [xfs]
  737. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0262f88>] xfs_trans_read_buf+0x1f8/0x400 [xfs]
  738. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0219086>] xfs_read_agf+0x66/0x1a0 [xfs]
  739. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02191f9>] xfs_alloc_read_agf+0x39/0xd0 [xfs]
  740. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa021bc01>] xfs_alloc_fix_freelist+0x411/0x480 [xfs]
  741. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81278250>] ? sg_init_table+0x30/0x50
  742. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff812782ee>] ? __sg_alloc_table+0x7e/0x130
  743. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8135d7c0>] ? scsi_sg_alloc+0x0/0x60
  744. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa025ccf8>] ? xfs_perag_get+0x38/0xb0 [xfs]
  745. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa025ccf8>] ? xfs_perag_get+0x38/0xb0 [xfs]
  746. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa021bf2b>] xfs_alloc_vextent+0x18b/0x610 [xfs]
  747. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0226504>] xfs_bmap_btalloc+0x6c4/0x700 [xfs]
  748. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa003caf0>] ? megasas_fire_cmd_gen2+0x50/0x70 [megaraid_sas]
  749. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa022654e>] xfs_bmap_alloc+0xe/0x10 [xfs]
  750. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa022cd80>] xfs_bmapi+0x9a0/0x1090 [xfs]
  751. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa022434d>] ? xfs_bmap_search_multi_extents+0xad/0x120 [xfs]
  752. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0260597>] ? xfs_trans_add_item+0x57/0x70 [xfs]
  753. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa024ed0e>] xfs_iomap_write_allocate+0x17e/0x340 [xfs]
  754. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0269ad3>] xfs_map_blocks+0x193/0x250 [xfs]
  755. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026a5c9>] xfs_vm_writepage+0x1f9/0x510 [xfs]
  756. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81123f87>] __writepage+0x17/0x40
  757. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff811252a9>] write_cache_pages+0x1c9/0x4a0
  758. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81123f70>] ? __writepage+0x0/0x40
  759. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff811255a4>] generic_writepages+0x24/0x30
  760. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02698fe>] xfs_vm_writepages+0x5e/0x80 [xfs]
  761. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff811255d1>] do_writepages+0x21/0x40
  762. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8119fe8d>] writeback_single_inode+0xdd/0x290
  763. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff811a029e>] writeback_sb_inodes+0xce/0x180
  764. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff811a03fb>] writeback_inodes_wb+0xab/0x1b0
  765. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8105ab19>] ? find_busiest_queue+0x69/0x150
  766. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff811a079b>] wb_writeback+0x29b/0x3f0
  767. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff814e9c50>] ? thread_return+0x4e/0x76e
  768. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8107d572>] ? del_timer_sync+0x22/0x30
  769. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff811a0a89>] wb_do_writeback+0x199/0x240
  770. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff811a0b93>] bdi_writeback_task+0x63/0x1b0
  771. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81090857>] ? bit_waitqueue+0x17/0xd0
  772. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81134170>] ? bdi_start_fn+0x0/0x100
  773. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff811341f6>] bdi_start_fn+0x86/0x100
  774. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81134170>] ? bdi_start_fn+0x0/0x100
  775. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81090626>] kthread+0x96/0xa0
  776. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
  777. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81090590>] ? kthread+0x0/0xa0
  778. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
  779. Aug 23 11:11:21 row1r05s03 kernel: __ratelimit: 117 callbacks suppressed
  780. Aug 23 11:11:21 row1r05s03 kernel: swapper: page allocation failure. order:0, mode:0x20
  781. Aug 23 11:11:21 row1r05s03 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.19.1.el6.x86_64 #1
  782. Aug 23 11:11:21 row1r05s03 kernel: Call Trace:
  783. Aug 23 11:11:21 row1r05s03 kernel: <IRQ> [<ffffffff811231ff>] ? __alloc_pages_nodemask+0x77f/0x940
  784. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8115d1a2>] ? kmem_getpages+0x62/0x170
  785. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8115ddba>] ? fallback_alloc+0x1ba/0x270
  786. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  787. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8115db39>] ? ____cache_alloc_node+0x99/0x160
  788. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8115ed00>] ? kmem_cache_alloc_node_trace+0x90/0x200
  789. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8115ef1d>] ? __kmalloc_node+0x4d/0x60
  790. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8141ea1d>] ? __alloc_skb+0x6d/0x190
  791. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff814732b2>] ? tcp_collapse+0x1a2/0x3f0
  792. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff81132ad9>] ? zone_statistics+0x99/0xc0
  793. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff81474655>] ? tcp_try_rmem_schedule+0x245/0x360
  794. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8147491b>] ? tcp_data_queue+0x1ab/0xc70
  795. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8121bf23>] ? selinux_socket_sock_rcv_skb+0x83/0x220
  796. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff81477ab9>] ? tcp_rcv_established+0x369/0x800
  797. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8147fa73>] ? tcp_v4_do_rcv+0x2e3/0x430
  798. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff814812fe>] ? tcp_v4_rcv+0x4fe/0x8d0
  799. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8145f1ad>] ? ip_local_deliver_finish+0xdd/0x2d0
  800. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8145f438>] ? ip_local_deliver+0x98/0xa0
  801. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8145e8fd>] ? ip_rcv_finish+0x12d/0x440
  802. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8145ee85>] ? ip_rcv+0x275/0x350
  803. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8142946b>] ? __netif_receive_skb+0x49b/0x6f0
  804. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8142b5e8>] ? netif_receive_skb+0x58/0x60
  805. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8142b6f0>] ? napi_skb_finish+0x50/0x70
  806. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8142d6a9>] ? napi_gro_receive+0x39/0x50
  807. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffffa013bd5f>] ? bnx2_poll_work+0xd4f/0x1280 [bnx2]
  808. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8127ddb0>] ? swiotlb_map_page+0x0/0x100
  809. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffffa013c2cd>] ? bnx2_poll_msix+0x3d/0xc0 [bnx2]
  810. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff810949d2>] ? enqueue_hrtimer+0x82/0xd0
  811. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff813fb7e8>] ? dma_issue_pending_all+0x68/0xa0
  812. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8142d7c3>] ? net_rx_action+0x103/0x2f0
  813. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff81072991>] ? __do_softirq+0xc1/0x1e0
  814. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff810d8b40>] ? handle_IRQ_event+0x60/0x170
  815. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
  816. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
  817. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff81072775>] ? irq_exit+0x85/0x90
  818. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff814f1eb5>] ? do_IRQ+0x75/0xf0
  819. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
  820. Aug 23 11:11:21 row1r05s03 kernel: <EOI> [<ffffffff812c2d4e>] ? intel_idle+0xde/0x170
  821. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff812c2d31>] ? intel_idle+0xc1/0x170
  822. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff813f6c67>] ? cpuidle_idle_call+0xa7/0x140
  823. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
  824. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff814d109a>] ? rest_init+0x7a/0x80
  825. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
  826. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
  827. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
  828. Aug 23 11:12:16 row1r05s03 kernel: swapper: page allocation failure. order:0, mode:0x20
  829. Aug 23 11:12:16 row1r05s03 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.19.1.el6.x86_64 #1
  830. Aug 23 11:12:16 row1r05s03 kernel: Call Trace:
  831. Aug 23 11:12:16 row1r05s03 kernel: <IRQ> [<ffffffff811231ff>] ? __alloc_pages_nodemask+0x77f/0x940
  832. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115d1a2>] ? kmem_getpages+0x62/0x170
  833. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ddba>] ? fallback_alloc+0x1ba/0x270
  834. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  835. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115db39>] ? ____cache_alloc_node+0x99/0x160
  836. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ed00>] ? kmem_cache_alloc_node_trace+0x90/0x200
  837. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ef1d>] ? __kmalloc_node+0x4d/0x60
  838. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8141ea1d>] ? __alloc_skb+0x6d/0x190
  839. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8141eba4>] ? __netdev_alloc_skb+0x24/0x50
  840. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffffa013b86c>] ? bnx2_poll_work+0x85c/0x1280 [bnx2]
  841. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8127ddb0>] ? swiotlb_map_page+0x0/0x100
  842. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffffa013c2cd>] ? bnx2_poll_msix+0x3d/0xc0 [bnx2]
  843. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff813fb7e8>] ? dma_issue_pending_all+0x68/0xa0
  844. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8142d7c3>] ? net_rx_action+0x103/0x2f0
  845. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81094cd3>] ? hrtimer_get_next_event+0xc3/0x100
  846. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81072991>] ? __do_softirq+0xc1/0x1e0
  847. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff810d8b40>] ? handle_IRQ_event+0x60/0x170
  848. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
  849. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
  850. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81072775>] ? irq_exit+0x85/0x90
  851. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff814f1eb5>] ? do_IRQ+0x75/0xf0
  852. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
  853. Aug 23 11:12:16 row1r05s03 kernel: <EOI> [<ffffffff812c2d4e>] ? intel_idle+0xde/0x170
  854. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff812c2d31>] ? intel_idle+0xc1/0x170
  855. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff810979ed>] ? sched_clock_cpu+0xcd/0x110
  856. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff813f6c67>] ? cpuidle_idle_call+0xa7/0x140
  857. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
  858. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff814d109a>] ? rest_init+0x7a/0x80
  859. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
  860. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
  861. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
  862. Aug 23 11:12:16 row1r05s03 kernel: swapper: page allocation failure. order:0, mode:0x20
  863. Aug 23 11:12:16 row1r05s03 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.19.1.el6.x86_64 #1
  864. Aug 23 11:12:16 row1r05s03 kernel: Call Trace:
  865. Aug 23 11:12:16 row1r05s03 kernel: <IRQ> [<ffffffff811231ff>] ? __alloc_pages_nodemask+0x77f/0x940
  866. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115d1a2>] ? kmem_getpages+0x62/0x170
  867. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ddba>] ? fallback_alloc+0x1ba/0x270
  868. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  869. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115db39>] ? ____cache_alloc_node+0x99/0x160
  870. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ed00>] ? kmem_cache_alloc_node_trace+0x90/0x200
  871. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ef1d>] ? __kmalloc_node+0x4d/0x60
  872. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8141ea1d>] ? __alloc_skb+0x6d/0x190
  873. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8141eba4>] ? __netdev_alloc_skb+0x24/0x50
  874. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffffa013b86c>] ? bnx2_poll_work+0x85c/0x1280 [bnx2]
  875. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8127ddb0>] ? swiotlb_map_page+0x0/0x100
  876. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffffa013c2cd>] ? bnx2_poll_msix+0x3d/0xc0 [bnx2]
  877. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff813fb7e8>] ? dma_issue_pending_all+0x68/0xa0
  878. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8142d7c3>] ? net_rx_action+0x103/0x2f0
  879. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81094cd3>] ? hrtimer_get_next_event+0xc3/0x100
  880. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81072991>] ? __do_softirq+0xc1/0x1e0
  881. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff810d8b40>] ? handle_IRQ_event+0x60/0x170
  882. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
  883. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
  884. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81072775>] ? irq_exit+0x85/0x90
  885. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff814f1eb5>] ? do_IRQ+0x75/0xf0
  886. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
  887. Aug 23 11:12:16 row1r05s03 kernel: <EOI> [<ffffffff812c2d4e>] ? intel_idle+0xde/0x170
  888. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff812c2d31>] ? intel_idle+0xc1/0x170
  889. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff810979ed>] ? sched_clock_cpu+0xcd/0x110
  890. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff813f6c67>] ? cpuidle_idle_call+0xa7/0x140
  891. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
  892. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff814d109a>] ? rest_init+0x7a/0x80
  893. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
  894. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
  895. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
  896. Aug 23 11:12:16 row1r05s03 kernel: swapper: page allocation failure. order:0, mode:0x20
  897. Aug 23 11:12:16 row1r05s03 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.19.1.el6.x86_64 #1
  898. Aug 23 11:12:16 row1r05s03 kernel: Call Trace:
  899. Aug 23 11:12:16 row1r05s03 kernel: <IRQ> [<ffffffff811231ff>] ? __alloc_pages_nodemask+0x77f/0x940
  900. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115d1a2>] ? kmem_getpages+0x62/0x170
  901. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ddba>] ? fallback_alloc+0x1ba/0x270
  902. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  903. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115db39>] ? ____cache_alloc_node+0x99/0x160
  904. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ed00>] ? kmem_cache_alloc_node_trace+0x90/0x200
  905. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ef1d>] ? __kmalloc_node+0x4d/0x60
  906. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8141ea1d>] ? __alloc_skb+0x6d/0x190
  907. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8141eba4>] ? __netdev_alloc_skb+0x24/0x50
  908. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffffa013b86c>] ? bnx2_poll_work+0x85c/0x1280 [bnx2]
  909. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8127ddb0>] ? swiotlb_map_page+0x0/0x100
  910. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffffa013c2cd>] ? bnx2_poll_msix+0x3d/0xc0 [bnx2]
  911. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff813fb7e8>] ? dma_issue_pending_all+0x68/0xa0
  912. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8142d7c3>] ? net_rx_action+0x103/0x2f0
  913. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81094cd3>] ? hrtimer_get_next_event+0xc3/0x100
  914. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81072991>] ? __do_softirq+0xc1/0x1e0
  915. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff810d8b40>] ? handle_IRQ_event+0x60/0x170
  916. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
  917. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
  918. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81072775>] ? irq_exit+0x85/0x90
  919. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff814f1eb5>] ? do_IRQ+0x75/0xf0
  920. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
  921. Aug 23 11:12:16 row1r05s03 kernel: <EOI> [<ffffffff812c2d4e>] ? intel_idle+0xde/0x170
  922. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff812c2d31>] ? intel_idle+0xc1/0x170
  923. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff810979ed>] ? sched_clock_cpu+0xcd/0x110
  924. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff813f6c67>] ? cpuidle_idle_call+0xa7/0x140
  925. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
  926. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff814d109a>] ? rest_init+0x7a/0x80
  927. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
  928. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
  929. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
  930. Aug 23 11:12:16 row1r05s03 kernel: swapper: page allocation failure. order:0, mode:0x20
  931. Aug 23 11:12:16 row1r05s03 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.19.1.el6.x86_64 #1
  932. Aug 23 11:12:16 row1r05s03 kernel: Call Trace:
  933. Aug 23 11:12:16 row1r05s03 kernel: <IRQ> [<ffffffff811231ff>] ? __alloc_pages_nodemask+0x77f/0x940
  934. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115d1a2>] ? kmem_getpages+0x62/0x170
  935. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ddba>] ? fallback_alloc+0x1ba/0x270
  936. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  937. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115db39>] ? ____cache_alloc_node+0x99/0x160
  938. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ed00>] ? kmem_cache_alloc_node_trace+0x90/0x200
  939. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ef1d>] ? __kmalloc_node+0x4d/0x60
  940. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8141ea1d>] ? __alloc_skb+0x6d/0x190
  941. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8141eba4>] ? __netdev_alloc_skb+0x24/0x50
  942. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffffa013b86c>] ? bnx2_poll_work+0x85c/0x1280 [bnx2]
  943. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8127ddb0>] ? swiotlb_map_page+0x0/0x100
  944. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffffa013c2cd>] ? bnx2_poll_msix+0x3d/0xc0 [bnx2]
  945. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff813fb7e8>] ? dma_issue_pending_all+0x68/0xa0
  946. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8142d7c3>] ? net_rx_action+0x103/0x2f0
  947. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81094cd3>] ? hrtimer_get_next_event+0xc3/0x100
  948. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81072991>] ? __do_softirq+0xc1/0x1e0
  949. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff810d8b40>] ? handle_IRQ_event+0x60/0x170
  950. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
  951. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
  952. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81072775>] ? irq_exit+0x85/0x90
  953. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff814f1eb5>] ? do_IRQ+0x75/0xf0
  954. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
  955. Aug 23 11:12:16 row1r05s03 kernel: <EOI> [<ffffffff812c2d4e>] ? intel_idle+0xde/0x170
  956. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff812c2d31>] ? intel_idle+0xc1/0x170
  957. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff810979ed>] ? sched_clock_cpu+0xcd/0x110
  958. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff813f6c67>] ? cpuidle_idle_call+0xa7/0x140
  959. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
  960. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff814d109a>] ? rest_init+0x7a/0x80
  961. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
  962. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
  963. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
  964. Aug 23 11:12:16 row1r05s03 kernel: swapper: page allocation failure. order:0, mode:0x20
  965. Aug 23 11:12:16 row1r05s03 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.19.1.el6.x86_64 #1
  966. Aug 23 11:12:16 row1r05s03 kernel: Call Trace:
  967. Aug 23 11:12:16 row1r05s03 kernel: <IRQ> [<ffffffff811231ff>] ? __alloc_pages_nodemask+0x77f/0x940
  968. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115d1a2>] ? kmem_getpages+0x62/0x170
  969. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ddba>] ? fallback_alloc+0x1ba/0x270
  970. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  971. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115db39>] ? ____cache_alloc_node+0x99/0x160
  972. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ed00>] ? kmem_cache_alloc_node_trace+0x90/0x200
  973. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ef1d>] ? __kmalloc_node+0x4d/0x60
  974. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8141ea1d>] ? __alloc_skb+0x6d/0x190
  975. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8141eba4>] ? __netdev_alloc_skb+0x24/0x50
  976. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffffa013b86c>] ? bnx2_poll_work+0x85c/0x1280 [bnx2]
  977. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8127ddb0>] ? swiotlb_map_page+0x0/0x100
  978. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffffa013c2cd>] ? bnx2_poll_msix+0x3d/0xc0 [bnx2]
  979. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff813fb7e8>] ? dma_issue_pending_all+0x68/0xa0
  980. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8142d7c3>] ? net_rx_action+0x103/0x2f0
  981. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81094cd3>] ? hrtimer_get_next_event+0xc3/0x100
  982. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81072991>] ? __do_softirq+0xc1/0x1e0
  983. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff810d8b40>] ? handle_IRQ_event+0x60/0x170
  984. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
  985. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
  986. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81072775>] ? irq_exit+0x85/0x90
  987. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff814f1eb5>] ? do_IRQ+0x75/0xf0
  988. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
  989. Aug 23 11:12:16 row1r05s03 kernel: <EOI> [<ffffffff812c2d4e>] ? intel_idle+0xde/0x170
  990. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff812c2d31>] ? intel_idle+0xc1/0x170
  991. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff810979ed>] ? sched_clock_cpu+0xcd/0x110
  992. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff813f6c67>] ? cpuidle_idle_call+0xa7/0x140
  993. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
  994. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff814d109a>] ? rest_init+0x7a/0x80
  995. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
  996. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
  997. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
  998. Aug 23 11:12:16 row1r05s03 kernel: swapper: page allocation failure. order:0, mode:0x20
  999. Aug 23 11:12:16 row1r05s03 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.19.1.el6.x86_64 #1
  1000. Aug 23 11:12:16 row1r05s03 kernel: Call Trace:
  1001. Aug 23 11:12:16 row1r05s03 kernel: <IRQ> [<ffffffff811231ff>] ? __alloc_pages_nodemask+0x77f/0x940
  1002. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115d1a2>] ? kmem_getpages+0x62/0x170
  1003. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ddba>] ? fallback_alloc+0x1ba/0x270
  1004. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  1005. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115db39>] ? ____cache_alloc_node+0x99/0x160
  1006. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ed00>] ? kmem_cache_alloc_node_trace+0x90/0x200
  1007. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ef1d>] ? __kmalloc_node+0x4d/0x60
  1008. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8141ea1d>] ? __alloc_skb+0x6d/0x190
  1009. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8141eba4>] ? __netdev_alloc_skb+0x24/0x50
  1010. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffffa013b86c>] ? bnx2_poll_work+0x85c/0x1280 [bnx2]
  1011. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8127ddb0>] ? swiotlb_map_page+0x0/0x100
  1012. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffffa013c2cd>] ? bnx2_poll_msix+0x3d/0xc0 [bnx2]
  1013. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff813fb7e8>] ? dma_issue_pending_all+0x68/0xa0
  1014. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8142d7c3>] ? net_rx_action+0x103/0x2f0
  1015. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81094cd3>] ? hrtimer_get_next_event+0xc3/0x100
  1016. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81072991>] ? __do_softirq+0xc1/0x1e0
  1017. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff810d8b40>] ? handle_IRQ_event+0x60/0x170
  1018. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
  1019. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
  1020. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81072775>] ? irq_exit+0x85/0x90
  1021. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff814f1eb5>] ? do_IRQ+0x75/0xf0
  1022. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
  1023. Aug 23 11:12:16 row1r05s03 kernel: <EOI> [<ffffffff812c2d4e>] ? intel_idle+0xde/0x170
  1024. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff812c2d31>] ? intel_idle+0xc1/0x170
  1025. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff810979ed>] ? sched_clock_cpu+0xcd/0x110
  1026. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff813f6c67>] ? cpuidle_idle_call+0xa7/0x140
  1027. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
  1028. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff814d109a>] ? rest_init+0x7a/0x80
  1029. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
  1030. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
  1031. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
  1032. Aug 23 11:12:16 row1r05s03 kernel: swapper: page allocation failure. order:0, mode:0x20
  1033. Aug 23 11:12:16 row1r05s03 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.19.1.el6.x86_64 #1
  1034. Aug 23 11:12:16 row1r05s03 kernel: Call Trace:
  1035. Aug 23 11:12:16 row1r05s03 kernel: <IRQ> [<ffffffff811231ff>] ? __alloc_pages_nodemask+0x77f/0x940
  1036. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115d1a2>] ? kmem_getpages+0x62/0x170
  1037. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ddba>] ? fallback_alloc+0x1ba/0x270
  1038. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  1039. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115db39>] ? ____cache_alloc_node+0x99/0x160
  1040. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ed00>] ? kmem_cache_alloc_node_trace+0x90/0x200
  1041. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ef1d>] ? __kmalloc_node+0x4d/0x60
  1042. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8141ea1d>] ? __alloc_skb+0x6d/0x190
  1043. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8141eba4>] ? __netdev_alloc_skb+0x24/0x50
  1044. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffffa013b86c>] ? bnx2_poll_work+0x85c/0x1280 [bnx2]
  1045. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8127ddb0>] ? swiotlb_map_page+0x0/0x100
  1046. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffffa013c2cd>] ? bnx2_poll_msix+0x3d/0xc0 [bnx2]
  1047. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff813fb7e8>] ? dma_issue_pending_all+0x68/0xa0
  1048. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8142d7c3>] ? net_rx_action+0x103/0x2f0
  1049. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81094cd3>] ? hrtimer_get_next_event+0xc3/0x100
  1050. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81072991>] ? __do_softirq+0xc1/0x1e0
  1051. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff810d8b40>] ? handle_IRQ_event+0x60/0x170
  1052. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
  1053. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
  1054. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81072775>] ? irq_exit+0x85/0x90
  1055. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff814f1eb5>] ? do_IRQ+0x75/0xf0
  1056. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
  1057. Aug 23 11:12:16 row1r05s03 kernel: <EOI> [<ffffffff812c2d4e>] ? intel_idle+0xde/0x170
  1058. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff812c2d31>] ? intel_idle+0xc1/0x170
  1059. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff810979ed>] ? sched_clock_cpu+0xcd/0x110
  1060. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff813f6c67>] ? cpuidle_idle_call+0xa7/0x140
  1061. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
  1062. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff814d109a>] ? rest_init+0x7a/0x80
  1063. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
  1064. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
  1065. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
  1066. Aug 23 11:12:16 row1r05s03 kernel: swapper: page allocation failure. order:0, mode:0x20
  1067. Aug 23 11:12:16 row1r05s03 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.19.1.el6.x86_64 #1
  1068. Aug 23 11:12:16 row1r05s03 kernel: Call Trace:
  1069. Aug 23 11:12:16 row1r05s03 kernel: <IRQ> [<ffffffff811231ff>] ? __alloc_pages_nodemask+0x77f/0x940
  1070. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115d1a2>] ? kmem_getpages+0x62/0x170
  1071. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ddba>] ? fallback_alloc+0x1ba/0x270
  1072. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  1073. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115db39>] ? ____cache_alloc_node+0x99/0x160
  1074. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ed00>] ? kmem_cache_alloc_node_trace+0x90/0x200
  1075. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ef1d>] ? __kmalloc_node+0x4d/0x60
  1076. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8141ea1d>] ? __alloc_skb+0x6d/0x190
  1077. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8141eba4>] ? __netdev_alloc_skb+0x24/0x50
  1078. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffffa013b86c>] ? bnx2_poll_work+0x85c/0x1280 [bnx2]
  1079. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8127ddb0>] ? swiotlb_map_page+0x0/0x100
  1080. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffffa013c2cd>] ? bnx2_poll_msix+0x3d/0xc0 [bnx2]
  1081. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff813fb7e8>] ? dma_issue_pending_all+0x68/0xa0
  1082. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8142d7c3>] ? net_rx_action+0x103/0x2f0
  1083. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81094cd3>] ? hrtimer_get_next_event+0xc3/0x100
  1084. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81072991>] ? __do_softirq+0xc1/0x1e0
  1085. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff810d8b40>] ? handle_IRQ_event+0x60/0x170
  1086. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
  1087. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
  1088. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81072775>] ? irq_exit+0x85/0x90
  1089. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff814f1eb5>] ? do_IRQ+0x75/0xf0
  1090. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
  1091. Aug 23 11:12:16 row1r05s03 kernel: <EOI> [<ffffffff812c2d4e>] ? intel_idle+0xde/0x170
  1092. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff812c2d31>] ? intel_idle+0xc1/0x170
  1093. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff810979ed>] ? sched_clock_cpu+0xcd/0x110
  1094. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff813f6c67>] ? cpuidle_idle_call+0xa7/0x140
  1095. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
  1096. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff814d109a>] ? rest_init+0x7a/0x80
  1097. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
  1098. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
  1099. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
  1100. Aug 23 11:12:16 row1r05s03 kernel: swapper: page allocation failure. order:0, mode:0x20
  1101. Aug 23 11:12:16 row1r05s03 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.19.1.el6.x86_64 #1
  1102. Aug 23 11:12:16 row1r05s03 kernel: Call Trace:
  1103. Aug 23 11:12:16 row1r05s03 kernel: <IRQ> [<ffffffff811231ff>] ? __alloc_pages_nodemask+0x77f/0x940
  1104. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115d1a2>] ? kmem_getpages+0x62/0x170
  1105. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ddba>] ? fallback_alloc+0x1ba/0x270
  1106. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  1107. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115db39>] ? ____cache_alloc_node+0x99/0x160
  1108. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ed00>] ? kmem_cache_alloc_node_trace+0x90/0x200
  1109. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ef1d>] ? __kmalloc_node+0x4d/0x60
  1110. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8141ea1d>] ? __alloc_skb+0x6d/0x190
  1111. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8141eba4>] ? __netdev_alloc_skb+0x24/0x50
  1112. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffffa013b86c>] ? bnx2_poll_work+0x85c/0x1280 [bnx2]
  1113. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8127ddb0>] ? swiotlb_map_page+0x0/0x100
  1114. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffffa013c2cd>] ? bnx2_poll_msix+0x3d/0xc0 [bnx2]
  1115. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff813fb7e8>] ? dma_issue_pending_all+0x68/0xa0
  1116. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8142d7c3>] ? net_rx_action+0x103/0x2f0
  1117. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81094cd3>] ? hrtimer_get_next_event+0xc3/0x100
  1118. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81072991>] ? __do_softirq+0xc1/0x1e0
  1119. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff810d8b40>] ? handle_IRQ_event+0x60/0x170
  1120. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
  1121. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
  1122. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81072775>] ? irq_exit+0x85/0x90
  1123. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff814f1eb5>] ? do_IRQ+0x75/0xf0
  1124. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
  1125. Aug 23 11:12:16 row1r05s03 kernel: <EOI> [<ffffffff812c2d4e>] ? intel_idle+0xde/0x170
  1126. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff812c2d31>] ? intel_idle+0xc1/0x170
  1127. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff810979ed>] ? sched_clock_cpu+0xcd/0x110
  1128. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff813f6c67>] ? cpuidle_idle_call+0xa7/0x140
  1129. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
  1130. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff814d109a>] ? rest_init+0x7a/0x80
  1131. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
  1132. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
  1133. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
  1134. Aug 23 11:12:35 row1r05s03 kernel: INFO: task kswapd0:98 blocked for more than 120 seconds.
  1135. Aug 23 11:12:35 row1r05s03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  1136. Aug 23 11:12:35 row1r05s03 kernel: kswapd0 D 0000000000000005 0 98 2 0x00000000
  1137. Aug 23 11:12:35 row1r05s03 kernel: ffff8802262ad530 0000000000000046 ffff8802262ad4f8 ffff8802262ad4f4
  1138. Aug 23 11:12:35 row1r05s03 kernel: ffff8802262ad4d0 ffff88022fc28e00 ffff88002f816680 0000000000000400
  1139. Aug 23 11:12:35 row1r05s03 kernel: ffff8802262abaf8 ffff8802262adfd8 000000000000fb88 ffff8802262abaf8
  1140. Aug 23 11:12:35 row1r05s03 kernel: Call Trace:
  1141. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff814eaac5>] schedule_timeout+0x215/0x2e0
  1142. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff810966e7>] ? down_trylock+0x37/0x50
  1143. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa026b67f>] ? xfs_buf_cond_lock+0x2f/0xc0 [xfs]
  1144. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff814eb9e2>] __down+0x72/0xb0
  1145. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa026cae2>] ? _xfs_buf_find+0x102/0x280 [xfs]
  1146. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff810967f1>] down+0x41/0x50
  1147. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa026c923>] xfs_buf_lock+0x53/0x110 [xfs]
  1148. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa026cae2>] _xfs_buf_find+0x102/0x280 [xfs]
  1149. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa026cccb>] xfs_buf_get+0x6b/0x1a0 [xfs]
  1150. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa026d33c>] xfs_buf_read+0x2c/0x100 [xfs]
  1151. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa0262f88>] xfs_trans_read_buf+0x1f8/0x400 [xfs]
  1152. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa0219086>] xfs_read_agf+0x66/0x1a0 [xfs]
  1153. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa02191f9>] xfs_alloc_read_agf+0x39/0xd0 [xfs]
  1154. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa021bc01>] xfs_alloc_fix_freelist+0x411/0x480 [xfs]
  1155. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa0260103>] ? xfs_trans_free_item_desc+0x33/0x40 [xfs]
  1156. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa0260247>] ? xfs_trans_free_items+0x87/0xb0 [xfs]
  1157. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa025ccf8>] ? xfs_perag_get+0x38/0xb0 [xfs]
  1158. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa021bd18>] xfs_free_extent+0xa8/0x130 [xfs]
  1159. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa02253bd>] xfs_bmap_finish+0x15d/0x1a0 [xfs]
  1160. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa024aaa7>] xfs_itruncate_finish+0x167/0x330 [xfs]
  1161. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa0266028>] xfs_free_eofblocks+0x1b8/0x280 [xfs]
  1162. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa02665c0>] xfs_inactive+0xc0/0x460 [xfs]
  1163. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa0276a94>] ? xfs_inode_set_reclaim_tag+0x84/0xa0 [xfs]
  1164. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa0273d80>] xfs_fs_clear_inode+0xa0/0xd0 [xfs]
  1165. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff81190c3c>] clear_inode+0xac/0x140
  1166. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff81190d10>] dispose_list+0x40/0x120
  1167. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff81191064>] shrink_icache_memory+0x274/0x2e0
  1168. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff81128c7a>] shrink_slab+0x12a/0x1a0
  1169. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff8112bb4d>] balance_pgdat+0x57d/0x7e0
  1170. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff8112c160>] ? isolate_pages_global+0x0/0x350
  1171. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff8112035d>] ? zone_watermark_ok_safe+0xad/0xc0
  1172. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff8112bee6>] kswapd+0x136/0x3b0
  1173. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff81090990>] ? autoremove_wake_function+0x0/0x40
  1174. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff8112bdb0>] ? kswapd+0x0/0x3b0
  1175. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff81090626>] kthread+0x96/0xa0
  1176. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
  1177. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff81090590>] ? kthread+0x0/0xa0
  1178. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
  1179. Aug 23 11:12:35 row1r05s03 kernel: INFO: task nfsd:2687 blocked for more than 120 seconds.
  1180. Aug 23 11:12:35 row1r05s03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  1181. Aug 23 11:12:35 row1r05s03 kernel: nfsd D 0000000000000005 0 2687 2 0x00000080
  1182. Aug 23 11:12:35 row1r05s03 kernel: ffff8802171d6960 0000000000000046 0000000000000000 0000000000000000
  1183. Aug 23 11:12:35 row1r05s03 kernel: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
  1184. Aug 23 11:12:35 row1r05s03 kernel: ffff8802277e7098 ffff8802171d7fd8 000000000000fb88 ffff8802277e7098
  1185. Aug 23 11:12:35 row1r05s03 kernel: Call Trace:
  1186. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff81132ad9>] ? zone_statistics+0x99/0xc0
  1187. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff814eaac5>] schedule_timeout+0x215/0x2e0
  1188. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff810966e7>] ? down_trylock+0x37/0x50
  1189. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa026b67f>] ? xfs_buf_cond_lock+0x2f/0xc0 [xfs]
  1190. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff814eb9e2>] __down+0x72/0xb0
  1191. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa026cae2>] ? _xfs_buf_find+0x102/0x280 [xfs]
  1192. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff810967f1>] down+0x41/0x50
  1193. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa026c923>] xfs_buf_lock+0x53/0x110 [xfs]
  1194. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa026cae2>] _xfs_buf_find+0x102/0x280 [xfs]
  1195. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa026cccb>] xfs_buf_get+0x6b/0x1a0 [xfs]
  1196. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa026d33c>] xfs_buf_read+0x2c/0x100 [xfs]
  1197. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa0262f88>] xfs_trans_read_buf+0x1f8/0x400 [xfs]
  1198. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa0219086>] xfs_read_agf+0x66/0x1a0 [xfs]
  1199. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa02191f9>] xfs_alloc_read_agf+0x39/0xd0 [xfs]
  1200. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa021bc01>] xfs_alloc_fix_freelist+0x411/0x480 [xfs]
  1201. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff8115d9bb>] ? cache_alloc_refill+0x15b/0x240
  1202. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa025ccf8>] ? xfs_perag_get+0x38/0xb0 [xfs]
  1203. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa021bd18>] xfs_free_extent+0xa8/0x130 [xfs]
  1204. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa02253bd>] xfs_bmap_finish+0x15d/0x1a0 [xfs]
  1205. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa024aaa7>] xfs_itruncate_finish+0x167/0x330 [xfs]
  1206. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa0266028>] xfs_free_eofblocks+0x1b8/0x280 [xfs]
  1207. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa02665c0>] xfs_inactive+0xc0/0x460 [xfs]
  1208. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa0276a94>] ? xfs_inode_set_reclaim_tag+0x84/0xa0 [xfs]
  1209. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa0273d80>] xfs_fs_clear_inode+0xa0/0xd0 [xfs]
  1210. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff81190c3c>] clear_inode+0xac/0x140
  1211. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff81190d10>] dispose_list+0x40/0x120
  1212. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff81191064>] shrink_icache_memory+0x274/0x2e0
  1213. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff81128c7a>] shrink_slab+0x12a/0x1a0
  1214. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff8112abff>] do_try_to_free_pages+0x2ff/0x520
  1215. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff81113e8f>] ? zone_watermark_ok+0x1f/0x30
  1216. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff8112b00d>] try_to_free_pages+0x9d/0x130
  1217. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff81131976>] ? next_online_pgdat+0x26/0x50
  1218. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff8112c160>] ? isolate_pages_global+0x0/0x350
  1219. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff81122e8d>] __alloc_pages_nodemask+0x40d/0x940
  1220. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff8115d1a2>] kmem_getpages+0x62/0x170
  1221. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff8115ddba>] fallback_alloc+0x1ba/0x270
  1222. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  1223. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff8115db39>] ____cache_alloc_node+0x99/0x160
  1224. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff8115ed00>] kmem_cache_alloc_node_trace+0x90/0x200
  1225. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff8114967f>] vm_map_ram+0x1cf/0x470
  1226. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa026b67f>] ? xfs_buf_cond_lock+0x2f/0xc0 [xfs]
  1227. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa026bbb1>] _xfs_buf_map_pages+0xc1/0x120 [xfs]
  1228. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa026cd63>] xfs_buf_get+0x103/0x1a0 [xfs]
  1229. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa026d33c>] xfs_buf_read+0x2c/0x100 [xfs]
  1230. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa0262f88>] xfs_trans_read_buf+0x1f8/0x400 [xfs]
  1231. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa023048e>] xfs_btree_read_buf_block+0x5e/0xd0 [xfs]
  1232. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa022e8cc>] ? xfs_btree_ptr_offset+0x4c/0x90 [xfs]
  1233. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa02310c2>] xfs_btree_increment+0x202/0x2a0 [xfs]
  1234. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa021ac91>] xfs_alloc_ag_vextent_near+0x231/0xb20 [xfs]
  1235. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa021bbdd>] ? xfs_alloc_fix_freelist+0x3ed/0x480 [xfs]
  1236. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa021b655>] xfs_alloc_ag_vextent+0xd5/0x100 [xfs]
  1237. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa021c216>] xfs_alloc_vextent+0x476/0x610 [xfs]
  1238. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa0246ca3>] xfs_ialloc_ag_alloc+0x3a3/0x4b0 [xfs]
  1239. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa0246e8f>] xfs_dialloc+0xdf/0x850 [xfs]
  1240. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff8109672f>] ? up+0x2f/0x50
  1241. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa026355b>] ? xfs_trans_brelse+0xdb/0x150 [xfs]
  1242. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff81221eda>] ? symhash+0x1a/0x50
  1243. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff81221eb4>] ? symcmp+0x14/0x20
  1244. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff81221ace>] ? hashtab_search+0x7e/0x90
  1245. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa024aed0>] xfs_ialloc+0x60/0x6f0 [xfs]
  1246. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa0263b14>] xfs_dir_ialloc+0x74/0x2b0 [xfs]
  1247. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa0265b60>] xfs_create+0x440/0x640 [xfs]
  1248. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa0272b3d>] xfs_vn_mknod+0xad/0x1c0 [xfs]
  1249. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa0272c80>] xfs_vn_create+0x10/0x20 [xfs]
  1250. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff81183b04>] vfs_create+0xb4/0xe0
  1251. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa03cf6c6>] nfsd_create_v3+0x366/0x4c0 [nfsd]
  1252. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa03d7413>] nfsd3_proc_create+0x123/0x1b0 [nfsd]
  1253. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa03c843e>] nfsd_dispatch+0xfe/0x240 [nfsd]
  1254. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa02fc794>] svc_process_common+0x344/0x640 [sunrpc]
  1255. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff8105fa40>] ? default_wake_function+0x0/0x20
  1256. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa02fcdd0>] svc_process+0x110/0x160 [sunrpc]
  1257. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa03c8b62>] nfsd+0xc2/0x160 [nfsd]
  1258. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa03c8aa0>] ? nfsd+0x0/0x160 [nfsd]
  1259. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff81090626>] kthread+0x96/0xa0
  1260. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
  1261. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff81090590>] ? kthread+0x0/0xa0
  1262. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
  1263. Aug 23 11:14:50 row1r05s03 rpc.mountd[2631]: authenticated unmount request from row1r07s32:949 for /exports/NEWTON1 (/exports)
  1264. Aug 23 11:14:50 row1r05s03 rpc.mountd[2631]: authenticated unmount request from row1r07s32:970 for /exports/IMAGE_MAIN (/exports)
  1265. Aug 23 11:17:02 row1r05s03 rpc.mountd[2631]: authenticated unmount request from sdats1:648 for /exports/NEWTON1 (/exports)
  1266. Aug 23 11:17:02 row1r05s03 rpc.mountd[2631]: authenticated unmount request from sdats1:651 for /exports/IMAGE_MAIN (/exports)
  1267. Aug 23 11:19:26 row1r05s03 rpc.mountd[2631]: authenticated unmount request from r1s18c12:886 for /exports/NEWTON1 (/exports)
  1268. Aug 23 11:22:07 row1r05s03 ntpd[2725]: synchronized to 142.58.190.1, stratum 2
  1269. Aug 23 11:24:26 row1r05s03 kernel: Slow work thread pool: Starting up
  1270. Aug 23 11:24:26 row1r05s03 kernel: Slow work thread pool: Ready
  1271. Aug 23 11:24:26 row1r05s03 kernel: FS-Cache: Loaded
  1272. Aug 23 11:24:26 row1r05s03 kernel: Registering the id_resolver key type
  1273. Aug 23 11:24:26 row1r05s03 kernel: FS-Cache: Netfs 'nfs' registered for caching
  1274. Aug 23 11:26:32 row1r05s03 xinetd[2717]: START: shell pid=7125 from=199.60.1.59
  1275. Aug 23 11:27:38 row1r05s03 ntpd[2725]: synchronized to 142.58.103.1, stratum 2
  1276. Aug 23 11:30:14 row1r05s03 init: tty (/dev/tty2) main process (3637) killed by TERM signal
  1277. Aug 23 11:30:14 row1r05s03 init: tty (/dev/tty3) main process (3639) killed by TERM signal
  1278. Aug 23 11:30:14 row1r05s03 init: tty (/dev/tty4) main process (3641) killed by TERM signal
  1279. Aug 23 11:30:14 row1r05s03 init: tty (/dev/tty5) main process (3643) killed by TERM signal
  1280. Aug 23 11:30:14 row1r05s03 init: tty (/dev/tty6) main process (3645) killed by TERM signal
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!