Advertisement
Guest User

system hang on busy NFS server

a guest
Aug 23rd, 2013
165
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 108.56 KB | None | 0 0
  1. Aug 23 11:10:20 row1r05s03 kernel: swapper: page allocation failure. order:0, mode:0x20
  2. Aug 23 11:10:20 row1r05s03 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.19.1.el6.x86_64 #1
  3. Aug 23 11:10:20 row1r05s03 kernel: Call Trace:
  4. Aug 23 11:10:20 row1r05s03 kernel: <IRQ> [<ffffffff811231ff>] ? __alloc_pages_nodemask+0x77f/0x940
  5. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115d1a2>] ? kmem_getpages+0x62/0x170
  6. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ddba>] ? fallback_alloc+0x1ba/0x270
  7. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  8. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115db39>] ? ____cache_alloc_node+0x99/0x160
  9. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ed00>] ? kmem_cache_alloc_node_trace+0x90/0x200
  10. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ef1d>] ? __kmalloc_node+0x4d/0x60
  11. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8141ea1d>] ? __alloc_skb+0x6d/0x190
  12. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8141eba4>] ? __netdev_alloc_skb+0x24/0x50
  13. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffffa013b86c>] ? bnx2_poll_work+0x85c/0x1280 [bnx2]
  14. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8127ddb0>] ? swiotlb_map_page+0x0/0x100
  15. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffffa013c2cd>] ? bnx2_poll_msix+0x3d/0xc0 [bnx2]
  16. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff813fb7e8>] ? dma_issue_pending_all+0x68/0xa0
  17. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8142d7c3>] ? net_rx_action+0x103/0x2f0
  18. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81094cd3>] ? hrtimer_get_next_event+0xc3/0x100
  19. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81072991>] ? __do_softirq+0xc1/0x1e0
  20. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff810d8b40>] ? handle_IRQ_event+0x60/0x170
  21. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
  22. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
  23. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81072775>] ? irq_exit+0x85/0x90
  24. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff814f1eb5>] ? do_IRQ+0x75/0xf0
  25. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
  26. Aug 23 11:10:20 row1r05s03 kernel: <EOI> [<ffffffff812c2d4e>] ? intel_idle+0xde/0x170
  27. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff812c2d31>] ? intel_idle+0xc1/0x170
  28. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff813f6c67>] ? cpuidle_idle_call+0xa7/0x140
  29. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
  30. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff814d109a>] ? rest_init+0x7a/0x80
  31. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
  32. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
  33. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
  34. Aug 23 11:10:20 row1r05s03 kernel: swapper: page allocation failure. order:0, mode:0x20
  35. Aug 23 11:10:20 row1r05s03 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.19.1.el6.x86_64 #1
  36. Aug 23 11:10:20 row1r05s03 kernel: Call Trace:
  37. Aug 23 11:10:20 row1r05s03 kernel: <IRQ> [<ffffffff811231ff>] ? __alloc_pages_nodemask+0x77f/0x940
  38. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115d1a2>] ? kmem_getpages+0x62/0x170
  39. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ddba>] ? fallback_alloc+0x1ba/0x270
  40. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  41. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115db39>] ? ____cache_alloc_node+0x99/0x160
  42. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ed00>] ? kmem_cache_alloc_node_trace+0x90/0x200
  43. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ef1d>] ? __kmalloc_node+0x4d/0x60
  44. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8141ea1d>] ? __alloc_skb+0x6d/0x190
  45. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8141eba4>] ? __netdev_alloc_skb+0x24/0x50
  46. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffffa013b86c>] ? bnx2_poll_work+0x85c/0x1280 [bnx2]
  47. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8127ddb0>] ? swiotlb_map_page+0x0/0x100
  48. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffffa013c2cd>] ? bnx2_poll_msix+0x3d/0xc0 [bnx2]
  49. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff813fb7e8>] ? dma_issue_pending_all+0x68/0xa0
  50. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8142d7c3>] ? net_rx_action+0x103/0x2f0
  51. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81094cd3>] ? hrtimer_get_next_event+0xc3/0x100
  52. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81072991>] ? __do_softirq+0xc1/0x1e0
  53. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff810d8b40>] ? handle_IRQ_event+0x60/0x170
  54. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
  55. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
  56. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81072775>] ? irq_exit+0x85/0x90
  57. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff814f1eb5>] ? do_IRQ+0x75/0xf0
  58. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
  59. Aug 23 11:10:20 row1r05s03 kernel: <EOI> [<ffffffff812c2d4e>] ? intel_idle+0xde/0x170
  60. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff812c2d31>] ? intel_idle+0xc1/0x170
  61. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff813f6c67>] ? cpuidle_idle_call+0xa7/0x140
  62. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
  63. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff814d109a>] ? rest_init+0x7a/0x80
  64. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
  65. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
  66. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
  67. Aug 23 11:10:20 row1r05s03 kernel: swapper: page allocation failure. order:0, mode:0x20
  68. Aug 23 11:10:20 row1r05s03 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.19.1.el6.x86_64 #1
  69. Aug 23 11:10:20 row1r05s03 kernel: Call Trace:
  70. Aug 23 11:10:20 row1r05s03 kernel: <IRQ> [<ffffffff811231ff>] ? __alloc_pages_nodemask+0x77f/0x940
  71. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115d1a2>] ? kmem_getpages+0x62/0x170
  72. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ddba>] ? fallback_alloc+0x1ba/0x270
  73. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  74. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115db39>] ? ____cache_alloc_node+0x99/0x160
  75. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ed00>] ? kmem_cache_alloc_node_trace+0x90/0x200
  76. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ef1d>] ? __kmalloc_node+0x4d/0x60
  77. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8141ea1d>] ? __alloc_skb+0x6d/0x190
  78. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8141eba4>] ? __netdev_alloc_skb+0x24/0x50
  79. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffffa013b86c>] ? bnx2_poll_work+0x85c/0x1280 [bnx2]
  80. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8127ddb0>] ? swiotlb_map_page+0x0/0x100
  81. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffffa013c2cd>] ? bnx2_poll_msix+0x3d/0xc0 [bnx2]
  82. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff813fb7e8>] ? dma_issue_pending_all+0x68/0xa0
  83. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8142d7c3>] ? net_rx_action+0x103/0x2f0
  84. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81094cd3>] ? hrtimer_get_next_event+0xc3/0x100
  85. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81072991>] ? __do_softirq+0xc1/0x1e0
  86. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff810d8b40>] ? handle_IRQ_event+0x60/0x170
  87. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
  88. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
  89. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81072775>] ? irq_exit+0x85/0x90
  90. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff814f1eb5>] ? do_IRQ+0x75/0xf0
  91. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
  92. Aug 23 11:10:20 row1r05s03 kernel: <EOI> [<ffffffff812c2d4e>] ? intel_idle+0xde/0x170
  93. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff812c2d31>] ? intel_idle+0xc1/0x170
  94. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff813f6c67>] ? cpuidle_idle_call+0xa7/0x140
  95. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
  96. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff814d109a>] ? rest_init+0x7a/0x80
  97. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
  98. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
  99. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
  100. Aug 23 11:10:20 row1r05s03 kernel: swapper: page allocation failure. order:0, mode:0x20
  101. Aug 23 11:10:20 row1r05s03 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.19.1.el6.x86_64 #1
  102. Aug 23 11:10:20 row1r05s03 kernel: Call Trace:
  103. Aug 23 11:10:20 row1r05s03 kernel: <IRQ> [<ffffffff811231ff>] ? __alloc_pages_nodemask+0x77f/0x940
  104. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115d1a2>] ? kmem_getpages+0x62/0x170
  105. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ddba>] ? fallback_alloc+0x1ba/0x270
  106. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  107. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115db39>] ? ____cache_alloc_node+0x99/0x160
  108. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ed00>] ? kmem_cache_alloc_node_trace+0x90/0x200
  109. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ef1d>] ? __kmalloc_node+0x4d/0x60
  110. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8141ea1d>] ? __alloc_skb+0x6d/0x190
  111. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8141eba4>] ? __netdev_alloc_skb+0x24/0x50
  112. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffffa013b86c>] ? bnx2_poll_work+0x85c/0x1280 [bnx2]
  113. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8127ddb0>] ? swiotlb_map_page+0x0/0x100
  114. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffffa013c2cd>] ? bnx2_poll_msix+0x3d/0xc0 [bnx2]
  115. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff813fb7e8>] ? dma_issue_pending_all+0x68/0xa0
  116. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8142d7c3>] ? net_rx_action+0x103/0x2f0
  117. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81094cd3>] ? hrtimer_get_next_event+0xc3/0x100
  118. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81072991>] ? __do_softirq+0xc1/0x1e0
  119. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff810d8b40>] ? handle_IRQ_event+0x60/0x170
  120. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
  121. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
  122. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81072775>] ? irq_exit+0x85/0x90
  123. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff814f1eb5>] ? do_IRQ+0x75/0xf0
  124. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
  125. Aug 23 11:10:20 row1r05s03 kernel: <EOI> [<ffffffff812c2d4e>] ? intel_idle+0xde/0x170
  126. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff812c2d31>] ? intel_idle+0xc1/0x170
  127. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff813f6c67>] ? cpuidle_idle_call+0xa7/0x140
  128. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
  129. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff814d109a>] ? rest_init+0x7a/0x80
  130. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
  131. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
  132. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
  133. Aug 23 11:10:20 row1r05s03 kernel: swapper: page allocation failure. order:0, mode:0x20
  134. Aug 23 11:10:20 row1r05s03 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.19.1.el6.x86_64 #1
  135. Aug 23 11:10:20 row1r05s03 kernel: Call Trace:
  136. Aug 23 11:10:20 row1r05s03 kernel: <IRQ> [<ffffffff811231ff>] ? __alloc_pages_nodemask+0x77f/0x940
  137. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115d1a2>] ? kmem_getpages+0x62/0x170
  138. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ddba>] ? fallback_alloc+0x1ba/0x270
  139. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  140. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115db39>] ? ____cache_alloc_node+0x99/0x160
  141. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ed00>] ? kmem_cache_alloc_node_trace+0x90/0x200
  142. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ef1d>] ? __kmalloc_node+0x4d/0x60
  143. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8141ea1d>] ? __alloc_skb+0x6d/0x190
  144. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8141eba4>] ? __netdev_alloc_skb+0x24/0x50
  145. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffffa013b86c>] ? bnx2_poll_work+0x85c/0x1280 [bnx2]
  146. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8127ddb0>] ? swiotlb_map_page+0x0/0x100
  147. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffffa013c2cd>] ? bnx2_poll_msix+0x3d/0xc0 [bnx2]
  148. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff813fb7e8>] ? dma_issue_pending_all+0x68/0xa0
  149. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8142d7c3>] ? net_rx_action+0x103/0x2f0
  150. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81094cd3>] ? hrtimer_get_next_event+0xc3/0x100
  151. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81072991>] ? __do_softirq+0xc1/0x1e0
  152. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff810d8b40>] ? handle_IRQ_event+0x60/0x170
  153. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
  154. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
  155. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81072775>] ? irq_exit+0x85/0x90
  156. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff814f1eb5>] ? do_IRQ+0x75/0xf0
  157. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
  158. Aug 23 11:10:20 row1r05s03 kernel: <EOI> [<ffffffff812c2d4e>] ? intel_idle+0xde/0x170
  159. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff812c2d31>] ? intel_idle+0xc1/0x170
  160. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff813f6c67>] ? cpuidle_idle_call+0xa7/0x140
  161. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
  162. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff814d109a>] ? rest_init+0x7a/0x80
  163. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
  164. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
  165. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
  166. Aug 23 11:10:20 row1r05s03 kernel: swapper: page allocation failure. order:0, mode:0x20
  167. Aug 23 11:10:20 row1r05s03 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.19.1.el6.x86_64 #1
  168. Aug 23 11:10:20 row1r05s03 kernel: Call Trace:
  169. Aug 23 11:10:20 row1r05s03 kernel: <IRQ> [<ffffffff811231ff>] ? __alloc_pages_nodemask+0x77f/0x940
  170. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115d1a2>] ? kmem_getpages+0x62/0x170
  171. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ddba>] ? fallback_alloc+0x1ba/0x270
  172. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  173. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115db39>] ? ____cache_alloc_node+0x99/0x160
  174. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ed00>] ? kmem_cache_alloc_node_trace+0x90/0x200
  175. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ef1d>] ? __kmalloc_node+0x4d/0x60
  176. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8141ea1d>] ? __alloc_skb+0x6d/0x190
  177. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8141eba4>] ? __netdev_alloc_skb+0x24/0x50
  178. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffffa013b86c>] ? bnx2_poll_work+0x85c/0x1280 [bnx2]
  179. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8127ddb0>] ? swiotlb_map_page+0x0/0x100
  180. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffffa013c2cd>] ? bnx2_poll_msix+0x3d/0xc0 [bnx2]
  181. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff813fb7e8>] ? dma_issue_pending_all+0x68/0xa0
  182. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8142d7c3>] ? net_rx_action+0x103/0x2f0
  183. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81094cd3>] ? hrtimer_get_next_event+0xc3/0x100
  184. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81072991>] ? __do_softirq+0xc1/0x1e0
  185. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff810d8b40>] ? handle_IRQ_event+0x60/0x170
  186. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
  187. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
  188. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81072775>] ? irq_exit+0x85/0x90
  189. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff814f1eb5>] ? do_IRQ+0x75/0xf0
  190. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
  191. Aug 23 11:10:20 row1r05s03 kernel: <EOI> [<ffffffff812c2d4e>] ? intel_idle+0xde/0x170
  192. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff812c2d31>] ? intel_idle+0xc1/0x170
  193. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff813f6c67>] ? cpuidle_idle_call+0xa7/0x140
  194. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
  195. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff814d109a>] ? rest_init+0x7a/0x80
  196. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
  197. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
  198. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
  199. Aug 23 11:10:20 row1r05s03 kernel: swapper: page allocation failure. order:0, mode:0x20
  200. Aug 23 11:10:20 row1r05s03 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.19.1.el6.x86_64 #1
  201. Aug 23 11:10:20 row1r05s03 kernel: Call Trace:
  202. Aug 23 11:10:20 row1r05s03 kernel: <IRQ> [<ffffffff811231ff>] ? __alloc_pages_nodemask+0x77f/0x940
  203. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115d1a2>] ? kmem_getpages+0x62/0x170
  204. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ddba>] ? fallback_alloc+0x1ba/0x270
  205. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  206. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115db39>] ? ____cache_alloc_node+0x99/0x160
  207. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ed00>] ? kmem_cache_alloc_node_trace+0x90/0x200
  208. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ef1d>] ? __kmalloc_node+0x4d/0x60
  209. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8141ea1d>] ? __alloc_skb+0x6d/0x190
  210. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8141eba4>] ? __netdev_alloc_skb+0x24/0x50
  211. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffffa013b86c>] ? bnx2_poll_work+0x85c/0x1280 [bnx2]
  212. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8127ddb0>] ? swiotlb_map_page+0x0/0x100
  213. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffffa013c2cd>] ? bnx2_poll_msix+0x3d/0xc0 [bnx2]
  214. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff813fb7e8>] ? dma_issue_pending_all+0x68/0xa0
  215. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8142d7c3>] ? net_rx_action+0x103/0x2f0
  216. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81094cd3>] ? hrtimer_get_next_event+0xc3/0x100
  217. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81072991>] ? __do_softirq+0xc1/0x1e0
  218. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff810d8b40>] ? handle_IRQ_event+0x60/0x170
  219. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
  220. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
  221. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81072775>] ? irq_exit+0x85/0x90
  222. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff814f1eb5>] ? do_IRQ+0x75/0xf0
  223. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
  224. Aug 23 11:10:20 row1r05s03 kernel: <EOI> [<ffffffff812c2d4e>] ? intel_idle+0xde/0x170
  225. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff812c2d31>] ? intel_idle+0xc1/0x170
  226. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff813f6c67>] ? cpuidle_idle_call+0xa7/0x140
  227. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
  228. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff814d109a>] ? rest_init+0x7a/0x80
  229. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
  230. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
  231. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
  232. Aug 23 11:10:20 row1r05s03 kernel: swapper: page allocation failure. order:0, mode:0x20
  233. Aug 23 11:10:20 row1r05s03 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.19.1.el6.x86_64 #1
  234. Aug 23 11:10:20 row1r05s03 kernel: Call Trace:
  235. Aug 23 11:10:20 row1r05s03 kernel: <IRQ> [<ffffffff811231ff>] ? __alloc_pages_nodemask+0x77f/0x940
  236. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115d1a2>] ? kmem_getpages+0x62/0x170
  237. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ddba>] ? fallback_alloc+0x1ba/0x270
  238. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  239. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115db39>] ? ____cache_alloc_node+0x99/0x160
  240. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ed00>] ? kmem_cache_alloc_node_trace+0x90/0x200
  241. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ef1d>] ? __kmalloc_node+0x4d/0x60
  242. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8141ea1d>] ? __alloc_skb+0x6d/0x190
  243. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8141eba4>] ? __netdev_alloc_skb+0x24/0x50
  244. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffffa013b86c>] ? bnx2_poll_work+0x85c/0x1280 [bnx2]
  245. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8127ddb0>] ? swiotlb_map_page+0x0/0x100
  246. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffffa013c2cd>] ? bnx2_poll_msix+0x3d/0xc0 [bnx2]
  247. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff813fb7e8>] ? dma_issue_pending_all+0x68/0xa0
  248. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8142d7c3>] ? net_rx_action+0x103/0x2f0
  249. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81094cd3>] ? hrtimer_get_next_event+0xc3/0x100
  250. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81072991>] ? __do_softirq+0xc1/0x1e0
  251. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff810d8b40>] ? handle_IRQ_event+0x60/0x170
  252. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
  253. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
  254. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81072775>] ? irq_exit+0x85/0x90
  255. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff814f1eb5>] ? do_IRQ+0x75/0xf0
  256. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
  257. Aug 23 11:10:20 row1r05s03 kernel: <EOI> [<ffffffff812c2d4e>] ? intel_idle+0xde/0x170
  258. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff812c2d31>] ? intel_idle+0xc1/0x170
  259. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff813f6c67>] ? cpuidle_idle_call+0xa7/0x140
  260. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
  261. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff814d109a>] ? rest_init+0x7a/0x80
  262. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
  263. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
  264. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
  265. Aug 23 11:10:20 row1r05s03 kernel: swapper: page allocation failure. order:0, mode:0x20
  266. Aug 23 11:10:20 row1r05s03 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.19.1.el6.x86_64 #1
  267. Aug 23 11:10:20 row1r05s03 kernel: Call Trace:
  268. Aug 23 11:10:20 row1r05s03 kernel: <IRQ> [<ffffffff811231ff>] ? __alloc_pages_nodemask+0x77f/0x940
  269. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115d1a2>] ? kmem_getpages+0x62/0x170
  270. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ddba>] ? fallback_alloc+0x1ba/0x270
  271. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  272. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115db39>] ? ____cache_alloc_node+0x99/0x160
  273. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ed00>] ? kmem_cache_alloc_node_trace+0x90/0x200
  274. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ef1d>] ? __kmalloc_node+0x4d/0x60
  275. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8141ea1d>] ? __alloc_skb+0x6d/0x190
  276. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8141eba4>] ? __netdev_alloc_skb+0x24/0x50
  277. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffffa013b86c>] ? bnx2_poll_work+0x85c/0x1280 [bnx2]
  278. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8127ddb0>] ? swiotlb_map_page+0x0/0x100
  279. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffffa013c2cd>] ? bnx2_poll_msix+0x3d/0xc0 [bnx2]
  280. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff813fb7e8>] ? dma_issue_pending_all+0x68/0xa0
  281. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8142d7c3>] ? net_rx_action+0x103/0x2f0
  282. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81094cd3>] ? hrtimer_get_next_event+0xc3/0x100
  283. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81072991>] ? __do_softirq+0xc1/0x1e0
  284. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff810d8b40>] ? handle_IRQ_event+0x60/0x170
  285. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
  286. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
  287. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81072775>] ? irq_exit+0x85/0x90
  288. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff814f1eb5>] ? do_IRQ+0x75/0xf0
  289. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
  290. Aug 23 11:10:20 row1r05s03 kernel: <EOI> [<ffffffff812c2d4e>] ? intel_idle+0xde/0x170
  291. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff812c2d31>] ? intel_idle+0xc1/0x170
  292. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff813f6c67>] ? cpuidle_idle_call+0xa7/0x140
  293. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
  294. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff814d109a>] ? rest_init+0x7a/0x80
  295. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
  296. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
  297. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
  298. Aug 23 11:10:20 row1r05s03 kernel: swapper: page allocation failure. order:0, mode:0x20
  299. Aug 23 11:10:20 row1r05s03 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.19.1.el6.x86_64 #1
  300. Aug 23 11:10:20 row1r05s03 kernel: Call Trace:
  301. Aug 23 11:10:20 row1r05s03 kernel: <IRQ> [<ffffffff811231ff>] ? __alloc_pages_nodemask+0x77f/0x940
  302. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115d1a2>] ? kmem_getpages+0x62/0x170
  303. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ddba>] ? fallback_alloc+0x1ba/0x270
  304. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  305. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115db39>] ? ____cache_alloc_node+0x99/0x160
  306. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ed00>] ? kmem_cache_alloc_node_trace+0x90/0x200
  307. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8115ef1d>] ? __kmalloc_node+0x4d/0x60
  308. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8141ea1d>] ? __alloc_skb+0x6d/0x190
  309. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8141eba4>] ? __netdev_alloc_skb+0x24/0x50
  310. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffffa013b86c>] ? bnx2_poll_work+0x85c/0x1280 [bnx2]
  311. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8127ddb0>] ? swiotlb_map_page+0x0/0x100
  312. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffffa013c2cd>] ? bnx2_poll_msix+0x3d/0xc0 [bnx2]
  313. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff813fb7e8>] ? dma_issue_pending_all+0x68/0xa0
  314. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8142d7c3>] ? net_rx_action+0x103/0x2f0
  315. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81094cd3>] ? hrtimer_get_next_event+0xc3/0x100
  316. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81072991>] ? __do_softirq+0xc1/0x1e0
  317. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff810d8b40>] ? handle_IRQ_event+0x60/0x170
  318. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
  319. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
  320. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81072775>] ? irq_exit+0x85/0x90
  321. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff814f1eb5>] ? do_IRQ+0x75/0xf0
  322. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
  323. Aug 23 11:10:20 row1r05s03 kernel: <EOI> [<ffffffff812c2d4e>] ? intel_idle+0xde/0x170
  324. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff812c2d31>] ? intel_idle+0xc1/0x170
  325. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff813f6c67>] ? cpuidle_idle_call+0xa7/0x140
  326. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
  327. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff814d109a>] ? rest_init+0x7a/0x80
  328. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
  329. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
  330. Aug 23 11:10:20 row1r05s03 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
  331. Aug 23 11:10:35 row1r05s03 kernel: INFO: task kswapd0:98 blocked for more than 120 seconds.
  332. Aug 23 11:10:35 row1r05s03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  333. Aug 23 11:10:35 row1r05s03 kernel: kswapd0 D 0000000000000005 0 98 2 0x00000000
  334. Aug 23 11:10:35 row1r05s03 kernel: ffff8802262ad530 0000000000000046 ffff8802262ad4f8 ffff8802262ad4f4
  335. Aug 23 11:10:35 row1r05s03 kernel: ffff8802262ad4d0 ffff88022fc28e00 ffff88002f816680 0000000000000400
  336. Aug 23 11:10:35 row1r05s03 kernel: ffff8802262abaf8 ffff8802262adfd8 000000000000fb88 ffff8802262abaf8
  337. Aug 23 11:10:35 row1r05s03 kernel: Call Trace:
  338. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff814eaac5>] schedule_timeout+0x215/0x2e0
  339. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff810966e7>] ? down_trylock+0x37/0x50
  340. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026b67f>] ? xfs_buf_cond_lock+0x2f/0xc0 [xfs]
  341. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff814eb9e2>] __down+0x72/0xb0
  342. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cae2>] ? _xfs_buf_find+0x102/0x280 [xfs]
  343. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff810967f1>] down+0x41/0x50
  344. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026c923>] xfs_buf_lock+0x53/0x110 [xfs]
  345. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cae2>] _xfs_buf_find+0x102/0x280 [xfs]
  346. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cccb>] xfs_buf_get+0x6b/0x1a0 [xfs]
  347. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026d33c>] xfs_buf_read+0x2c/0x100 [xfs]
  348. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0262f88>] xfs_trans_read_buf+0x1f8/0x400 [xfs]
  349. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0219086>] xfs_read_agf+0x66/0x1a0 [xfs]
  350. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02191f9>] xfs_alloc_read_agf+0x39/0xd0 [xfs]
  351. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa021bc01>] xfs_alloc_fix_freelist+0x411/0x480 [xfs]
  352. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0260103>] ? xfs_trans_free_item_desc+0x33/0x40 [xfs]
  353. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0260247>] ? xfs_trans_free_items+0x87/0xb0 [xfs]
  354. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa025ccf8>] ? xfs_perag_get+0x38/0xb0 [xfs]
  355. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa021bd18>] xfs_free_extent+0xa8/0x130 [xfs]
  356. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02253bd>] xfs_bmap_finish+0x15d/0x1a0 [xfs]
  357. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa024aaa7>] xfs_itruncate_finish+0x167/0x330 [xfs]
  358. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0266028>] xfs_free_eofblocks+0x1b8/0x280 [xfs]
  359. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02665c0>] xfs_inactive+0xc0/0x460 [xfs]
  360. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0276a94>] ? xfs_inode_set_reclaim_tag+0x84/0xa0 [xfs]
  361. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0273d80>] xfs_fs_clear_inode+0xa0/0xd0 [xfs]
  362. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81190c3c>] clear_inode+0xac/0x140
  363. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81190d10>] dispose_list+0x40/0x120
  364. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81191064>] shrink_icache_memory+0x274/0x2e0
  365. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81128c7a>] shrink_slab+0x12a/0x1a0
  366. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8112bb4d>] balance_pgdat+0x57d/0x7e0
  367. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8112c160>] ? isolate_pages_global+0x0/0x350
  368. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8112035d>] ? zone_watermark_ok_safe+0xad/0xc0
  369. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8112bee6>] kswapd+0x136/0x3b0
  370. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81090990>] ? autoremove_wake_function+0x0/0x40
  371. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8112bdb0>] ? kswapd+0x0/0x3b0
  372. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81090626>] kthread+0x96/0xa0
  373. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
  374. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81090590>] ? kthread+0x0/0xa0
  375. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
  376. Aug 23 11:10:35 row1r05s03 kernel: INFO: task nfsd:2687 blocked for more than 120 seconds.
  377. Aug 23 11:10:35 row1r05s03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  378. Aug 23 11:10:35 row1r05s03 kernel: nfsd D 0000000000000005 0 2687 2 0x00000080
  379. Aug 23 11:10:35 row1r05s03 kernel: ffff8802171d6960 0000000000000046 0000000000000000 0000000000000000
  380. Aug 23 11:10:35 row1r05s03 kernel: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
  381. Aug 23 11:10:35 row1r05s03 kernel: ffff8802277e7098 ffff8802171d7fd8 000000000000fb88 ffff8802277e7098
  382. Aug 23 11:10:35 row1r05s03 kernel: Call Trace:
  383. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81132ad9>] ? zone_statistics+0x99/0xc0
  384. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff814eaac5>] schedule_timeout+0x215/0x2e0
  385. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff810966e7>] ? down_trylock+0x37/0x50
  386. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026b67f>] ? xfs_buf_cond_lock+0x2f/0xc0 [xfs]
  387. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff814eb9e2>] __down+0x72/0xb0
  388. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cae2>] ? _xfs_buf_find+0x102/0x280 [xfs]
  389. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff810967f1>] down+0x41/0x50
  390. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026c923>] xfs_buf_lock+0x53/0x110 [xfs]
  391. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cae2>] _xfs_buf_find+0x102/0x280 [xfs]
  392. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cccb>] xfs_buf_get+0x6b/0x1a0 [xfs]
  393. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026d33c>] xfs_buf_read+0x2c/0x100 [xfs]
  394. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0262f88>] xfs_trans_read_buf+0x1f8/0x400 [xfs]
  395. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0219086>] xfs_read_agf+0x66/0x1a0 [xfs]
  396. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02191f9>] xfs_alloc_read_agf+0x39/0xd0 [xfs]
  397. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa021bc01>] xfs_alloc_fix_freelist+0x411/0x480 [xfs]
  398. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8115d9bb>] ? cache_alloc_refill+0x15b/0x240
  399. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa025ccf8>] ? xfs_perag_get+0x38/0xb0 [xfs]
  400. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa021bd18>] xfs_free_extent+0xa8/0x130 [xfs]
  401. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02253bd>] xfs_bmap_finish+0x15d/0x1a0 [xfs]
  402. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa024aaa7>] xfs_itruncate_finish+0x167/0x330 [xfs]
  403. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0266028>] xfs_free_eofblocks+0x1b8/0x280 [xfs]
  404. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02665c0>] xfs_inactive+0xc0/0x460 [xfs]
  405. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0276a94>] ? xfs_inode_set_reclaim_tag+0x84/0xa0 [xfs]
  406. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0273d80>] xfs_fs_clear_inode+0xa0/0xd0 [xfs]
  407. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81190c3c>] clear_inode+0xac/0x140
  408. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81190d10>] dispose_list+0x40/0x120
  409. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81191064>] shrink_icache_memory+0x274/0x2e0
  410. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81128c7a>] shrink_slab+0x12a/0x1a0
  411. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8112abff>] do_try_to_free_pages+0x2ff/0x520
  412. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81113e8f>] ? zone_watermark_ok+0x1f/0x30
  413. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8112b00d>] try_to_free_pages+0x9d/0x130
  414. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81131976>] ? next_online_pgdat+0x26/0x50
  415. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8112c160>] ? isolate_pages_global+0x0/0x350
  416. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81122e8d>] __alloc_pages_nodemask+0x40d/0x940
  417. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8115d1a2>] kmem_getpages+0x62/0x170
  418. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8115ddba>] fallback_alloc+0x1ba/0x270
  419. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  420. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8115db39>] ____cache_alloc_node+0x99/0x160
  421. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8115ed00>] kmem_cache_alloc_node_trace+0x90/0x200
  422. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8114967f>] vm_map_ram+0x1cf/0x470
  423. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026b67f>] ? xfs_buf_cond_lock+0x2f/0xc0 [xfs]
  424. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026bbb1>] _xfs_buf_map_pages+0xc1/0x120 [xfs]
  425. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cd63>] xfs_buf_get+0x103/0x1a0 [xfs]
  426. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026d33c>] xfs_buf_read+0x2c/0x100 [xfs]
  427. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0262f88>] xfs_trans_read_buf+0x1f8/0x400 [xfs]
  428. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa023048e>] xfs_btree_read_buf_block+0x5e/0xd0 [xfs]
  429. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa022e8cc>] ? xfs_btree_ptr_offset+0x4c/0x90 [xfs]
  430. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02310c2>] xfs_btree_increment+0x202/0x2a0 [xfs]
  431. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa021ac91>] xfs_alloc_ag_vextent_near+0x231/0xb20 [xfs]
  432. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa021bbdd>] ? xfs_alloc_fix_freelist+0x3ed/0x480 [xfs]
  433. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa021b655>] xfs_alloc_ag_vextent+0xd5/0x100 [xfs]
  434. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa021c216>] xfs_alloc_vextent+0x476/0x610 [xfs]
  435. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0246ca3>] xfs_ialloc_ag_alloc+0x3a3/0x4b0 [xfs]
  436. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0246e8f>] xfs_dialloc+0xdf/0x850 [xfs]
  437. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8109672f>] ? up+0x2f/0x50
  438. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026355b>] ? xfs_trans_brelse+0xdb/0x150 [xfs]
  439. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81221eda>] ? symhash+0x1a/0x50
  440. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81221eb4>] ? symcmp+0x14/0x20
  441. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81221ace>] ? hashtab_search+0x7e/0x90
  442. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa024aed0>] xfs_ialloc+0x60/0x6f0 [xfs]
  443. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0263b14>] xfs_dir_ialloc+0x74/0x2b0 [xfs]
  444. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0265b60>] xfs_create+0x440/0x640 [xfs]
  445. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0272b3d>] xfs_vn_mknod+0xad/0x1c0 [xfs]
  446. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0272c80>] xfs_vn_create+0x10/0x20 [xfs]
  447. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81183b04>] vfs_create+0xb4/0xe0
  448. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03cf6c6>] nfsd_create_v3+0x366/0x4c0 [nfsd]
  449. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03d7413>] nfsd3_proc_create+0x123/0x1b0 [nfsd]
  450. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03c843e>] nfsd_dispatch+0xfe/0x240 [nfsd]
  451. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02fc794>] svc_process_common+0x344/0x640 [sunrpc]
  452. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8105fa40>] ? default_wake_function+0x0/0x20
  453. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02fcdd0>] svc_process+0x110/0x160 [sunrpc]
  454. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03c8b62>] nfsd+0xc2/0x160 [nfsd]
  455. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03c8aa0>] ? nfsd+0x0/0x160 [nfsd]
  456. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81090626>] kthread+0x96/0xa0
  457. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
  458. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81090590>] ? kthread+0x0/0xa0
  459. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
  460. Aug 23 11:10:35 row1r05s03 kernel: INFO: task nfsd:2688 blocked for more than 120 seconds.
  461. Aug 23 11:10:35 row1r05s03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  462. Aug 23 11:10:35 row1r05s03 kernel: nfsd D 0000000000000000 0 2688 2 0x00000080
  463. Aug 23 11:10:35 row1r05s03 kernel: ffff8802172d1730 0000000000000046 0000000000000000 0000000000000003
  464. Aug 23 11:10:35 row1r05s03 kernel: ffff8802288f8020 ffff880224440900 0000000000000002 00000000814bfedc
  465. Aug 23 11:10:35 row1r05s03 kernel: ffff8802240485f8 ffff8802172d1fd8 000000000000fb88 ffff8802240485f8
  466. Aug 23 11:10:35 row1r05s03 kernel: Call Trace:
  467. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff814ec1b5>] rwsem_down_failed_common+0x95/0x1d0
  468. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff814ec313>] rwsem_down_write_failed+0x23/0x30
  469. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81275213>] call_rwsem_down_write_failed+0x13/0x20
  470. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff814eb812>] ? down_write+0x32/0x40
  471. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0247cfe>] xfs_ilock+0x7e/0xd0 [xfs]
  472. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026f1f5>] xfs_file_aio_write_checks+0x45/0x1d0 [xfs]
  473. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026f6b1>] xfs_file_buffered_aio_write+0x81/0x1a0 [xfs]
  474. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026fbdb>] xfs_file_aio_write+0x1bb/0x2e0 [xfs]
  475. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0248493>] ? xfs_iget+0x473/0x6f0 [xfs]
  476. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026fa20>] ? xfs_file_aio_write+0x0/0x2e0 [xfs]
  477. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8117614b>] do_sync_readv_writev+0xfb/0x140
  478. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81090990>] ? autoremove_wake_function+0x0/0x40
  479. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81219b4b>] ? selinux_file_permission+0xfb/0x150
  480. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8120ca26>] ? security_file_permission+0x16/0x20
  481. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff811770d6>] do_readv_writev+0xd6/0x1f0
  482. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81215ec4>] ? inode_has_perm+0x54/0xa0
  483. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81177236>] vfs_writev+0x46/0x60
  484. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03cd555>] nfsd_vfs_write+0x105/0x430 [nfsd]
  485. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81173f32>] ? dentry_open+0x52/0xc0
  486. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03cf9fb>] ? nfsd_open+0x1db/0x230 [nfsd]
  487. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03cfe17>] nfsd_write+0xe7/0x100 [nfsd]
  488. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03d754f>] nfsd3_proc_write+0xaf/0x140 [nfsd]
  489. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03c843e>] nfsd_dispatch+0xfe/0x240 [nfsd]
  490. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02fc794>] svc_process_common+0x344/0x640 [sunrpc]
  491. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8105fa40>] ? default_wake_function+0x0/0x20
  492. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02fcdd0>] svc_process+0x110/0x160 [sunrpc]
  493. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03c8b62>] nfsd+0xc2/0x160 [nfsd]
  494. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03c8aa0>] ? nfsd+0x0/0x160 [nfsd]
  495. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81090626>] kthread+0x96/0xa0
  496. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
  497. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81090590>] ? kthread+0x0/0xa0
  498. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
  499. Aug 23 11:10:35 row1r05s03 kernel: INFO: task nfsd:2689 blocked for more than 120 seconds.
  500. Aug 23 11:10:35 row1r05s03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  501. Aug 23 11:10:35 row1r05s03 kernel: nfsd D 0000000000000004 0 2689 2 0x00000080
  502. Aug 23 11:10:35 row1r05s03 kernel: ffff8802173cd5e0 0000000000000046 ffff8802173cd5c0 ffffffffa013aa51
  503. Aug 23 11:10:35 row1r05s03 kernel: 00000001f530e0ce 0000000000000001 ffff8802288f8020 ffff880224440800
  504. Aug 23 11:10:35 row1r05s03 kernel: ffff880227309ab8 ffff8802173cdfd8 000000000000fb88 ffff880227309ab8
  505. Aug 23 11:10:35 row1r05s03 kernel: Call Trace:
  506. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa013aa51>] ? bnx2_start_xmit+0x241/0x800 [bnx2]
  507. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff814eaac5>] schedule_timeout+0x215/0x2e0
  508. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81446ff8>] ? sch_direct_xmit+0x78/0x1c0
  509. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff814eb9e2>] __down+0x72/0xb0
  510. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cae2>] ? _xfs_buf_find+0x102/0x280 [xfs]
  511. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff810967f1>] down+0x41/0x50
  512. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026c923>] xfs_buf_lock+0x53/0x110 [xfs]
  513. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cae2>] _xfs_buf_find+0x102/0x280 [xfs]
  514. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cccb>] xfs_buf_get+0x6b/0x1a0 [xfs]
  515. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026d33c>] xfs_buf_read+0x2c/0x100 [xfs]
  516. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0262f88>] xfs_trans_read_buf+0x1f8/0x400 [xfs]
  517. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0245774>] xfs_read_agi+0x74/0x100 [xfs]
  518. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa024582e>] xfs_ialloc_read_agi+0x2e/0x90 [xfs]
  519. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0245b13>] xfs_ialloc_ag_select+0x133/0x270 [xfs]
  520. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0247187>] xfs_dialloc+0x3d7/0x850 [xfs]
  521. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81419fde>] ? release_sock+0xce/0xe0
  522. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81221eda>] ? symhash+0x1a/0x50
  523. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81221eb4>] ? symcmp+0x14/0x20
  524. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81221ace>] ? hashtab_search+0x7e/0x90
  525. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa024aed0>] xfs_ialloc+0x60/0x6f0 [xfs]
  526. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0263b14>] xfs_dir_ialloc+0x74/0x2b0 [xfs]
  527. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0265b60>] xfs_create+0x440/0x640 [xfs]
  528. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0272b3d>] xfs_vn_mknod+0xad/0x1c0 [xfs]
  529. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0272c80>] xfs_vn_create+0x10/0x20 [xfs]
  530. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81183b04>] vfs_create+0xb4/0xe0
  531. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03cf6c6>] nfsd_create_v3+0x366/0x4c0 [nfsd]
  532. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03d7413>] nfsd3_proc_create+0x123/0x1b0 [nfsd]
  533. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03c843e>] nfsd_dispatch+0xfe/0x240 [nfsd]
  534. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02fc794>] svc_process_common+0x344/0x640 [sunrpc]
  535. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8105fa40>] ? default_wake_function+0x0/0x20
  536. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02fcdd0>] svc_process+0x110/0x160 [sunrpc]
  537. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03c8b62>] nfsd+0xc2/0x160 [nfsd]
  538. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03c8aa0>] ? nfsd+0x0/0x160 [nfsd]
  539. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81090626>] kthread+0x96/0xa0
  540. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
  541. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81090590>] ? kthread+0x0/0xa0
  542. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
  543. Aug 23 11:10:35 row1r05s03 kernel: INFO: task nfsd:2690 blocked for more than 120 seconds.
  544. Aug 23 11:10:35 row1r05s03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  545. Aug 23 11:10:35 row1r05s03 kernel: nfsd D 0000000000000001 0 2690 2 0x00000080
  546. Aug 23 11:10:35 row1r05s03 kernel: ffff880216ce88c0 0000000000000046 0000000000000000 ffff880000023de8
  547. Aug 23 11:10:35 row1r05s03 kernel: ffff880216ce88c0 ffffffff8111fb93 ffffea00069887b8 0000000000000058
  548. Aug 23 11:10:35 row1r05s03 kernel: ffff88021e399af8 ffff880216ce9fd8 000000000000fb88 ffff88021e399af8
  549. Aug 23 11:10:35 row1r05s03 kernel: Call Trace:
  550. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8111fb93>] ? __rmqueue+0x143/0x490
  551. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff814eaac5>] schedule_timeout+0x215/0x2e0
  552. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff810966e7>] ? down_trylock+0x37/0x50
  553. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026b67f>] ? xfs_buf_cond_lock+0x2f/0xc0 [xfs]
  554. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff814eb9e2>] __down+0x72/0xb0
  555. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cae2>] ? _xfs_buf_find+0x102/0x280 [xfs]
  556. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff810967f1>] down+0x41/0x50
  557. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026c923>] xfs_buf_lock+0x53/0x110 [xfs]
  558. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cae2>] _xfs_buf_find+0x102/0x280 [xfs]
  559. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cccb>] xfs_buf_get+0x6b/0x1a0 [xfs]
  560. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026d33c>] xfs_buf_read+0x2c/0x100 [xfs]
  561. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0262f88>] xfs_trans_read_buf+0x1f8/0x400 [xfs]
  562. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0219086>] xfs_read_agf+0x66/0x1a0 [xfs]
  563. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02191f9>] xfs_alloc_read_agf+0x39/0xd0 [xfs]
  564. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa021bc01>] xfs_alloc_fix_freelist+0x411/0x480 [xfs]
  565. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0260103>] ? xfs_trans_free_item_desc+0x33/0x40 [xfs]
  566. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0260247>] ? xfs_trans_free_items+0x87/0xb0 [xfs]
  567. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa025ccf8>] ? xfs_perag_get+0x38/0xb0 [xfs]
  568. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa021bd18>] xfs_free_extent+0xa8/0x130 [xfs]
  569. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02253bd>] xfs_bmap_finish+0x15d/0x1a0 [xfs]
  570. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa024aaa7>] xfs_itruncate_finish+0x167/0x330 [xfs]
  571. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0266028>] xfs_free_eofblocks+0x1b8/0x280 [xfs]
  572. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02665c0>] xfs_inactive+0xc0/0x460 [xfs]
  573. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0276a94>] ? xfs_inode_set_reclaim_tag+0x84/0xa0 [xfs]
  574. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0273d80>] xfs_fs_clear_inode+0xa0/0xd0 [xfs]
  575. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81190c3c>] clear_inode+0xac/0x140
  576. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81190d10>] dispose_list+0x40/0x120
  577. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81191064>] shrink_icache_memory+0x274/0x2e0
  578. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81128c7a>] shrink_slab+0x12a/0x1a0
  579. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8112abff>] do_try_to_free_pages+0x2ff/0x520
  580. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81113e8f>] ? zone_watermark_ok+0x1f/0x30
  581. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8112b00d>] try_to_free_pages+0x9d/0x130
  582. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81131976>] ? next_online_pgdat+0x26/0x50
  583. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8112c160>] ? isolate_pages_global+0x0/0x350
  584. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81122e8d>] __alloc_pages_nodemask+0x40d/0x940
  585. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8115d1a2>] kmem_getpages+0x62/0x170
  586. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8115ddba>] fallback_alloc+0x1ba/0x270
  587. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  588. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8115db39>] ____cache_alloc_node+0x99/0x160
  589. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8115ed00>] kmem_cache_alloc_node_trace+0x90/0x200
  590. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8114872b>] alloc_vmap_area+0x5b/0x310
  591. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff811496b7>] vm_map_ram+0x207/0x470
  592. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026b67f>] ? xfs_buf_cond_lock+0x2f/0xc0 [xfs]
  593. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026bbb1>] _xfs_buf_map_pages+0xc1/0x120 [xfs]
  594. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cd63>] xfs_buf_get+0x103/0x1a0 [xfs]
  595. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026d33c>] xfs_buf_read+0x2c/0x100 [xfs]
  596. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0262f88>] xfs_trans_read_buf+0x1f8/0x400 [xfs]
  597. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa023048e>] xfs_btree_read_buf_block+0x5e/0xd0 [xfs]
  598. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa022e8cc>] ? xfs_btree_ptr_offset+0x4c/0x90 [xfs]
  599. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02310c2>] xfs_btree_increment+0x202/0x2a0 [xfs]
  600. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa021ac91>] xfs_alloc_ag_vextent_near+0x231/0xb20 [xfs]
  601. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa021bbdd>] ? xfs_alloc_fix_freelist+0x3ed/0x480 [xfs]
  602. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa021b655>] xfs_alloc_ag_vextent+0xd5/0x100 [xfs]
  603. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa021c216>] xfs_alloc_vextent+0x476/0x610 [xfs]
  604. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0246ca3>] xfs_ialloc_ag_alloc+0x3a3/0x4b0 [xfs]
  605. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0246e8f>] xfs_dialloc+0xdf/0x850 [xfs]
  606. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81419fde>] ? release_sock+0xce/0xe0
  607. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81221eda>] ? symhash+0x1a/0x50
  608. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81221eb4>] ? symcmp+0x14/0x20
  609. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81221ace>] ? hashtab_search+0x7e/0x90
  610. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa024aed0>] xfs_ialloc+0x60/0x6f0 [xfs]
  611. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0263b14>] xfs_dir_ialloc+0x74/0x2b0 [xfs]
  612. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0265b60>] xfs_create+0x440/0x640 [xfs]
  613. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0272b3d>] xfs_vn_mknod+0xad/0x1c0 [xfs]
  614. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0272c80>] xfs_vn_create+0x10/0x20 [xfs]
  615. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81183b04>] vfs_create+0xb4/0xe0
  616. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03cf6c6>] nfsd_create_v3+0x366/0x4c0 [nfsd]
  617. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03d7413>] nfsd3_proc_create+0x123/0x1b0 [nfsd]
  618. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03c843e>] nfsd_dispatch+0xfe/0x240 [nfsd]
  619. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02fc794>] svc_process_common+0x344/0x640 [sunrpc]
  620. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8105fa40>] ? default_wake_function+0x0/0x20
  621. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02fcdd0>] svc_process+0x110/0x160 [sunrpc]
  622. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03c8b62>] nfsd+0xc2/0x160 [nfsd]
  623. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03c8aa0>] ? nfsd+0x0/0x160 [nfsd]
  624. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81090626>] kthread+0x96/0xa0
  625. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
  626. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81090590>] ? kthread+0x0/0xa0
  627. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
  628. Aug 23 11:10:35 row1r05s03 kernel: INFO: task nfsd:2691 blocked for more than 120 seconds.
  629. Aug 23 11:10:35 row1r05s03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  630. Aug 23 11:10:35 row1r05s03 kernel: nfsd D 0000000000000000 0 2691 2 0x00000080
  631. Aug 23 11:10:35 row1r05s03 kernel: ffff880216de55e0 0000000000000046 ffff880216de55c0 ffffffffa013aa51
  632. Aug 23 11:10:35 row1r05s03 kernel: 00000002171e94c2 0000000000000001 ffff8802288f8020 ffff880224440800
  633. Aug 23 11:10:35 row1r05s03 kernel: ffff8802277e7af8 ffff880216de5fd8 000000000000fb88 ffff8802277e7af8
  634. Aug 23 11:10:35 row1r05s03 kernel: Call Trace:
  635. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa013aa51>] ? bnx2_start_xmit+0x241/0x800 [bnx2]
  636. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff814eaac5>] schedule_timeout+0x215/0x2e0
  637. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8100b9ce>] ? common_interrupt+0xe/0x13
  638. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8100b9ce>] ? common_interrupt+0xe/0x13
  639. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff814eb9e2>] __down+0x72/0xb0
  640. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cae2>] ? _xfs_buf_find+0x102/0x280 [xfs]
  641. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff810967f1>] down+0x41/0x50
  642. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026c923>] xfs_buf_lock+0x53/0x110 [xfs]
  643. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cae2>] _xfs_buf_find+0x102/0x280 [xfs]
  644. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cccb>] xfs_buf_get+0x6b/0x1a0 [xfs]
  645. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026d33c>] xfs_buf_read+0x2c/0x100 [xfs]
  646. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0262f88>] xfs_trans_read_buf+0x1f8/0x400 [xfs]
  647. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0245774>] xfs_read_agi+0x74/0x100 [xfs]
  648. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa024582e>] xfs_ialloc_read_agi+0x2e/0x90 [xfs]
  649. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0245b13>] xfs_ialloc_ag_select+0x133/0x270 [xfs]
  650. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0247187>] xfs_dialloc+0x3d7/0x850 [xfs]
  651. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff814232b9>] ? skb_copy_datagram_iovec+0x1e9/0x2c0
  652. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8141a09c>] ? lock_sock_nested+0xac/0xc0
  653. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81419fde>] ? release_sock+0xce/0xe0
  654. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81221eda>] ? symhash+0x1a/0x50
  655. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81221eb4>] ? symcmp+0x14/0x20
  656. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81221ace>] ? hashtab_search+0x7e/0x90
  657. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa024aed0>] xfs_ialloc+0x60/0x6f0 [xfs]
  658. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0263b14>] xfs_dir_ialloc+0x74/0x2b0 [xfs]
  659. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0265b60>] xfs_create+0x440/0x640 [xfs]
  660. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0272b3d>] xfs_vn_mknod+0xad/0x1c0 [xfs]
  661. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0272c80>] xfs_vn_create+0x10/0x20 [xfs]
  662. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81183b04>] vfs_create+0xb4/0xe0
  663. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03cf6c6>] nfsd_create_v3+0x366/0x4c0 [nfsd]
  664. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03d7413>] nfsd3_proc_create+0x123/0x1b0 [nfsd]
  665. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03c843e>] nfsd_dispatch+0xfe/0x240 [nfsd]
  666. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02fc794>] svc_process_common+0x344/0x640 [sunrpc]
  667. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8105fa40>] ? default_wake_function+0x0/0x20
  668. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02fcdd0>] svc_process+0x110/0x160 [sunrpc]
  669. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03c8b62>] nfsd+0xc2/0x160 [nfsd]
  670. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03c8aa0>] ? nfsd+0x0/0x160 [nfsd]
  671. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81090626>] kthread+0x96/0xa0
  672. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
  673. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81090590>] ? kthread+0x0/0xa0
  674. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
  675. Aug 23 11:10:35 row1r05s03 kernel: INFO: task nfsd:2692 blocked for more than 120 seconds.
  676. Aug 23 11:10:35 row1r05s03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  677. Aug 23 11:10:35 row1r05s03 kernel: nfsd D 0000000000000002 0 2692 2 0x00000080
  678. Aug 23 11:10:35 row1r05s03 kernel: ffff880216eff670 0000000000000046 ffff880216eff620 ffffffff81429d7c
  679. Aug 23 11:10:35 row1r05s03 kernel: ffff880227253800 0000000400000000 ffff880227309458 ffffffffa013e0e0
  680. Aug 23 11:10:35 row1r05s03 kernel: ffff880227309058 ffff880216efffd8 000000000000fb88 ffff880227309058
  681. Aug 23 11:10:35 row1r05s03 kernel: Call Trace:
  682. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81429d7c>] ? dev_hard_start_xmit+0x2bc/0x3f0
  683. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81446ff8>] ? sch_direct_xmit+0x78/0x1c0
  684. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff814eaac5>] schedule_timeout+0x215/0x2e0
  685. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8127ddb0>] ? swiotlb_map_page+0x0/0x100
  686. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff814eb9e2>] __down+0x72/0xb0
  687. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cae2>] ? _xfs_buf_find+0x102/0x280 [xfs]
  688. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff810967f1>] down+0x41/0x50
  689. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026c923>] xfs_buf_lock+0x53/0x110 [xfs]
  690. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cae2>] _xfs_buf_find+0x102/0x280 [xfs]
  691. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cccb>] xfs_buf_get+0x6b/0x1a0 [xfs]
  692. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026d33c>] xfs_buf_read+0x2c/0x100 [xfs]
  693. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0262f88>] xfs_trans_read_buf+0x1f8/0x400 [xfs]
  694. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0219086>] xfs_read_agf+0x66/0x1a0 [xfs]
  695. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02191f9>] xfs_alloc_read_agf+0x39/0xd0 [xfs]
  696. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa021bc01>] xfs_alloc_fix_freelist+0x411/0x480 [xfs]
  697. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0260103>] ? xfs_trans_free_item_desc+0x33/0x40 [xfs]
  698. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0260247>] ? xfs_trans_free_items+0x87/0xb0 [xfs]
  699. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa025ccf8>] ? xfs_perag_get+0x38/0xb0 [xfs]
  700. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa021bd18>] xfs_free_extent+0xa8/0x130 [xfs]
  701. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02253bd>] xfs_bmap_finish+0x15d/0x1a0 [xfs]
  702. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa024aaa7>] xfs_itruncate_finish+0x167/0x330 [xfs]
  703. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0267243>] xfs_setattr+0x373/0xab0 [xfs]
  704. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8121649b>] ? dentry_has_perm+0x5b/0x80
  705. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02730eb>] xfs_vn_setattr+0x1b/0x20 [xfs]
  706. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81192398>] notify_change+0x168/0x340
  707. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03cdcdc>] nfsd_setattr+0x32c/0x410 [nfsd]
  708. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03d6be6>] nfsd3_proc_setattr+0x76/0xc0 [nfsd]
  709. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03c843e>] nfsd_dispatch+0xfe/0x240 [nfsd]
  710. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02fc794>] svc_process_common+0x344/0x640 [sunrpc]
  711. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8105fa40>] ? default_wake_function+0x0/0x20
  712. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02fcdd0>] svc_process+0x110/0x160 [sunrpc]
  713. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03c8b62>] nfsd+0xc2/0x160 [nfsd]
  714. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa03c8aa0>] ? nfsd+0x0/0x160 [nfsd]
  715. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81090626>] kthread+0x96/0xa0
  716. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
  717. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81090590>] ? kthread+0x0/0xa0
  718. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
  719. Aug 23 11:10:35 row1r05s03 kernel: INFO: task flush-253:2:2945 blocked for more than 120 seconds.
  720. Aug 23 11:10:35 row1r05s03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  721. Aug 23 11:10:35 row1r05s03 kernel: flush-253:2 D 0000000000000002 0 2945 2 0x00000080
  722. Aug 23 11:10:35 row1r05s03 kernel: ffff8802166d9120 0000000000000046 0000000000000000 ffffffff8124cf3e
  723. Aug 23 11:10:35 row1r05s03 kernel: ffffea0006d8bd18 0000000000000286 ffff8802166d90b0 ffffffff810966e7
  724. Aug 23 11:10:35 row1r05s03 kernel: ffff880225681058 ffff8802166d9fd8 000000000000fb88 ffff880225681058
  725. Aug 23 11:10:35 row1r05s03 kernel: Call Trace:
  726. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8124cf3e>] ? generic_unplug_device+0x2e/0x50
  727. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff810966e7>] ? down_trylock+0x37/0x50
  728. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff814eaac5>] schedule_timeout+0x215/0x2e0
  729. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0260597>] ? xfs_trans_add_item+0x57/0x70 [xfs]
  730. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff814eb9e2>] __down+0x72/0xb0
  731. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cae2>] ? _xfs_buf_find+0x102/0x280 [xfs]
  732. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff810967f1>] down+0x41/0x50
  733. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026c923>] xfs_buf_lock+0x53/0x110 [xfs]
  734. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cae2>] _xfs_buf_find+0x102/0x280 [xfs]
  735. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026cccb>] xfs_buf_get+0x6b/0x1a0 [xfs]
  736. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026d33c>] xfs_buf_read+0x2c/0x100 [xfs]
  737. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0262f88>] xfs_trans_read_buf+0x1f8/0x400 [xfs]
  738. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0219086>] xfs_read_agf+0x66/0x1a0 [xfs]
  739. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02191f9>] xfs_alloc_read_agf+0x39/0xd0 [xfs]
  740. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa021bc01>] xfs_alloc_fix_freelist+0x411/0x480 [xfs]
  741. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81278250>] ? sg_init_table+0x30/0x50
  742. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff812782ee>] ? __sg_alloc_table+0x7e/0x130
  743. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8135d7c0>] ? scsi_sg_alloc+0x0/0x60
  744. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa025ccf8>] ? xfs_perag_get+0x38/0xb0 [xfs]
  745. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa025ccf8>] ? xfs_perag_get+0x38/0xb0 [xfs]
  746. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa021bf2b>] xfs_alloc_vextent+0x18b/0x610 [xfs]
  747. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0226504>] xfs_bmap_btalloc+0x6c4/0x700 [xfs]
  748. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa003caf0>] ? megasas_fire_cmd_gen2+0x50/0x70 [megaraid_sas]
  749. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa022654e>] xfs_bmap_alloc+0xe/0x10 [xfs]
  750. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa022cd80>] xfs_bmapi+0x9a0/0x1090 [xfs]
  751. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa022434d>] ? xfs_bmap_search_multi_extents+0xad/0x120 [xfs]
  752. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0260597>] ? xfs_trans_add_item+0x57/0x70 [xfs]
  753. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa024ed0e>] xfs_iomap_write_allocate+0x17e/0x340 [xfs]
  754. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa0269ad3>] xfs_map_blocks+0x193/0x250 [xfs]
  755. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa026a5c9>] xfs_vm_writepage+0x1f9/0x510 [xfs]
  756. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81123f87>] __writepage+0x17/0x40
  757. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff811252a9>] write_cache_pages+0x1c9/0x4a0
  758. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81123f70>] ? __writepage+0x0/0x40
  759. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff811255a4>] generic_writepages+0x24/0x30
  760. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffffa02698fe>] xfs_vm_writepages+0x5e/0x80 [xfs]
  761. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff811255d1>] do_writepages+0x21/0x40
  762. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8119fe8d>] writeback_single_inode+0xdd/0x290
  763. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff811a029e>] writeback_sb_inodes+0xce/0x180
  764. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff811a03fb>] writeback_inodes_wb+0xab/0x1b0
  765. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8105ab19>] ? find_busiest_queue+0x69/0x150
  766. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff811a079b>] wb_writeback+0x29b/0x3f0
  767. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff814e9c50>] ? thread_return+0x4e/0x76e
  768. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8107d572>] ? del_timer_sync+0x22/0x30
  769. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff811a0a89>] wb_do_writeback+0x199/0x240
  770. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff811a0b93>] bdi_writeback_task+0x63/0x1b0
  771. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81090857>] ? bit_waitqueue+0x17/0xd0
  772. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81134170>] ? bdi_start_fn+0x0/0x100
  773. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff811341f6>] bdi_start_fn+0x86/0x100
  774. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81134170>] ? bdi_start_fn+0x0/0x100
  775. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81090626>] kthread+0x96/0xa0
  776. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
  777. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff81090590>] ? kthread+0x0/0xa0
  778. Aug 23 11:10:35 row1r05s03 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
  779. Aug 23 11:11:21 row1r05s03 kernel: __ratelimit: 117 callbacks suppressed
  780. Aug 23 11:11:21 row1r05s03 kernel: swapper: page allocation failure. order:0, mode:0x20
  781. Aug 23 11:11:21 row1r05s03 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.19.1.el6.x86_64 #1
  782. Aug 23 11:11:21 row1r05s03 kernel: Call Trace:
  783. Aug 23 11:11:21 row1r05s03 kernel: <IRQ> [<ffffffff811231ff>] ? __alloc_pages_nodemask+0x77f/0x940
  784. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8115d1a2>] ? kmem_getpages+0x62/0x170
  785. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8115ddba>] ? fallback_alloc+0x1ba/0x270
  786. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  787. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8115db39>] ? ____cache_alloc_node+0x99/0x160
  788. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8115ed00>] ? kmem_cache_alloc_node_trace+0x90/0x200
  789. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8115ef1d>] ? __kmalloc_node+0x4d/0x60
  790. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8141ea1d>] ? __alloc_skb+0x6d/0x190
  791. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff814732b2>] ? tcp_collapse+0x1a2/0x3f0
  792. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff81132ad9>] ? zone_statistics+0x99/0xc0
  793. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff81474655>] ? tcp_try_rmem_schedule+0x245/0x360
  794. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8147491b>] ? tcp_data_queue+0x1ab/0xc70
  795. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8121bf23>] ? selinux_socket_sock_rcv_skb+0x83/0x220
  796. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff81477ab9>] ? tcp_rcv_established+0x369/0x800
  797. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8147fa73>] ? tcp_v4_do_rcv+0x2e3/0x430
  798. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff814812fe>] ? tcp_v4_rcv+0x4fe/0x8d0
  799. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8145f1ad>] ? ip_local_deliver_finish+0xdd/0x2d0
  800. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8145f438>] ? ip_local_deliver+0x98/0xa0
  801. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8145e8fd>] ? ip_rcv_finish+0x12d/0x440
  802. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8145ee85>] ? ip_rcv+0x275/0x350
  803. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8142946b>] ? __netif_receive_skb+0x49b/0x6f0
  804. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8142b5e8>] ? netif_receive_skb+0x58/0x60
  805. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8142b6f0>] ? napi_skb_finish+0x50/0x70
  806. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8142d6a9>] ? napi_gro_receive+0x39/0x50
  807. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffffa013bd5f>] ? bnx2_poll_work+0xd4f/0x1280 [bnx2]
  808. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8127ddb0>] ? swiotlb_map_page+0x0/0x100
  809. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffffa013c2cd>] ? bnx2_poll_msix+0x3d/0xc0 [bnx2]
  810. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff810949d2>] ? enqueue_hrtimer+0x82/0xd0
  811. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff813fb7e8>] ? dma_issue_pending_all+0x68/0xa0
  812. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8142d7c3>] ? net_rx_action+0x103/0x2f0
  813. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff81072991>] ? __do_softirq+0xc1/0x1e0
  814. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff810d8b40>] ? handle_IRQ_event+0x60/0x170
  815. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
  816. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
  817. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff81072775>] ? irq_exit+0x85/0x90
  818. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff814f1eb5>] ? do_IRQ+0x75/0xf0
  819. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
  820. Aug 23 11:11:21 row1r05s03 kernel: <EOI> [<ffffffff812c2d4e>] ? intel_idle+0xde/0x170
  821. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff812c2d31>] ? intel_idle+0xc1/0x170
  822. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff813f6c67>] ? cpuidle_idle_call+0xa7/0x140
  823. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
  824. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff814d109a>] ? rest_init+0x7a/0x80
  825. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
  826. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
  827. Aug 23 11:11:21 row1r05s03 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
  828. Aug 23 11:12:16 row1r05s03 kernel: swapper: page allocation failure. order:0, mode:0x20
  829. Aug 23 11:12:16 row1r05s03 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.19.1.el6.x86_64 #1
  830. Aug 23 11:12:16 row1r05s03 kernel: Call Trace:
  831. Aug 23 11:12:16 row1r05s03 kernel: <IRQ> [<ffffffff811231ff>] ? __alloc_pages_nodemask+0x77f/0x940
  832. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115d1a2>] ? kmem_getpages+0x62/0x170
  833. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ddba>] ? fallback_alloc+0x1ba/0x270
  834. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  835. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115db39>] ? ____cache_alloc_node+0x99/0x160
  836. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ed00>] ? kmem_cache_alloc_node_trace+0x90/0x200
  837. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ef1d>] ? __kmalloc_node+0x4d/0x60
  838. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8141ea1d>] ? __alloc_skb+0x6d/0x190
  839. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8141eba4>] ? __netdev_alloc_skb+0x24/0x50
  840. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffffa013b86c>] ? bnx2_poll_work+0x85c/0x1280 [bnx2]
  841. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8127ddb0>] ? swiotlb_map_page+0x0/0x100
  842. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffffa013c2cd>] ? bnx2_poll_msix+0x3d/0xc0 [bnx2]
  843. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff813fb7e8>] ? dma_issue_pending_all+0x68/0xa0
  844. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8142d7c3>] ? net_rx_action+0x103/0x2f0
  845. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81094cd3>] ? hrtimer_get_next_event+0xc3/0x100
  846. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81072991>] ? __do_softirq+0xc1/0x1e0
  847. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff810d8b40>] ? handle_IRQ_event+0x60/0x170
  848. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
  849. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
  850. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81072775>] ? irq_exit+0x85/0x90
  851. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff814f1eb5>] ? do_IRQ+0x75/0xf0
  852. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
  853. Aug 23 11:12:16 row1r05s03 kernel: <EOI> [<ffffffff812c2d4e>] ? intel_idle+0xde/0x170
  854. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff812c2d31>] ? intel_idle+0xc1/0x170
  855. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff810979ed>] ? sched_clock_cpu+0xcd/0x110
  856. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff813f6c67>] ? cpuidle_idle_call+0xa7/0x140
  857. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
  858. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff814d109a>] ? rest_init+0x7a/0x80
  859. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
  860. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
  861. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
  862. Aug 23 11:12:16 row1r05s03 kernel: swapper: page allocation failure. order:0, mode:0x20
  863. Aug 23 11:12:16 row1r05s03 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.19.1.el6.x86_64 #1
  864. Aug 23 11:12:16 row1r05s03 kernel: Call Trace:
  865. Aug 23 11:12:16 row1r05s03 kernel: <IRQ> [<ffffffff811231ff>] ? __alloc_pages_nodemask+0x77f/0x940
  866. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115d1a2>] ? kmem_getpages+0x62/0x170
  867. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ddba>] ? fallback_alloc+0x1ba/0x270
  868. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  869. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115db39>] ? ____cache_alloc_node+0x99/0x160
  870. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ed00>] ? kmem_cache_alloc_node_trace+0x90/0x200
  871. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ef1d>] ? __kmalloc_node+0x4d/0x60
  872. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8141ea1d>] ? __alloc_skb+0x6d/0x190
  873. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8141eba4>] ? __netdev_alloc_skb+0x24/0x50
  874. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffffa013b86c>] ? bnx2_poll_work+0x85c/0x1280 [bnx2]
  875. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8127ddb0>] ? swiotlb_map_page+0x0/0x100
  876. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffffa013c2cd>] ? bnx2_poll_msix+0x3d/0xc0 [bnx2]
  877. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff813fb7e8>] ? dma_issue_pending_all+0x68/0xa0
  878. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8142d7c3>] ? net_rx_action+0x103/0x2f0
  879. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81094cd3>] ? hrtimer_get_next_event+0xc3/0x100
  880. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81072991>] ? __do_softirq+0xc1/0x1e0
  881. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff810d8b40>] ? handle_IRQ_event+0x60/0x170
  882. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
  883. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
  884. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81072775>] ? irq_exit+0x85/0x90
  885. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff814f1eb5>] ? do_IRQ+0x75/0xf0
  886. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
  887. Aug 23 11:12:16 row1r05s03 kernel: <EOI> [<ffffffff812c2d4e>] ? intel_idle+0xde/0x170
  888. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff812c2d31>] ? intel_idle+0xc1/0x170
  889. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff810979ed>] ? sched_clock_cpu+0xcd/0x110
  890. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff813f6c67>] ? cpuidle_idle_call+0xa7/0x140
  891. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
  892. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff814d109a>] ? rest_init+0x7a/0x80
  893. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
  894. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
  895. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
  896. Aug 23 11:12:16 row1r05s03 kernel: swapper: page allocation failure. order:0, mode:0x20
  897. Aug 23 11:12:16 row1r05s03 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.19.1.el6.x86_64 #1
  898. Aug 23 11:12:16 row1r05s03 kernel: Call Trace:
  899. Aug 23 11:12:16 row1r05s03 kernel: <IRQ> [<ffffffff811231ff>] ? __alloc_pages_nodemask+0x77f/0x940
  900. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115d1a2>] ? kmem_getpages+0x62/0x170
  901. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ddba>] ? fallback_alloc+0x1ba/0x270
  902. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  903. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115db39>] ? ____cache_alloc_node+0x99/0x160
  904. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ed00>] ? kmem_cache_alloc_node_trace+0x90/0x200
  905. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ef1d>] ? __kmalloc_node+0x4d/0x60
  906. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8141ea1d>] ? __alloc_skb+0x6d/0x190
  907. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8141eba4>] ? __netdev_alloc_skb+0x24/0x50
  908. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffffa013b86c>] ? bnx2_poll_work+0x85c/0x1280 [bnx2]
  909. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8127ddb0>] ? swiotlb_map_page+0x0/0x100
  910. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffffa013c2cd>] ? bnx2_poll_msix+0x3d/0xc0 [bnx2]
  911. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff813fb7e8>] ? dma_issue_pending_all+0x68/0xa0
  912. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8142d7c3>] ? net_rx_action+0x103/0x2f0
  913. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81094cd3>] ? hrtimer_get_next_event+0xc3/0x100
  914. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81072991>] ? __do_softirq+0xc1/0x1e0
  915. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff810d8b40>] ? handle_IRQ_event+0x60/0x170
  916. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
  917. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
  918. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81072775>] ? irq_exit+0x85/0x90
  919. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff814f1eb5>] ? do_IRQ+0x75/0xf0
  920. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
  921. Aug 23 11:12:16 row1r05s03 kernel: <EOI> [<ffffffff812c2d4e>] ? intel_idle+0xde/0x170
  922. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff812c2d31>] ? intel_idle+0xc1/0x170
  923. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff810979ed>] ? sched_clock_cpu+0xcd/0x110
  924. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff813f6c67>] ? cpuidle_idle_call+0xa7/0x140
  925. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
  926. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff814d109a>] ? rest_init+0x7a/0x80
  927. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
  928. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
  929. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
  930. Aug 23 11:12:16 row1r05s03 kernel: swapper: page allocation failure. order:0, mode:0x20
  931. Aug 23 11:12:16 row1r05s03 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.19.1.el6.x86_64 #1
  932. Aug 23 11:12:16 row1r05s03 kernel: Call Trace:
  933. Aug 23 11:12:16 row1r05s03 kernel: <IRQ> [<ffffffff811231ff>] ? __alloc_pages_nodemask+0x77f/0x940
  934. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115d1a2>] ? kmem_getpages+0x62/0x170
  935. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ddba>] ? fallback_alloc+0x1ba/0x270
  936. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  937. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115db39>] ? ____cache_alloc_node+0x99/0x160
  938. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ed00>] ? kmem_cache_alloc_node_trace+0x90/0x200
  939. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ef1d>] ? __kmalloc_node+0x4d/0x60
  940. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8141ea1d>] ? __alloc_skb+0x6d/0x190
  941. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8141eba4>] ? __netdev_alloc_skb+0x24/0x50
  942. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffffa013b86c>] ? bnx2_poll_work+0x85c/0x1280 [bnx2]
  943. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8127ddb0>] ? swiotlb_map_page+0x0/0x100
  944. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffffa013c2cd>] ? bnx2_poll_msix+0x3d/0xc0 [bnx2]
  945. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff813fb7e8>] ? dma_issue_pending_all+0x68/0xa0
  946. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8142d7c3>] ? net_rx_action+0x103/0x2f0
  947. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81094cd3>] ? hrtimer_get_next_event+0xc3/0x100
  948. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81072991>] ? __do_softirq+0xc1/0x1e0
  949. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff810d8b40>] ? handle_IRQ_event+0x60/0x170
  950. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
  951. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
  952. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81072775>] ? irq_exit+0x85/0x90
  953. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff814f1eb5>] ? do_IRQ+0x75/0xf0
  954. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
  955. Aug 23 11:12:16 row1r05s03 kernel: <EOI> [<ffffffff812c2d4e>] ? intel_idle+0xde/0x170
  956. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff812c2d31>] ? intel_idle+0xc1/0x170
  957. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff810979ed>] ? sched_clock_cpu+0xcd/0x110
  958. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff813f6c67>] ? cpuidle_idle_call+0xa7/0x140
  959. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
  960. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff814d109a>] ? rest_init+0x7a/0x80
  961. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
  962. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
  963. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
  964. Aug 23 11:12:16 row1r05s03 kernel: swapper: page allocation failure. order:0, mode:0x20
  965. Aug 23 11:12:16 row1r05s03 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.19.1.el6.x86_64 #1
  966. Aug 23 11:12:16 row1r05s03 kernel: Call Trace:
  967. Aug 23 11:12:16 row1r05s03 kernel: <IRQ> [<ffffffff811231ff>] ? __alloc_pages_nodemask+0x77f/0x940
  968. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115d1a2>] ? kmem_getpages+0x62/0x170
  969. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ddba>] ? fallback_alloc+0x1ba/0x270
  970. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  971. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115db39>] ? ____cache_alloc_node+0x99/0x160
  972. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ed00>] ? kmem_cache_alloc_node_trace+0x90/0x200
  973. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ef1d>] ? __kmalloc_node+0x4d/0x60
  974. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8141ea1d>] ? __alloc_skb+0x6d/0x190
  975. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8141eba4>] ? __netdev_alloc_skb+0x24/0x50
  976. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffffa013b86c>] ? bnx2_poll_work+0x85c/0x1280 [bnx2]
  977. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8127ddb0>] ? swiotlb_map_page+0x0/0x100
  978. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffffa013c2cd>] ? bnx2_poll_msix+0x3d/0xc0 [bnx2]
  979. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff813fb7e8>] ? dma_issue_pending_all+0x68/0xa0
  980. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8142d7c3>] ? net_rx_action+0x103/0x2f0
  981. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81094cd3>] ? hrtimer_get_next_event+0xc3/0x100
  982. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81072991>] ? __do_softirq+0xc1/0x1e0
  983. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff810d8b40>] ? handle_IRQ_event+0x60/0x170
  984. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
  985. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
  986. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81072775>] ? irq_exit+0x85/0x90
  987. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff814f1eb5>] ? do_IRQ+0x75/0xf0
  988. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
  989. Aug 23 11:12:16 row1r05s03 kernel: <EOI> [<ffffffff812c2d4e>] ? intel_idle+0xde/0x170
  990. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff812c2d31>] ? intel_idle+0xc1/0x170
  991. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff810979ed>] ? sched_clock_cpu+0xcd/0x110
  992. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff813f6c67>] ? cpuidle_idle_call+0xa7/0x140
  993. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
  994. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff814d109a>] ? rest_init+0x7a/0x80
  995. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
  996. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
  997. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
  998. Aug 23 11:12:16 row1r05s03 kernel: swapper: page allocation failure. order:0, mode:0x20
  999. Aug 23 11:12:16 row1r05s03 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.19.1.el6.x86_64 #1
  1000. Aug 23 11:12:16 row1r05s03 kernel: Call Trace:
  1001. Aug 23 11:12:16 row1r05s03 kernel: <IRQ> [<ffffffff811231ff>] ? __alloc_pages_nodemask+0x77f/0x940
  1002. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115d1a2>] ? kmem_getpages+0x62/0x170
  1003. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ddba>] ? fallback_alloc+0x1ba/0x270
  1004. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  1005. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115db39>] ? ____cache_alloc_node+0x99/0x160
  1006. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ed00>] ? kmem_cache_alloc_node_trace+0x90/0x200
  1007. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ef1d>] ? __kmalloc_node+0x4d/0x60
  1008. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8141ea1d>] ? __alloc_skb+0x6d/0x190
  1009. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8141eba4>] ? __netdev_alloc_skb+0x24/0x50
  1010. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffffa013b86c>] ? bnx2_poll_work+0x85c/0x1280 [bnx2]
  1011. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8127ddb0>] ? swiotlb_map_page+0x0/0x100
  1012. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffffa013c2cd>] ? bnx2_poll_msix+0x3d/0xc0 [bnx2]
  1013. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff813fb7e8>] ? dma_issue_pending_all+0x68/0xa0
  1014. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8142d7c3>] ? net_rx_action+0x103/0x2f0
  1015. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81094cd3>] ? hrtimer_get_next_event+0xc3/0x100
  1016. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81072991>] ? __do_softirq+0xc1/0x1e0
  1017. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff810d8b40>] ? handle_IRQ_event+0x60/0x170
  1018. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
  1019. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
  1020. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81072775>] ? irq_exit+0x85/0x90
  1021. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff814f1eb5>] ? do_IRQ+0x75/0xf0
  1022. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
  1023. Aug 23 11:12:16 row1r05s03 kernel: <EOI> [<ffffffff812c2d4e>] ? intel_idle+0xde/0x170
  1024. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff812c2d31>] ? intel_idle+0xc1/0x170
  1025. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff810979ed>] ? sched_clock_cpu+0xcd/0x110
  1026. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff813f6c67>] ? cpuidle_idle_call+0xa7/0x140
  1027. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
  1028. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff814d109a>] ? rest_init+0x7a/0x80
  1029. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
  1030. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
  1031. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
  1032. Aug 23 11:12:16 row1r05s03 kernel: swapper: page allocation failure. order:0, mode:0x20
  1033. Aug 23 11:12:16 row1r05s03 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.19.1.el6.x86_64 #1
  1034. Aug 23 11:12:16 row1r05s03 kernel: Call Trace:
  1035. Aug 23 11:12:16 row1r05s03 kernel: <IRQ> [<ffffffff811231ff>] ? __alloc_pages_nodemask+0x77f/0x940
  1036. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115d1a2>] ? kmem_getpages+0x62/0x170
  1037. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ddba>] ? fallback_alloc+0x1ba/0x270
  1038. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  1039. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115db39>] ? ____cache_alloc_node+0x99/0x160
  1040. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ed00>] ? kmem_cache_alloc_node_trace+0x90/0x200
  1041. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ef1d>] ? __kmalloc_node+0x4d/0x60
  1042. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8141ea1d>] ? __alloc_skb+0x6d/0x190
  1043. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8141eba4>] ? __netdev_alloc_skb+0x24/0x50
  1044. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffffa013b86c>] ? bnx2_poll_work+0x85c/0x1280 [bnx2]
  1045. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8127ddb0>] ? swiotlb_map_page+0x0/0x100
  1046. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffffa013c2cd>] ? bnx2_poll_msix+0x3d/0xc0 [bnx2]
  1047. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff813fb7e8>] ? dma_issue_pending_all+0x68/0xa0
  1048. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8142d7c3>] ? net_rx_action+0x103/0x2f0
  1049. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81094cd3>] ? hrtimer_get_next_event+0xc3/0x100
  1050. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81072991>] ? __do_softirq+0xc1/0x1e0
  1051. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff810d8b40>] ? handle_IRQ_event+0x60/0x170
  1052. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
  1053. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
  1054. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81072775>] ? irq_exit+0x85/0x90
  1055. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff814f1eb5>] ? do_IRQ+0x75/0xf0
  1056. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
  1057. Aug 23 11:12:16 row1r05s03 kernel: <EOI> [<ffffffff812c2d4e>] ? intel_idle+0xde/0x170
  1058. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff812c2d31>] ? intel_idle+0xc1/0x170
  1059. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff810979ed>] ? sched_clock_cpu+0xcd/0x110
  1060. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff813f6c67>] ? cpuidle_idle_call+0xa7/0x140
  1061. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
  1062. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff814d109a>] ? rest_init+0x7a/0x80
  1063. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
  1064. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
  1065. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
  1066. Aug 23 11:12:16 row1r05s03 kernel: swapper: page allocation failure. order:0, mode:0x20
  1067. Aug 23 11:12:16 row1r05s03 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.19.1.el6.x86_64 #1
  1068. Aug 23 11:12:16 row1r05s03 kernel: Call Trace:
  1069. Aug 23 11:12:16 row1r05s03 kernel: <IRQ> [<ffffffff811231ff>] ? __alloc_pages_nodemask+0x77f/0x940
  1070. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115d1a2>] ? kmem_getpages+0x62/0x170
  1071. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ddba>] ? fallback_alloc+0x1ba/0x270
  1072. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  1073. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115db39>] ? ____cache_alloc_node+0x99/0x160
  1074. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ed00>] ? kmem_cache_alloc_node_trace+0x90/0x200
  1075. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ef1d>] ? __kmalloc_node+0x4d/0x60
  1076. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8141ea1d>] ? __alloc_skb+0x6d/0x190
  1077. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8141eba4>] ? __netdev_alloc_skb+0x24/0x50
  1078. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffffa013b86c>] ? bnx2_poll_work+0x85c/0x1280 [bnx2]
  1079. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8127ddb0>] ? swiotlb_map_page+0x0/0x100
  1080. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffffa013c2cd>] ? bnx2_poll_msix+0x3d/0xc0 [bnx2]
  1081. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff813fb7e8>] ? dma_issue_pending_all+0x68/0xa0
  1082. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8142d7c3>] ? net_rx_action+0x103/0x2f0
  1083. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81094cd3>] ? hrtimer_get_next_event+0xc3/0x100
  1084. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81072991>] ? __do_softirq+0xc1/0x1e0
  1085. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff810d8b40>] ? handle_IRQ_event+0x60/0x170
  1086. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
  1087. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
  1088. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81072775>] ? irq_exit+0x85/0x90
  1089. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff814f1eb5>] ? do_IRQ+0x75/0xf0
  1090. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
  1091. Aug 23 11:12:16 row1r05s03 kernel: <EOI> [<ffffffff812c2d4e>] ? intel_idle+0xde/0x170
  1092. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff812c2d31>] ? intel_idle+0xc1/0x170
  1093. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff810979ed>] ? sched_clock_cpu+0xcd/0x110
  1094. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff813f6c67>] ? cpuidle_idle_call+0xa7/0x140
  1095. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
  1096. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff814d109a>] ? rest_init+0x7a/0x80
  1097. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
  1098. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
  1099. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
  1100. Aug 23 11:12:16 row1r05s03 kernel: swapper: page allocation failure. order:0, mode:0x20
  1101. Aug 23 11:12:16 row1r05s03 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.19.1.el6.x86_64 #1
  1102. Aug 23 11:12:16 row1r05s03 kernel: Call Trace:
  1103. Aug 23 11:12:16 row1r05s03 kernel: <IRQ> [<ffffffff811231ff>] ? __alloc_pages_nodemask+0x77f/0x940
  1104. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115d1a2>] ? kmem_getpages+0x62/0x170
  1105. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ddba>] ? fallback_alloc+0x1ba/0x270
  1106. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  1107. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115db39>] ? ____cache_alloc_node+0x99/0x160
  1108. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ed00>] ? kmem_cache_alloc_node_trace+0x90/0x200
  1109. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8115ef1d>] ? __kmalloc_node+0x4d/0x60
  1110. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8141ea1d>] ? __alloc_skb+0x6d/0x190
  1111. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8141eba4>] ? __netdev_alloc_skb+0x24/0x50
  1112. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffffa013b86c>] ? bnx2_poll_work+0x85c/0x1280 [bnx2]
  1113. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8127ddb0>] ? swiotlb_map_page+0x0/0x100
  1114. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffffa013c2cd>] ? bnx2_poll_msix+0x3d/0xc0 [bnx2]
  1115. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff813fb7e8>] ? dma_issue_pending_all+0x68/0xa0
  1116. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8142d7c3>] ? net_rx_action+0x103/0x2f0
  1117. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81094cd3>] ? hrtimer_get_next_event+0xc3/0x100
  1118. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81072991>] ? __do_softirq+0xc1/0x1e0
  1119. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff810d8b40>] ? handle_IRQ_event+0x60/0x170
  1120. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
  1121. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
  1122. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81072775>] ? irq_exit+0x85/0x90
  1123. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff814f1eb5>] ? do_IRQ+0x75/0xf0
  1124. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
  1125. Aug 23 11:12:16 row1r05s03 kernel: <EOI> [<ffffffff812c2d4e>] ? intel_idle+0xde/0x170
  1126. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff812c2d31>] ? intel_idle+0xc1/0x170
  1127. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff810979ed>] ? sched_clock_cpu+0xcd/0x110
  1128. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff813f6c67>] ? cpuidle_idle_call+0xa7/0x140
  1129. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
  1130. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff814d109a>] ? rest_init+0x7a/0x80
  1131. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
  1132. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
  1133. Aug 23 11:12:16 row1r05s03 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
  1134. Aug 23 11:12:35 row1r05s03 kernel: INFO: task kswapd0:98 blocked for more than 120 seconds.
  1135. Aug 23 11:12:35 row1r05s03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  1136. Aug 23 11:12:35 row1r05s03 kernel: kswapd0 D 0000000000000005 0 98 2 0x00000000
  1137. Aug 23 11:12:35 row1r05s03 kernel: ffff8802262ad530 0000000000000046 ffff8802262ad4f8 ffff8802262ad4f4
  1138. Aug 23 11:12:35 row1r05s03 kernel: ffff8802262ad4d0 ffff88022fc28e00 ffff88002f816680 0000000000000400
  1139. Aug 23 11:12:35 row1r05s03 kernel: ffff8802262abaf8 ffff8802262adfd8 000000000000fb88 ffff8802262abaf8
  1140. Aug 23 11:12:35 row1r05s03 kernel: Call Trace:
  1141. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff814eaac5>] schedule_timeout+0x215/0x2e0
  1142. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff810966e7>] ? down_trylock+0x37/0x50
  1143. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa026b67f>] ? xfs_buf_cond_lock+0x2f/0xc0 [xfs]
  1144. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff814eb9e2>] __down+0x72/0xb0
  1145. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa026cae2>] ? _xfs_buf_find+0x102/0x280 [xfs]
  1146. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff810967f1>] down+0x41/0x50
  1147. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa026c923>] xfs_buf_lock+0x53/0x110 [xfs]
  1148. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa026cae2>] _xfs_buf_find+0x102/0x280 [xfs]
  1149. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa026cccb>] xfs_buf_get+0x6b/0x1a0 [xfs]
  1150. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa026d33c>] xfs_buf_read+0x2c/0x100 [xfs]
  1151. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa0262f88>] xfs_trans_read_buf+0x1f8/0x400 [xfs]
  1152. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa0219086>] xfs_read_agf+0x66/0x1a0 [xfs]
  1153. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa02191f9>] xfs_alloc_read_agf+0x39/0xd0 [xfs]
  1154. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa021bc01>] xfs_alloc_fix_freelist+0x411/0x480 [xfs]
  1155. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa0260103>] ? xfs_trans_free_item_desc+0x33/0x40 [xfs]
  1156. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa0260247>] ? xfs_trans_free_items+0x87/0xb0 [xfs]
  1157. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa025ccf8>] ? xfs_perag_get+0x38/0xb0 [xfs]
  1158. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa021bd18>] xfs_free_extent+0xa8/0x130 [xfs]
  1159. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa02253bd>] xfs_bmap_finish+0x15d/0x1a0 [xfs]
  1160. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa024aaa7>] xfs_itruncate_finish+0x167/0x330 [xfs]
  1161. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa0266028>] xfs_free_eofblocks+0x1b8/0x280 [xfs]
  1162. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa02665c0>] xfs_inactive+0xc0/0x460 [xfs]
  1163. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa0276a94>] ? xfs_inode_set_reclaim_tag+0x84/0xa0 [xfs]
  1164. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa0273d80>] xfs_fs_clear_inode+0xa0/0xd0 [xfs]
  1165. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff81190c3c>] clear_inode+0xac/0x140
  1166. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff81190d10>] dispose_list+0x40/0x120
  1167. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff81191064>] shrink_icache_memory+0x274/0x2e0
  1168. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff81128c7a>] shrink_slab+0x12a/0x1a0
  1169. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff8112bb4d>] balance_pgdat+0x57d/0x7e0
  1170. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff8112c160>] ? isolate_pages_global+0x0/0x350
  1171. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff8112035d>] ? zone_watermark_ok_safe+0xad/0xc0
  1172. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff8112bee6>] kswapd+0x136/0x3b0
  1173. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff81090990>] ? autoremove_wake_function+0x0/0x40
  1174. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff8112bdb0>] ? kswapd+0x0/0x3b0
  1175. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff81090626>] kthread+0x96/0xa0
  1176. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
  1177. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff81090590>] ? kthread+0x0/0xa0
  1178. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
  1179. Aug 23 11:12:35 row1r05s03 kernel: INFO: task nfsd:2687 blocked for more than 120 seconds.
  1180. Aug 23 11:12:35 row1r05s03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  1181. Aug 23 11:12:35 row1r05s03 kernel: nfsd D 0000000000000005 0 2687 2 0x00000080
  1182. Aug 23 11:12:35 row1r05s03 kernel: ffff8802171d6960 0000000000000046 0000000000000000 0000000000000000
  1183. Aug 23 11:12:35 row1r05s03 kernel: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
  1184. Aug 23 11:12:35 row1r05s03 kernel: ffff8802277e7098 ffff8802171d7fd8 000000000000fb88 ffff8802277e7098
  1185. Aug 23 11:12:35 row1r05s03 kernel: Call Trace:
  1186. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff81132ad9>] ? zone_statistics+0x99/0xc0
  1187. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff814eaac5>] schedule_timeout+0x215/0x2e0
  1188. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff810966e7>] ? down_trylock+0x37/0x50
  1189. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa026b67f>] ? xfs_buf_cond_lock+0x2f/0xc0 [xfs]
  1190. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff814eb9e2>] __down+0x72/0xb0
  1191. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa026cae2>] ? _xfs_buf_find+0x102/0x280 [xfs]
  1192. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff810967f1>] down+0x41/0x50
  1193. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa026c923>] xfs_buf_lock+0x53/0x110 [xfs]
  1194. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa026cae2>] _xfs_buf_find+0x102/0x280 [xfs]
  1195. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa026cccb>] xfs_buf_get+0x6b/0x1a0 [xfs]
  1196. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa026d33c>] xfs_buf_read+0x2c/0x100 [xfs]
  1197. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa0262f88>] xfs_trans_read_buf+0x1f8/0x400 [xfs]
  1198. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa0219086>] xfs_read_agf+0x66/0x1a0 [xfs]
  1199. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa02191f9>] xfs_alloc_read_agf+0x39/0xd0 [xfs]
  1200. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa021bc01>] xfs_alloc_fix_freelist+0x411/0x480 [xfs]
  1201. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff8115d9bb>] ? cache_alloc_refill+0x15b/0x240
  1202. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa025ccf8>] ? xfs_perag_get+0x38/0xb0 [xfs]
  1203. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa021bd18>] xfs_free_extent+0xa8/0x130 [xfs]
  1204. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa02253bd>] xfs_bmap_finish+0x15d/0x1a0 [xfs]
  1205. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa024aaa7>] xfs_itruncate_finish+0x167/0x330 [xfs]
  1206. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa0266028>] xfs_free_eofblocks+0x1b8/0x280 [xfs]
  1207. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa02665c0>] xfs_inactive+0xc0/0x460 [xfs]
  1208. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa0276a94>] ? xfs_inode_set_reclaim_tag+0x84/0xa0 [xfs]
  1209. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa0273d80>] xfs_fs_clear_inode+0xa0/0xd0 [xfs]
  1210. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff81190c3c>] clear_inode+0xac/0x140
  1211. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff81190d10>] dispose_list+0x40/0x120
  1212. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff81191064>] shrink_icache_memory+0x274/0x2e0
  1213. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff81128c7a>] shrink_slab+0x12a/0x1a0
  1214. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff8112abff>] do_try_to_free_pages+0x2ff/0x520
  1215. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff81113e8f>] ? zone_watermark_ok+0x1f/0x30
  1216. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff8112b00d>] try_to_free_pages+0x9d/0x130
  1217. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff81131976>] ? next_online_pgdat+0x26/0x50
  1218. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff8112c160>] ? isolate_pages_global+0x0/0x350
  1219. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff81122e8d>] __alloc_pages_nodemask+0x40d/0x940
  1220. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff8115d1a2>] kmem_getpages+0x62/0x170
  1221. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff8115ddba>] fallback_alloc+0x1ba/0x270
  1222. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
  1223. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff8115db39>] ____cache_alloc_node+0x99/0x160
  1224. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff8115ed00>] kmem_cache_alloc_node_trace+0x90/0x200
  1225. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff8114967f>] vm_map_ram+0x1cf/0x470
  1226. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa026b67f>] ? xfs_buf_cond_lock+0x2f/0xc0 [xfs]
  1227. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa026bbb1>] _xfs_buf_map_pages+0xc1/0x120 [xfs]
  1228. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa026cd63>] xfs_buf_get+0x103/0x1a0 [xfs]
  1229. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa026d33c>] xfs_buf_read+0x2c/0x100 [xfs]
  1230. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa0262f88>] xfs_trans_read_buf+0x1f8/0x400 [xfs]
  1231. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa023048e>] xfs_btree_read_buf_block+0x5e/0xd0 [xfs]
  1232. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa022e8cc>] ? xfs_btree_ptr_offset+0x4c/0x90 [xfs]
  1233. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa02310c2>] xfs_btree_increment+0x202/0x2a0 [xfs]
  1234. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa021ac91>] xfs_alloc_ag_vextent_near+0x231/0xb20 [xfs]
  1235. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa021bbdd>] ? xfs_alloc_fix_freelist+0x3ed/0x480 [xfs]
  1236. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa021b655>] xfs_alloc_ag_vextent+0xd5/0x100 [xfs]
  1237. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa021c216>] xfs_alloc_vextent+0x476/0x610 [xfs]
  1238. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa0246ca3>] xfs_ialloc_ag_alloc+0x3a3/0x4b0 [xfs]
  1239. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa0246e8f>] xfs_dialloc+0xdf/0x850 [xfs]
  1240. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff8109672f>] ? up+0x2f/0x50
  1241. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa026355b>] ? xfs_trans_brelse+0xdb/0x150 [xfs]
  1242. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff81221eda>] ? symhash+0x1a/0x50
  1243. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff81221eb4>] ? symcmp+0x14/0x20
  1244. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff81221ace>] ? hashtab_search+0x7e/0x90
  1245. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa024aed0>] xfs_ialloc+0x60/0x6f0 [xfs]
  1246. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa0263b14>] xfs_dir_ialloc+0x74/0x2b0 [xfs]
  1247. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa0265b60>] xfs_create+0x440/0x640 [xfs]
  1248. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa0272b3d>] xfs_vn_mknod+0xad/0x1c0 [xfs]
  1249. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa0272c80>] xfs_vn_create+0x10/0x20 [xfs]
  1250. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff81183b04>] vfs_create+0xb4/0xe0
  1251. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa03cf6c6>] nfsd_create_v3+0x366/0x4c0 [nfsd]
  1252. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa03d7413>] nfsd3_proc_create+0x123/0x1b0 [nfsd]
  1253. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa03c843e>] nfsd_dispatch+0xfe/0x240 [nfsd]
  1254. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa02fc794>] svc_process_common+0x344/0x640 [sunrpc]
  1255. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff8105fa40>] ? default_wake_function+0x0/0x20
  1256. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa02fcdd0>] svc_process+0x110/0x160 [sunrpc]
  1257. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa03c8b62>] nfsd+0xc2/0x160 [nfsd]
  1258. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffffa03c8aa0>] ? nfsd+0x0/0x160 [nfsd]
  1259. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff81090626>] kthread+0x96/0xa0
  1260. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
  1261. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff81090590>] ? kthread+0x0/0xa0
  1262. Aug 23 11:12:35 row1r05s03 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
  1263. Aug 23 11:14:50 row1r05s03 rpc.mountd[2631]: authenticated unmount request from row1r07s32:949 for /exports/NEWTON1 (/exports)
  1264. Aug 23 11:14:50 row1r05s03 rpc.mountd[2631]: authenticated unmount request from row1r07s32:970 for /exports/IMAGE_MAIN (/exports)
  1265. Aug 23 11:17:02 row1r05s03 rpc.mountd[2631]: authenticated unmount request from sdats1:648 for /exports/NEWTON1 (/exports)
  1266. Aug 23 11:17:02 row1r05s03 rpc.mountd[2631]: authenticated unmount request from sdats1:651 for /exports/IMAGE_MAIN (/exports)
  1267. Aug 23 11:19:26 row1r05s03 rpc.mountd[2631]: authenticated unmount request from r1s18c12:886 for /exports/NEWTON1 (/exports)
  1268. Aug 23 11:22:07 row1r05s03 ntpd[2725]: synchronized to 142.58.190.1, stratum 2
  1269. Aug 23 11:24:26 row1r05s03 kernel: Slow work thread pool: Starting up
  1270. Aug 23 11:24:26 row1r05s03 kernel: Slow work thread pool: Ready
  1271. Aug 23 11:24:26 row1r05s03 kernel: FS-Cache: Loaded
  1272. Aug 23 11:24:26 row1r05s03 kernel: Registering the id_resolver key type
  1273. Aug 23 11:24:26 row1r05s03 kernel: FS-Cache: Netfs 'nfs' registered for caching
  1274. Aug 23 11:26:32 row1r05s03 xinetd[2717]: START: shell pid=7125 from=199.60.1.59
  1275. Aug 23 11:27:38 row1r05s03 ntpd[2725]: synchronized to 142.58.103.1, stratum 2
  1276. Aug 23 11:30:14 row1r05s03 init: tty (/dev/tty2) main process (3637) killed by TERM signal
  1277. Aug 23 11:30:14 row1r05s03 init: tty (/dev/tty3) main process (3639) killed by TERM signal
  1278. Aug 23 11:30:14 row1r05s03 init: tty (/dev/tty4) main process (3641) killed by TERM signal
  1279. Aug 23 11:30:14 row1r05s03 init: tty (/dev/tty5) main process (3643) killed by TERM signal
  1280. Aug 23 11:30:14 row1r05s03 init: tty (/dev/tty6) main process (3645) killed by TERM signal
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement