API tools faq
paste
Advertisement
Guest User

Untitled

a guest
Feb 3rd, 2020
360
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 28.63 KB | None | 0 0
raw download clone embed print report
  1. Jan 30 16:16:35 hpc-be028.cern.ch kernel: kworker/1:36: page allocation failure: order:5, mode:0x104050
  2. Jan 30 16:16:35 hpc-be028.cern.ch kernel: CPU: 1 PID: 78445 Comm: kworker/1:36 Kdump: loaded Tainted: P OE ------------ T 3.10.0-957.27.2.el7.x86_64 #1
  3. Jan 30 16:16:35 hpc-be028.cern.ch kernel: Hardware name: Intel Corporation S2600KPR/S2600KPR, BIOS SE5C610.86B.01.01.0020.122820161512 12/28/2016
  4. Jan 30 16:16:35 hpc-be028.cern.ch kernel: Workqueue: ceph-msgr ceph_con_workfn [libceph]
  5. Jan 30 16:16:35 hpc-be028.cern.ch kernel: Call Trace:
  6. Jan 30 16:16:35 hpc-be028.cern.ch kernel: [<ffffffffb6164147>] dump_stack+0x19/0x1b
  7. Jan 30 16:16:35 hpc-be028.cern.ch kernel: [<ffffffffb5bbdec0>] warn_alloc_failed+0x110/0x180
  8. Jan 30 16:16:35 hpc-be028.cern.ch kernel: [<ffffffffb615f74e>] __alloc_pages_slowpath+0x6b6/0x724
  9. Jan 30 16:16:35 hpc-be028.cern.ch kernel: [<ffffffffb5bc2524>] __alloc_pages_nodemask+0x404/0x420
  10. Jan 30 16:16:35 hpc-be028.cern.ch kernel: [<ffffffffb5c0f438>] alloc_pages_current+0x98/0x110
  11. Jan 30 16:16:35 hpc-be028.cern.ch kernel: [<ffffffffb5bbcc6e>] __get_free_pages+0xe/0x40
  12. Jan 30 16:16:35 hpc-be028.cern.ch kernel: [<ffffffffb5c1ab4e>] kmalloc_order_trace+0x2e/0xa0
  13. Jan 30 16:16:36 hpc-be028.cern.ch kernel: [<ffffffffb5c208d1>] __kmalloc_track_caller+0x221/0x240
  14. Jan 30 16:16:36 hpc-be028.cern.ch kernel: [<ffffffffc0d101e6>] ? osdmap_set_max_osd+0x76/0x1d0 [libceph]
  15. Jan 30 16:16:36 hpc-be028.cern.ch kernel: [<ffffffffb5bd73ef>] krealloc+0x4f/0xa0
  16. Jan 30 16:16:36 hpc-be028.cern.ch kernel: [<ffffffffc0d101e6>] osdmap_set_max_osd+0x76/0x1d0 [libceph]
  17. Jan 30 16:16:36 hpc-be028.cern.ch kernel: [<ffffffffc0d134d5>] ceph_osdmap_decode+0x195/0x860 [libceph]
  18. Jan 30 16:16:36 hpc-be028.cern.ch kernel: [<ffffffffc0d09c34>] handle_one_map+0x224/0x250 [libceph]
  19. Jan 30 16:16:36 hpc-be028.cern.ch kernel: [<ffffffffc0d0e69c>] ceph_osdc_handle_map+0x7dc/0x8c0 [libceph]
  20. Jan 30 16:16:36 hpc-be028.cern.ch kernel: [<ffffffffc0d0ea81>] dispatch+0x301/0xca0 [libceph]
  21. Jan 30 16:16:36 hpc-be028.cern.ch kernel: [<ffffffffc0cfcfb4>] try_read+0x514/0x12c0 [libceph]
  22. Jan 30 16:16:36 hpc-be028.cern.ch kernel: [<ffffffffb5addd9e>] ? account_entity_dequeue+0xae/0xd0
  23. Jan 30 16:16:36 hpc-be028.cern.ch kernel: [<ffffffffb5ae192c>] ? dequeue_entity+0x11c/0x5e0
  24. Jan 30 16:16:36 hpc-be028.cern.ch kernel: [<ffffffffb601b417>] ? kernel_sendmsg+0x37/0x50
  25. Jan 30 16:16:36 hpc-be028.cern.ch kernel: [<ffffffffc0cfdf64>] ceph_con_workfn+0xe4/0x1530 [libceph]
  26. Jan 30 16:16:36 hpc-be028.cern.ch kernel: [<ffffffffb6169aba>] ? __schedule+0x42a/0x860
  27. Jan 30 16:16:36 hpc-be028.cern.ch kernel: [<ffffffffb5abaf9f>] process_one_work+0x17f/0x440
  28. Jan 30 16:16:36 hpc-be028.cern.ch kernel: [<ffffffffb5abc036>] worker_thread+0x126/0x3c0
  29. Jan 30 16:16:36 hpc-be028.cern.ch kernel: [<ffffffffb5abbf10>] ? manage_workers.isra.25+0x2a0/0x2a0
  30. Jan 30 16:16:36 hpc-be028.cern.ch kernel: [<ffffffffb5ac2e81>] kthread+0xd1/0xe0
  31. Jan 30 16:16:36 hpc-be028.cern.ch kernel: [<ffffffffb5ac2db0>] ? insert_kthread_work+0x40/0x40
  32. Jan 30 16:16:36 hpc-be028.cern.ch kernel: [<ffffffffb6176c1d>] ret_from_fork_nospec_begin+0x7/0x21
  33. Jan 30 16:16:36 hpc-be028.cern.ch kernel: [<ffffffffb5ac2db0>] ? insert_kthread_work+0x40/0x40
  34. Jan 30 16:16:36 hpc-be028.cern.ch kernel: Mem-Info:
  35. Jan 30 16:16:36 hpc-be028.cern.ch kernel: active_anon:17199252 inactive_anon:1384796 isolated_anon:0
  36. active_file:6645205 inactive_file:6831015 isolated_file:64
  37. unevictable:592 dirty:2964882 writeback:201597 unstable:0
  38. slab_reclaimable:163441 slab_unreclaimable:80435
  39. mapped:61869 shmem:46608 pagetables:48720 bounce:0
  40. free:249748 free_pcp:1338 free_cma:0
  41. Jan 30 16:16:36 hpc-be028.cern.ch kernel: Node 0 DMA free:15892kB min:60kB low:72kB high:88kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0k
  42. Jan 30 16:16:36 hpc-be028.cern.ch kernel: lowmem_reserve[]: 0 1705 64136 64136
  43. Jan 30 16:16:36 hpc-be028.cern.ch kernel: Node 0 DMA32 free:256916kB min:6948kB low:8684kB high:10420kB active_anon:1059064kB inactive_anon:360964kB active_file:144kB inactive_file:6772kB unevictable:0kB isolated(a
  44. Jan 30 16:16:36 hpc-be028.cern.ch kernel: lowmem_reserve[]: 0 0 62430 62430
  45. Jan 30 16:16:36 hpc-be028.cern.ch kernel: Node 0 Normal free:686628kB min:254432kB low:318040kB high:381648kB active_anon:33449040kB inactive_anon:2586528kB active_file:12803548kB inactive_file:13007272kB unevictab
  46. Jan 30 16:16:36 hpc-be028.cern.ch kernel: lowmem_reserve[]: 0 0 0 0
  47. Jan 30 16:16:36 hpc-be028.cern.ch kernel: Node 1 Normal free:461520kB min:262840kB low:328548kB high:394260kB active_anon:34316120kB inactive_anon:2591692kB active_file:13766564kB inactive_file:13852284kB unevictab
  48. Jan 30 16:16:36 hpc-be028.cern.ch kernel: lowmem_reserve[]: 0 0 0 0
  49. Jan 30 16:16:36 hpc-be028.cern.ch kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB
  50. Jan 30 16:16:36 hpc-be028.cern.ch kernel: Node 0 DMA32: 1213*4kB (UEM) 812*8kB (UEM) 420*16kB (UEM) 83*32kB (UEM) 233*64kB (UEM) 168*128kB (UEM) 122*256kB (UEM) 100*512kB (UEM) 115*1024kB (UEM) 0*2048kB 0*4096kB =
  51. Jan 30 16:16:36 hpc-be028.cern.ch kernel: Node 0 Normal: 96701*4kB (UEM) 34788*8kB (UEM) 4239*16kB (UEM) 57*32kB (UEM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 734756kB
  52. Jan 30 16:16:36 hpc-be028.cern.ch kernel: Node 1 Normal: 81934*4kB (UEM) 34559*8kB (UEM) 1082*16kB (UEM) 97*32kB (UE) 3*64kB (UE) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 624816kB
  53. Jan 30 16:16:36 hpc-be028.cern.ch kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
  54. Jan 30 16:16:36 hpc-be028.cern.ch kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
  55. Jan 30 16:16:36 hpc-be028.cern.ch kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
  56. Jan 30 16:16:36 hpc-be028.cern.ch kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
  57. Jan 30 16:16:36 hpc-be028.cern.ch kernel: 13386727 total pagecache pages
  58. Jan 30 16:16:36 hpc-be028.cern.ch kernel: 35711 pages in swap cache
  59. Jan 30 16:16:36 hpc-be028.cern.ch kernel: Swap cache stats: add 6414255, delete 6385221, find 2627543/2678239
  60. Jan 30 16:16:36 hpc-be028.cern.ch kernel: Free swap = 65962236kB
  61. Jan 30 16:16:36 hpc-be028.cern.ch kernel: Total swap = 67108860kB
  62. Jan 30 16:16:36 hpc-be028.cern.ch kernel: 33530063 pages RAM
  63. Jan 30 16:16:36 hpc-be028.cern.ch kernel: 0 pages HighMem/MovableOnly
  64. Jan 30 16:16:36 hpc-be028.cern.ch kernel: 595264 pages reserved
  65. Jan 30 16:16:36 hpc-be028.cern.ch kernel: libceph: corrupt full osdmap (-12) epoch 676797 off 895 (ffffbcb316713d77 of ffffbcb3167139f8-ffffbcb316781502)
  66. Jan 30 16:16:36 hpc-be028.cern.ch kernel: osdmap: 00000000: ...
  67. ...
  68.  
  69. then nothing relevant until 1 day later when the client got stuck.
  70.  
  71. Now today the same node got stuck again:
  72.  
  73. Feb 02 17:03:12 hpc-be028.cern.ch kernel: ceph: mds0 caps stale
  74. Feb 02 17:05:16 hpc-be028.cern.ch kernel: INFO: task cp:169181 blocked for more than 120 seconds.
  75. Feb 02 17:05:16 hpc-be028.cern.ch kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  76. Feb 02 17:05:16 hpc-be028.cern.ch kernel: cp D ffff9a71612e30c0 0 169181 1 0x00000086
  77. Feb 02 17:05:16 hpc-be028.cern.ch kernel: Call Trace:
  78. Feb 02 17:05:16 hpc-be028.cern.ch kernel: [<ffffffffc0cfc7fa>] ? ceph_con_send+0xba/0x1c0 [libceph]
  79. Feb 02 17:05:16 hpc-be028.cern.ch kernel: [<ffffffffc0cf9416>] ? ceph_kvmalloc+0x26/0x60 [libceph]
  80. Feb 02 17:05:16 hpc-be028.cern.ch kernel: [<ffffffffb6168060>] ? bit_wait+0x50/0x50
  81. Feb 02 17:05:16 hpc-be028.cern.ch kernel: [<ffffffffb6169f19>] schedule+0x29/0x70
  82. Feb 02 17:05:16 hpc-be028.cern.ch kernel: [<ffffffffb6167a21>] schedule_timeout+0x221/0x2d0
  83. Feb 02 17:05:16 hpc-be028.cern.ch kernel: [<ffffffffc0ee522e>] ? send_cap_msg+0x28e/0x470 [ceph]
  84. Feb 02 17:05:16 hpc-be028.cern.ch kernel: [<ffffffffb5b02372>] ? ktime_get_ts64+0x52/0xf0
  85. Feb 02 17:05:16 hpc-be028.cern.ch kernel: [<ffffffffb6168060>] ? bit_wait+0x50/0x50
  86. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb61695ed>] io_schedule_timeout+0xad/0x130
  87. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb6169688>] io_schedule+0x18/0x20
  88. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb6168071>] bit_wait_io+0x11/0x50
  89. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb6167c21>] __wait_on_bit_lock+0x61/0xc0
  90. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5bba6de>] ? __find_get_pages+0x11e/0x1c0
  91. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5bb6f64>] __lock_page+0x74/0x90
  92. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5ac4010>] ? wake_bit_function+0x40/0x40
  93. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5bc859c>] truncate_inode_pages_range+0x6cc/0x700
  94. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffc0ee6c82>] ? __ceph_caps_issued+0x82/0xe0 [ceph]
  95. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5c70eca>] ? __inode_wait_for_writeback+0x7a/0xf0
  96. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5ac4010>] ? wake_bit_function+0x40/0x40
  97. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5bc863f>] truncate_inode_pages_final+0x4f/0x60
  98. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5c5ffdc>] evict+0x16c/0x180
  99. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5c6082c>] iput+0xfc/0x190
  100. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5c5b1b0>] __dentry_kill+0x120/0x180
  101. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5c5b2c0>] dput+0xb0/0x160
  102. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5c44ace>] __fput+0x17e/0x260
  103. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5c44c9e>] ____fput+0xe/0x10
  104. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5abf9eb>] task_work_run+0xbb/0xe0
  105. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5a9eeb1>] do_exit+0x2d1/0xa40
  106. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5a9f69f>] do_group_exit+0x3f/0xa0
  107. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5ab049e>] get_signal_to_deliver+0x1ce/0x5e0
  108. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5a2b527>] do_signal+0x57/0x6f0
  109. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5a2bc32>] do_notify_resume+0x72/0xc0
  110. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb6177134>] int_signal+0x12/0x17
  111. Feb 02 17:05:17 hpc-be028.cern.ch kernel: INFO: task kworker/5:2:300661 blocked for more than 120 seconds.
  112. Feb 02 17:05:17 hpc-be028.cern.ch kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  113. Feb 02 17:05:17 hpc-be028.cern.ch kernel: kworker/5:2 D ffff9a7149f22080 0 300661 2 0x00000080
  114. Feb 02 17:05:17 hpc-be028.cern.ch kernel: Workqueue: ceph-msgr ceph_con_workfn [libceph]
  115. Feb 02 17:05:17 hpc-be028.cern.ch kernel: Call Trace:
  116. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffc0ee45f0>] ? ceph_fh_to_parent+0x70/0x70 [ceph]
  117. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb6169f19>] schedule+0x29/0x70
  118. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5c5f0c0>] __wait_on_freeing_inode+0xb0/0xf0
  119. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5ac4010>] ? wake_bit_function+0x40/0x40
  120. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5c5f199>] find_inode+0x99/0xc0
  121. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffc0ee45f0>] ? ceph_fh_to_parent+0x70/0x70 [ceph]
  122. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5c5f231>] ilookup5_nowait+0x71/0x90
  123. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5c6004f>] ilookup5+0xf/0x60
  124. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffc0eebb8a>] ceph_handle_caps+0x26a/0x1a00 [ceph]
  125. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffc0ef28b0>] ? mds_check_message_signature+0x30/0x40 [ceph]
  126. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffc0efadfa>] dispatch+0x7fa/0xb00 [ceph]
  127. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb601b56a>] ? kernel_recvmsg+0x3a/0x50
  128. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffc0cfcfb4>] try_read+0x514/0x12c0 [libceph]
  129. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5addd9e>] ? account_entity_dequeue+0xae/0xd0
  130. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5ae192c>] ? dequeue_entity+0x11c/0x5e0
  131. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffc0cfdf64>] ceph_con_workfn+0xe4/0x1530 [libceph]
  132. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb6169aba>] ? __schedule+0x42a/0x860
  133. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5abaf9f>] process_one_work+0x17f/0x440
  134. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5abc036>] worker_thread+0x126/0x3c0
  135. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5abbf10>] ? manage_workers.isra.25+0x2a0/0x2a0
  136. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5ac2e81>] kthread+0xd1/0xe0
  137. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5ac2db0>] ? insert_kthread_work+0x40/0x40
  138. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb6176c1d>] ret_from_fork_nospec_begin+0x7/0x21
  139. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5ac2db0>] ? insert_kthread_work+0x40/0x40
  140. Feb 02 17:07:17 hpc-be028.cern.ch kernel: INFO: task cp:169181 blocked for more than 120 seconds.
  141. Feb 02 17:07:17 hpc-be028.cern.ch kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  142. Feb 02 17:07:17 hpc-be028.cern.ch kernel: cp D ffff9a71612e30c0 0 169181 1 0x00000086
  143. Feb 02 17:07:17 hpc-be028.cern.ch kernel: Call Trace:
  144. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffc0cfc7fa>] ? ceph_con_send+0xba/0x1c0 [libceph]
  145. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffc0cf9416>] ? ceph_kvmalloc+0x26/0x60 [libceph]
  146. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb6168060>] ? bit_wait+0x50/0x50
  147. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb6169f19>] schedule+0x29/0x70
  148. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb6167a21>] schedule_timeout+0x221/0x2d0
  149. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffc0ee522e>] ? send_cap_msg+0x28e/0x470 [ceph]
  150. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb5b02372>] ? ktime_get_ts64+0x52/0xf0
  151. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb6168060>] ? bit_wait+0x50/0x50
  152. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb61695ed>] io_schedule_timeout+0xad/0x130
  153. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb6169688>] io_schedule+0x18/0x20
  154. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb6168071>] bit_wait_io+0x11/0x50
  155. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb6167c21>] __wait_on_bit_lock+0x61/0xc0
  156. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb5bba6de>] ? __find_get_pages+0x11e/0x1c0
  157. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb5bb6f64>] __lock_page+0x74/0x90
  158. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb5ac4010>] ? wake_bit_function+0x40/0x40
  159. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb5bc859c>] truncate_inode_pages_range+0x6cc/0x700
  160. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffc0ee6c82>] ? __ceph_caps_issued+0x82/0xe0 [ceph]
  161. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb5c70eca>] ? __inode_wait_for_writeback+0x7a/0xf0
  162. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb5ac4010>] ? wake_bit_function+0x40/0x40
  163. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5bc863f>] truncate_inode_pages_final+0x4f/0x60
  164. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5c5ffdc>] evict+0x16c/0x180
  165. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5c6082c>] iput+0xfc/0x190
  166. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5c5b1b0>] __dentry_kill+0x120/0x180
  167. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5c5b2c0>] dput+0xb0/0x160
  168. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5c44ace>] __fput+0x17e/0x260
  169. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5c44c9e>] ____fput+0xe/0x10
  170. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5abf9eb>] task_work_run+0xbb/0xe0
  171. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5a9eeb1>] do_exit+0x2d1/0xa40
  172. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5a9f69f>] do_group_exit+0x3f/0xa0
  173. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5ab049e>] get_signal_to_deliver+0x1ce/0x5e0
  174. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5a2b527>] do_signal+0x57/0x6f0
  175. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5a2bc32>] do_notify_resume+0x72/0xc0
  176. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb6177134>] int_signal+0x12/0x17
  177. Feb 02 17:05:17 hpc-be028.cern.ch kernel: INFO: task kworker/5:2:300661 blocked for more than 120 seconds.
  178. Feb 02 17:05:17 hpc-be028.cern.ch kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  179. Feb 02 17:05:17 hpc-be028.cern.ch kernel: kworker/5:2 D ffff9a7149f22080 0 300661 2 0x00000080
  180. Feb 02 17:05:17 hpc-be028.cern.ch kernel: Workqueue: ceph-msgr ceph_con_workfn [libceph]
  181. Feb 02 17:05:17 hpc-be028.cern.ch kernel: Call Trace:
  182. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffc0ee45f0>] ? ceph_fh_to_parent+0x70/0x70 [ceph]
  183. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb6169f19>] schedule+0x29/0x70
  184. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5c5f0c0>] __wait_on_freeing_inode+0xb0/0xf0
  185. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5ac4010>] ? wake_bit_function+0x40/0x40
  186. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5c5f199>] find_inode+0x99/0xc0
  187. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffc0ee45f0>] ? ceph_fh_to_parent+0x70/0x70 [ceph]
  188. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5c5f231>] ilookup5_nowait+0x71/0x90
  189. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5c6004f>] ilookup5+0xf/0x60
  190. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffc0eebb8a>] ceph_handle_caps+0x26a/0x1a00 [ceph]
  191. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffc0ef28b0>] ? mds_check_message_signature+0x30/0x40 [ceph]
  192. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffc0efadfa>] dispatch+0x7fa/0xb00 [ceph]
  193. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb601b56a>] ? kernel_recvmsg+0x3a/0x50
  194. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffc0cfcfb4>] try_read+0x514/0x12c0 [libceph]
  195. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5addd9e>] ? account_entity_dequeue+0xae/0xd0
  196. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5ae192c>] ? dequeue_entity+0x11c/0x5e0
  197. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffc0cfdf64>] ceph_con_workfn+0xe4/0x1530 [libceph]
  198. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb6169aba>] ? __schedule+0x42a/0x860
  199. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5abaf9f>] process_one_work+0x17f/0x440
  200. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5abc036>] worker_thread+0x126/0x3c0
  201. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5abbf10>] ? manage_workers.isra.25+0x2a0/0x2a0
  202. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5ac2e81>] kthread+0xd1/0xe0
  203. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5ac2db0>] ? insert_kthread_work+0x40/0x40
  204. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb6176c1d>] ret_from_fork_nospec_begin+0x7/0x21
  205. Feb 02 17:05:17 hpc-be028.cern.ch kernel: [<ffffffffb5ac2db0>] ? insert_kthread_work+0x40/0x40
  206. Feb 02 17:07:17 hpc-be028.cern.ch kernel: INFO: task cp:169181 blocked for more than 120 seconds.
  207. Feb 02 17:07:17 hpc-be028.cern.ch kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  208. Feb 02 17:07:17 hpc-be028.cern.ch kernel: cp D ffff9a71612e30c0 0 169181 1 0x00000086
  209. Feb 02 17:07:17 hpc-be028.cern.ch kernel: Call Trace:
  210. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffc0cfc7fa>] ? ceph_con_send+0xba/0x1c0 [libceph]
  211. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffc0cf9416>] ? ceph_kvmalloc+0x26/0x60 [libceph]
  212. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb6168060>] ? bit_wait+0x50/0x50
  213. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb6169f19>] schedule+0x29/0x70
  214. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb6167a21>] schedule_timeout+0x221/0x2d0
  215. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffc0ee522e>] ? send_cap_msg+0x28e/0x470 [ceph]
  216. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb5b02372>] ? ktime_get_ts64+0x52/0xf0
  217. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb6168060>] ? bit_wait+0x50/0x50
  218. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb61695ed>] io_schedule_timeout+0xad/0x130
  219. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb6169688>] io_schedule+0x18/0x20
  220. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb6168071>] bit_wait_io+0x11/0x50
  221. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb6167c21>] __wait_on_bit_lock+0x61/0xc0
  222. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb5bba6de>] ? __find_get_pages+0x11e/0x1c0
  223. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb5bb6f64>] __lock_page+0x74/0x90
  224. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb5ac4010>] ? wake_bit_function+0x40/0x40
  225. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb5bc859c>] truncate_inode_pages_range+0x6cc/0x700
  226. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffc0ee6c82>] ? __ceph_caps_issued+0x82/0xe0 [ceph]
  227. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb5c70eca>] ? __inode_wait_for_writeback+0x7a/0xf0
  228. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb5ac4010>] ? wake_bit_function+0x40/0x40
  229. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb5bc863f>] truncate_inode_pages_final+0x4f/0x60
  230. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb5c5ffdc>] evict+0x16c/0x180
  231. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb5c6082c>] iput+0xfc/0x190
  232. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb5c5b1b0>] __dentry_kill+0x120/0x180
  233. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb5c5b2c0>] dput+0xb0/0x160
  234. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb5c44ace>] __fput+0x17e/0x260
  235. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb5c44c9e>] ____fput+0xe/0x10
  236. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb5abf9eb>] task_work_run+0xbb/0xe0
  237. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb5a9eeb1>] do_exit+0x2d1/0xa40
  238. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb5a9f69f>] do_group_exit+0x3f/0xa0
  239. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb5ab049e>] get_signal_to_deliver+0x1ce/0x5e0
  240. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb5a2b527>] do_signal+0x57/0x6f0
  241. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb5a2bc32>] do_notify_resume+0x72/0xc0
  242. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb6177134>] int_signal+0x12/0x17
  243. Feb 02 17:07:17 hpc-be028.cern.ch kernel: INFO: task kworker/5:2:300661 blocked for more than 120 seconds.
  244. Feb 02 17:07:17 hpc-be028.cern.ch kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  245. Feb 02 17:07:17 hpc-be028.cern.ch kernel: kworker/5:2 D ffff9a7149f22080 0 300661 2 0x00000080
  246. Feb 02 17:07:17 hpc-be028.cern.ch kernel: Workqueue: ceph-msgr ceph_con_workfn [libceph]
  247. Feb 02 17:07:17 hpc-be028.cern.ch kernel: Call Trace:
  248. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffc0ee45f0>] ? ceph_fh_to_parent+0x70/0x70 [ceph]
  249. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb6169f19>] schedule+0x29/0x70
  250. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb5c5f0c0>] __wait_on_freeing_inode+0xb0/0xf0
  251. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb5ac4010>] ? wake_bit_function+0x40/0x40
  252. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb5c5f199>] find_inode+0x99/0xc0
  253. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffc0ee45f0>] ? ceph_fh_to_parent+0x70/0x70 [ceph]
  254. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb5c5f231>] ilookup5_nowait+0x71/0x90
  255. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb5c6004f>] ilookup5+0xf/0x60
  256. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffc0eebb8a>] ceph_handle_caps+0x26a/0x1a00 [ceph]
  257. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffc0ef28b0>] ? mds_check_message_signature+0x30/0x40 [ceph]
  258. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffc0efadfa>] dispatch+0x7fa/0xb00 [ceph]
  259. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb601b56a>] ? kernel_recvmsg+0x3a/0x50
  260. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffc0cfcfb4>] try_read+0x514/0x12c0 [libceph]
  261. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb5addd9e>] ? account_entity_dequeue+0xae/0xd0
  262. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb5ae192c>] ? dequeue_entity+0x11c/0x5e0
  263. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffc0cfdf64>] ceph_con_workfn+0xe4/0x1530 [libceph]
  264. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb6169aba>] ? __schedule+0x42a/0x860
  265. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb5abaf9f>] process_one_work+0x17f/0x440
  266. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb5abc036>] worker_thread+0x126/0x3c0
  267. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb5abbf10>] ? manage_workers.isra.25+0x2a0/0x2a0
  268. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb5ac2e81>] kthread+0xd1/0xe0
  269. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb5ac2db0>] ? insert_kthread_work+0x40/0x40
  270. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb6176c1d>] ret_from_fork_nospec_begin+0x7/0x21
  271. Feb 02 17:07:17 hpc-be028.cern.ch kernel: [<ffffffffb5ac2db0>] ? insert_kthread_work+0x40/0x40
  272. Feb 02 17:07:17 hpc-be028.cern.ch kernel: ceph: mds0 hung
  273. Feb 02 17:09:17 hpc-be028.cern.ch kernel: INFO: task cp:169181 blocked for more than 120 seconds.
  274. Feb 02 17:09:17 hpc-be028.cern.ch kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  275. Feb 02 17:09:17 hpc-be028.cern.ch kernel: cp D ffff9a71612e30c0 0 169181 1 0x00000086
  276. Feb 02 17:09:17 hpc-be028.cern.ch kernel: Call Trace:
  277. Feb 02 17:09:17 hpc-be028.cern.ch kernel: [<ffffffffc0cfc7fa>] ? ceph_con_send+0xba/0x1c0 [libceph]
  278. Feb 02 17:09:17 hpc-be028.cern.ch kernel: [<ffffffffc0cf9416>] ? ceph_kvmalloc+0x26/0x60 [libceph]
  279. Feb 02 17:09:17 hpc-be028.cern.ch kernel: [<ffffffffb6168060>] ? bit_wait+0x50/0x50
  280. Feb 02 17:09:17 hpc-be028.cern.ch kernel: [<ffffffffb6169f19>] schedule+0x29/0x70
  281. Feb 02 17:09:17 hpc-be028.cern.ch kernel: [<ffffffffb6167a21>] schedule_timeout+0x221/0x2d0
  282. Feb 02 17:09:17 hpc-be028.cern.ch kernel: [<ffffffffc0ee522e>] ? send_cap_msg+0x28e/0x470 [ceph]
  283. Feb 02 17:09:17 hpc-be028.cern.ch kernel: [<ffffffffb5b02372>] ? ktime_get_ts64+0x52/0xf0
  284. Feb 02 17:09:17 hpc-be028.cern.ch kernel: [<ffffffffb6168060>] ? bit_wait+0x50/0x50
  285. Feb 02 17:09:17 hpc-be028.cern.ch kernel: [<ffffffffb61695ed>] io_schedule_timeout+0xad/0x130
  286. Feb 02 17:09:18 hpc-be028.cern.ch kernel: [<ffffffffb6169688>] io_schedule+0x18/0x20
  287. Feb 02 17:09:18 hpc-be028.cern.ch kernel: [<ffffffffb6168071>] bit_wait_io+0x11/0x50
  288. Feb 02 17:09:18 hpc-be028.cern.ch kernel: [<ffffffffb6167c21>] __wait_on_bit_lock+0x61/0xc0
  289. Feb 02 17:09:18 hpc-be028.cern.ch kernel: [<ffffffffb5bba6de>] ? __find_get_pages+0x11e/0x1c0
  290. Feb 02 17:09:18 hpc-be028.cern.ch kernel: [<ffffffffb5bb6f64>] __lock_page+0x74/0x90
  291. Feb 02 17:09:18 hpc-be028.cern.ch kernel: [<ffffffffb5ac4010>] ? wake_bit_function+0x40/0x40
  292. Feb 02 17:09:18 hpc-be028.cern.ch kernel: [<ffffffffb5bc859c>] truncate_inode_pages_range+0x6cc/0x700
  293. Feb 02 17:09:18 hpc-be028.cern.ch kernel: [<ffffffffc0ee6c82>] ? __ceph_caps_issued+0x82/0xe0 [ceph]
  294. Feb 02 17:09:18 hpc-be028.cern.ch kernel: [<ffffffffb5c70eca>] ? __inode_wait_for_writeback+0x7a/0xf0
  295. Feb 02 17:09:18 hpc-be028.cern.ch kernel: [<ffffffffb5ac4010>] ? wake_bit_function+0x40/0x40
  296. Feb 02 17:09:18 hpc-be028.cern.ch kernel: [<ffffffffb5bc863f>] truncate_inode_pages_final+0x4f/0x60
  297. Feb 02 17:09:18 hpc-be028.cern.ch kernel: [<ffffffffb5c5ffdc>] evict+0x16c/0x180
  298. Feb 02 17:09:18 hpc-be028.cern.ch kernel: [<ffffffffb5c6082c>] iput+0xfc/0x190
  299. Feb 02 17:09:18 hpc-be028.cern.ch kernel: [<ffffffffb5c5b1b0>] __dentry_kill+0x120/0x180
  300. Feb 02 17:09:18 hpc-be028.cern.ch kernel: [<ffffffffb5c5b2c0>] dput+0xb0/0x160
  301. Feb 02 17:09:18 hpc-be028.cern.ch kernel: [<ffffffffb5c44ace>] __fput+0x17e/0x260
  302. Feb 02 17:09:18 hpc-be028.cern.ch kernel: [<ffffffffb5c44c9e>] ____fput+0xe/0x10
  303. Feb 02 17:09:18 hpc-be028.cern.ch kernel: [<ffffffffb5abf9eb>] task_work_run+0xbb/0xe0
  304. Feb 02 17:09:18 hpc-be028.cern.ch kernel: [<ffffffffb5a9eeb1>] do_exit+0x2d1/0xa40
  305. Feb 02 17:09:18 hpc-be028.cern.ch kernel: [<ffffffffb5a9f69f>] do_group_exit+0x3f/0xa0
  306. Feb 02 17:09:18 hpc-be028.cern.ch kernel: [<ffffffffb5ab049e>] get_signal_to_deliver+0x1ce/0x5e0
  307. Feb 02 17:09:18 hpc-be028.cern.ch kernel: [<ffffffffb5a2b527>] do_signal+0x57/0x6f0
  308. Feb 02 17:09:18 hpc-be028.cern.ch kernel: [<ffffffffb5a2bc32>] do_notify_resume+0x72/0xc0
  309. Feb 02 17:09:18 hpc-be028.cern.ch kernel: [<ffffffffb6177134>] int_signal+0x12/0x17
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!