Advertisement
Guest User

[CASE-16] Linux DRBD log2

a guest
Apr 11th, 2016
171
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 29.80 KB | None | 0 0
  1. Apr 11 10:13:46 drbd9-02 kernel: drbd r1/0 drbd1 drbd9-03: repl( Ahead -> SyncSource )
  2. Apr 11 10:13:46 drbd9-02 kernel: drbd r1/0 drbd1 drbd9-03: Began resync as SyncSource (will sync 345288 KB [86322 bits set
  3. Apr 11 10:13:46 drbd9-02 kernel: drbd r1/0 drbd1: Congestion-fill threshold reached
  4. Apr 11 10:13:46 drbd9-02 kernel: drbd r1/0 drbd1 drbd9-03: repl( SyncSource -> Ahead )
  5. Apr 11 10:13:47 drbd9-02 kernel: drbd r1/0 drbd1 drbd9-03: helper command: /sbin/drbdadm before-resync-source
  6. Apr 11 10:13:47 drbd9-02 kernel: drbd r1/0 drbd1 drbd9-03: helper command: /sbin/drbdadm before-resync-source exit code 0 (0x0)
  7. Apr 11 10:13:47 drbd9-02 kernel: drbd r1/0 drbd1 drbd9-03: repl( Ahead -> SyncSource )
  8. Apr 11 10:13:47 drbd9-02 kernel: drbd r1/0 drbd1 drbd9-03: Began resync as SyncSource (will sync 358536 KB [89634 bits set]).
  9. Apr 11 10:13:47 drbd9-02 kernel: drbd r1/0 drbd1: Congestion-fill threshold reached
  10. Apr 11 10:13:47 drbd9-02 kernel: drbd r1/0 drbd1 drbd9-03: repl( SyncSource -> Ahead )
  11. Apr 11 10:13:48 drbd9-02 kernel: drbd r1/0 drbd1 drbd9-03: helper command: /sbin/drbdadm before-resync-source
  12. Apr 11 10:13:48 drbd9-02 kernel: drbd r1/0 drbd1 drbd9-03: helper command: /sbin/drbdadm before-resync-source exit code 0 (0x0)
  13. Apr 11 10:13:48 drbd9-02 kernel: drbd r1/0 drbd1 drbd9-03: repl( Ahead -> SyncSource )
  14. Apr 11 10:13:48 drbd9-02 kernel: drbd r1/0 drbd1 drbd9-03: Began resync as SyncSource (will sync 602836 KB [150709 bits set]).
  15. Apr 11 10:13:48 drbd9-02 kernel: drbd r1/0 drbd1: Congestion-fill threshold reached
  16. Apr 11 10:13:48 drbd9-02 kernel: drbd r1/0 drbd1 drbd9-03: repl( SyncSource -> Ahead )
  17. Apr 11 10:13:49 drbd9-02 kernel: drbd r1/0 drbd1 drbd9-03: helper command: /sbin/drbdadm before-resync-source
  18. Apr 11 10:13:49 drbd9-02 kernel: drbd r1/0 drbd1 drbd9-03: helper command: /sbin/drbdadm before-resync-source exit code 0 (0x0)
  19. Apr 11 10:13:49 drbd9-02 kernel: drbd r1/0 drbd1 drbd9-03: repl( Ahead -> SyncSource )
  20. Apr 11 10:13:49 drbd9-02 kernel: drbd r1/0 drbd1 drbd9-03: Began resync as SyncSource (will sync 715444 KB [178861 bits set]).
  21. Apr 11 10:13:50 drbd9-02 kernel: drbd r1/0 drbd1: Congestion-fill threshold reached
  22. Apr 11 10:13:50 drbd9-02 kernel: drbd r1/0 drbd1 drbd9-03: repl( SyncSource -> Ahead ) //-----------------> VM pause
  23. Apr 11 10:14:45 drbd9-02 kernel: drbd r1 drbd9-03: meta connection shut down by peer.
  24. Apr 11 10:14:45 drbd9-02 kernel: drbd r1 drbd9-03: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  25. Apr 11 10:14:45 drbd9-02 kernel: drbd r1/0 drbd1 drbd9-03: repl( Ahead -> Off )
  26. Apr 11 10:14:45 drbd9-02 kernel: drbd r1 drbd9-03: ack_receiver terminated
  27. Apr 11 10:14:45 drbd9-02 kernel: drbd r1 drbd9-03: Terminating ack_recv thread
  28. Apr 11 10:14:45 drbd9-02 kernel: drbd r1 drbd9-03: sock was shut down by peer
  29. Apr 11 10:14:45 drbd9-02 kernel: drbd r1 drbd9-03: Connection closed
  30. Apr 11 10:14:45 drbd9-02 kernel: drbd r1 drbd9-03: conn( NetworkFailure -> Unconnected )
  31. Apr 11 10:14:46 drbd9-02 kernel: drbd r1 drbd9-03: Restarting receiver thread
  32. Apr 11 10:14:46 drbd9-02 kernel: drbd r1 drbd9-03: conn( Unconnected -> Connecting )
  33. Apr 11 10:14:46 drbd9-02 kernel: drbd r1 drbd9-03: Handshake successful: Agreed network protocol version 111
  34. Apr 11 10:14:46 drbd9-02 kernel: drbd r1 drbd9-03: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
  35. Apr 11 10:14:46 drbd9-02 kernel: drbd r1 drbd9-03: Starting ack_recv thread (from drbd_r_r1 [1704])
  36. Apr 11 10:14:46 drbd9-02 kernel: drbd r1: Preparing cluster-wide state change 3076172425 (1->2 499/145)
  37. Apr 11 10:14:46 drbd9-02 kernel: drbd r1: State change 3076172425: primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFF9
  38. Apr 11 10:14:46 drbd9-02 kernel: drbd r1: Committing cluster-wide state change 3076172425 (0ms)
  39. Apr 11 10:14:46 drbd9-02 kernel: drbd r1 drbd9-03: conn( Connecting -> Connected ) peer( Unknown -> Secondary )
  40. Apr 11 10:14:46 drbd9-02 kernel: drbd r1/0 drbd1: peer does not support WRITE_SAME
  41. Apr 11 10:14:46 drbd9-02 kernel: drbd r1/0 drbd1 drbd9-03: updated UUIDs 9D2D64E77C03610B:0000000000000000:72BF2B4A67324B6C:15D43B83F5FF2550
  42. Apr 11 10:14:46 drbd9-02 kernel: drbd r1/0 drbd1: bitmap WRITE of 5 pages took 7 ms
  43. Apr 11 10:14:46 drbd9-02 kernel: drbd r1/0 drbd1 drbd9-03: drbd_sync_handshake:
  44. Apr 11 10:14:46 drbd9-02 kernel: drbd r1/0 drbd1 drbd9-03: self 9D2D64E77C03610B:0000000000000000:72BF2B4A67324B6C:15D43B83F5FF2550 bits:235985 flags:0
  45. Apr 11 10:14:46 drbd9-02 kernel: drbd r1/0 drbd1 drbd9-03: peer 9D2D64E77C03610A:0000000000000000:72BF2B4A67324B6C:0000000000000000 bits:190914 flags:104
  46. Apr 11 10:14:46 drbd9-02 kernel: drbd r1/0 drbd1 drbd9-03: uuid_compare()=0 by rule 38
  47. Apr 11 10:14:46 drbd9-02 kernel: drbd r1/0 drbd1: Becoming sync source due to disk states.
  48. Apr 11 10:14:46 drbd9-02 kernel: drbd r1/0 drbd1 drbd9-03: repl( Off -> WFBitMapS ) //-------------------------> pending!
  49. Apr 11 10:16:57 drbd9-02 kernel: INFO: task pool:2916 blocked for more than 120 seconds.
  50. Apr 11 10:16:57 drbd9-02 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  51. Apr 11 10:16:57 drbd9-02 kernel: pool D ffff88003d613680 0 2916 1 0x00000080
  52. Apr 11 10:16:57 drbd9-02 kernel: ffff880027637b28 0000000000000086 ffff880014ef5b00 ffff880027637fd8
  53. Apr 11 10:16:57 drbd9-02 kernel: ffff880027637fd8 ffff880027637fd8 ffff880014ef5b00 ffff88003d613f48
  54. Apr 11 10:16:57 drbd9-02 kernel: ffff88003ffb8f50 0000000000000002 ffffffffa04e0720 ffff880027637ba0
  55. Apr 11 10:16:57 drbd9-02 kernel: Call Trace:
  56. Apr 11 10:16:57 drbd9-02 kernel: [<ffffffffa04e0720>] ? start_this_handle+0x5d0/0x5d0 [jbd2]
  57. Apr 11 10:16:57 drbd9-02 kernel: [<ffffffff816099ad>] io_schedule+0x9d/0x130
  58. Apr 11 10:16:57 drbd9-02 kernel: [<ffffffffa04e072e>] sleep_on_shadow_bh+0xe/0x20 [jbd2]
  59. Apr 11 10:16:57 drbd9-02 kernel: [<ffffffff81607780>] __wait_on_bit+0x60/0x90
  60. Apr 11 10:16:57 drbd9-02 kernel: [<ffffffff81012609>] ? __switch_to+0x179/0x4a0
  61. Apr 11 10:16:57 drbd9-02 kernel: [<ffffffffa04e0720>] ? start_this_handle+0x5d0/0x5d0 [jbd2]
  62. Apr 11 10:16:57 drbd9-02 kernel: [<ffffffff81607837>] out_of_line_wait_on_bit+0x87/0xb0
  63. Apr 11 10:16:57 drbd9-02 kernel: [<ffffffff81098270>] ? autoremove_wake_function+0x40/0x40
  64. Apr 11 10:16:57 drbd9-02 kernel: [<ffffffffa04e1c0d>] do_get_write_access+0x2ed/0x500 [jbd2]
  65. Apr 11 10:16:57 drbd9-02 kernel: [<ffffffff811f9eed>] ? __getblk+0x2d/0x2e0
  66. Apr 11 10:16:57 drbd9-02 kernel: [<ffffffffa04e1e47>] jbd2_journal_get_write_access+0x27/0x40 [jbd2]
  67. Apr 11 10:16:57 drbd9-02 kernel: [<ffffffffa05434cb>] __ext4_journal_get_write_access+0x3b/0x80 [ext4]
  68. Apr 11 10:16:57 drbd9-02 kernel: [<ffffffffa05159a0>] ext4_reserve_inode_write+0x70/0xa0 [ext4]
  69. Apr 11 10:16:57 drbd9-02 kernel: [<ffffffffa0518e30>] ? ext4_dirty_inode+0x40/0x60 [ext4]
  70. Apr 11 10:16:57 drbd9-02 kernel: [<ffffffffa0515a23>] ext4_mark_inode_dirty+0x53/0x210 [ext4]
  71. Apr 11 10:16:57 drbd9-02 kernel: [<ffffffffa0518e30>] ext4_dirty_inode+0x40/0x60 [ext4]
  72. Apr 11 10:16:57 drbd9-02 kernel: [<ffffffff811f069a>] __mark_inode_dirty+0xca/0x290
  73. Apr 11 10:16:57 drbd9-02 kernel: [<ffffffff811e1721>] update_time+0x81/0xd0
  74. Apr 11 10:16:57 drbd9-02 kernel: [<ffffffff811e1970>] file_update_time+0xa0/0xf0
  75. Apr 11 10:16:57 drbd9-02 kernel: [<ffffffff811f40d8>] generic_file_splice_write+0xc8/0x180
  76. Apr 11 10:16:57 drbd9-02 kernel: [<ffffffff811f4010>] ? pipe_to_file+0x170/0x170
  77. Apr 11 10:16:57 drbd9-02 kernel: [<ffffffff811f4603>] do_splice_from+0xc3/0x140
  78. Apr 11 10:16:57 drbd9-02 kernel: [<ffffffff811f619e>] SyS_splice+0x27e/0x690
  79. Apr 11 10:16:57 drbd9-02 kernel: [<ffffffff81614209>] system_call_fastpath+0x16/0x1b
  80. Apr 11 10:18:57 drbd9-02 kernel: INFO: task kworker/u256:0:6 blocked for more than 120 seconds.
  81. Apr 11 10:18:57 drbd9-02 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  82. Apr 11 10:18:57 drbd9-02 kernel: kworker/u256:0 D ffff88003d613680 0 6 2 0x00000000
  83. Apr 11 10:18:57 drbd9-02 kernel: Workqueue: writeback bdi_writeback_workfn (flush-147:1)
  84. Apr 11 10:18:57 drbd9-02 kernel: ffff88003cfff948 0000000000000046 ffff88003cee38e0 ffff88003cffffd8
  85. Apr 11 10:18:57 drbd9-02 kernel: ffff88003cffffd8 ffff88003cffffd8 ffff88003cee38e0 ffff88002a743c00
  86. Apr 11 10:18:57 drbd9-02 kernel: ffff88002a743d50 ffff88002d595000 000000010000c0e1 ffff88002a743d74
  87. Apr 11 10:18:57 drbd9-02 kernel: Call Trace:
  88. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff816096a9>] schedule+0x29/0x70
  89. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffffa040759a>] drbd_make_request+0x14a/0x360 [drbd]
  90. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff812aa56f>] ? generic_make_request_checks+0x24f/0x380
  91. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff81098230>] ? wake_up_bit+0x30/0x30
  92. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff812aa782>] generic_make_request+0xe2/0x130
  93. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff812aa841>] submit_bio+0x71/0x150
  94. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffffa0519c55>] ext4_io_submit+0x25/0x50 [ext4]
  95. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffffa05164b3>] ext4_writepages+0x8d3/0xd60 [ext4]
  96. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff81162bfe>] do_writepages+0x1e/0x40
  97. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff811f0000>] __writeback_single_inode+0x40/0x220
  98. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff811f0cfe>] writeback_sb_inodes+0x25e/0x420
  99. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff811f0f5f>] __writeback_inodes_wb+0x9f/0xd0
  100. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff811f17a3>] wb_writeback+0x263/0x2f0
  101. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff811f2cdc>] bdi_writeback_workfn+0x1cc/0x460
  102. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff8108f0bb>] process_one_work+0x17b/0x470
  103. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff8108fe8b>] worker_thread+0x11b/0x400
  104. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff8108fd70>] ? rescuer_thread+0x400/0x400
  105. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff8109726f>] kthread+0xcf/0xe0
  106. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff810971a0>] ? kthread_create_on_node+0x140/0x140
  107. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff81614158>] ret_from_fork+0x58/0x90
  108. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff810971a0>] ? kthread_create_on_node+0x140/0x140
  109. Apr 11 10:18:57 drbd9-02 kernel: INFO: task pool:2916 blocked for more than 120 seconds.
  110. Apr 11 10:18:57 drbd9-02 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  111. Apr 11 10:18:57 drbd9-02 kernel: pool D ffff88003d613680 0 2916 1 0x00000080
  112. Apr 11 10:18:57 drbd9-02 kernel: ffff880027637b28 0000000000000086 ffff880014ef5b00 ffff880027637fd8
  113. Apr 11 10:18:57 drbd9-02 kernel: ffff880027637fd8 ffff880027637fd8 ffff880014ef5b00 ffff88003d613f48
  114. Apr 11 10:18:57 drbd9-02 kernel: ffff88003ffb8f50 0000000000000002 ffffffffa04e0720 ffff880027637ba0
  115. Apr 11 10:18:57 drbd9-02 kernel: Call Trace:
  116. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffffa04e0720>] ? start_this_handle+0x5d0/0x5d0 [jbd2]
  117. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff816099ad>] io_schedule+0x9d/0x130
  118. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffffa04e072e>] sleep_on_shadow_bh+0xe/0x20 [jbd2]
  119. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff81607780>] __wait_on_bit+0x60/0x90
  120. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff81012609>] ? __switch_to+0x179/0x4a0
  121. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffffa04e0720>] ? start_this_handle+0x5d0/0x5d0 [jbd2]
  122. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff81607837>] out_of_line_wait_on_bit+0x87/0xb0
  123. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff81098270>] ? autoremove_wake_function+0x40/0x40
  124. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffffa04e1c0d>] do_get_write_access+0x2ed/0x500 [jbd2]
  125. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff811f9eed>] ? __getblk+0x2d/0x2e0
  126. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffffa04e1e47>] jbd2_journal_get_write_access+0x27/0x40 [jbd2]
  127. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffffa05434cb>] __ext4_journal_get_write_access+0x3b/0x80 [ext4]
  128. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffffa05159a0>] ext4_reserve_inode_write+0x70/0xa0 [ext4]
  129. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffffa0518e30>] ? ext4_dirty_inode+0x40/0x60 [ext4]
  130. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffffa0515a23>] ext4_mark_inode_dirty+0x53/0x210 [ext4]
  131. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffffa0518e30>] ext4_dirty_inode+0x40/0x60 [ext4]
  132. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff811f069a>] __mark_inode_dirty+0xca/0x290
  133. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff811e1721>] update_time+0x81/0xd0
  134. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff811e1970>] file_update_time+0xa0/0xf0
  135. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff811f40d8>] generic_file_splice_write+0xc8/0x180
  136. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff811f4010>] ? pipe_to_file+0x170/0x170
  137. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff811f4603>] do_splice_from+0xc3/0x140
  138. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff811f619e>] SyS_splice+0x27e/0x690
  139. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff81614209>] system_call_fastpath+0x16/0x1b
  140. Apr 11 10:18:57 drbd9-02 kernel: INFO: task jbd2/drbd1-8:2900 blocked for more than 120 seconds.
  141. Apr 11 10:18:57 drbd9-02 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  142. Apr 11 10:18:57 drbd9-02 kernel: jbd2/drbd1-8 D ffff88003d613680 0 2900 2 0x00000080
  143. Apr 11 10:18:57 drbd9-02 kernel: ffff8800189c7b48 0000000000000046 ffff880014cda220 ffff8800189c7fd8
  144. Apr 11 10:18:57 drbd9-02 kernel: ffff8800189c7fd8 ffff8800189c7fd8 ffff880014cda220 ffff88002a743c00
  145. Apr 11 10:18:57 drbd9-02 kernel: ffff88002a743d50 ffff880014f86400 000000010000d287 ffff88002a743d74
  146. Apr 11 10:18:57 drbd9-02 kernel: Call Trace:
  147. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff816096a9>] schedule+0x29/0x70
  148. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffffa040759a>] drbd_make_request+0x14a/0x360 [drbd]
  149. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff812aa56f>] ? generic_make_request_checks+0x24f/0x380
  150. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff81098230>] ? wake_up_bit+0x30/0x30
  151. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff812aa782>] generic_make_request+0xe2/0x130
  152. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff812aa841>] submit_bio+0x71/0x150
  153. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff811fe98e>] ? bio_alloc_bioset+0x1be/0x2e0
  154. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff811f99b3>] _submit_bh+0x143/0x210
  155. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff811f9a90>] submit_bh+0x10/0x20
  156. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffffa04e36d0>] jbd2_journal_commit_transaction+0x6c0/0x19a0 [jbd2]
  157. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff810ad756>] ? __dequeue_entity+0x26/0x40
  158. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff810125c6>] ? __switch_to+0x136/0x4a0
  159. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffffa04e8da9>] kjournald2+0xc9/0x260 [jbd2]
  160. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff81098230>] ? wake_up_bit+0x30/0x30
  161. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffffa04e8ce0>] ? commit_timeout+0x10/0x10 [jbd2]
  162. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff8109726f>] kthread+0xcf/0xe0
  163. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff810971a0>] ? kthread_create_on_node+0x140/0x140
  164. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff81614158>] ret_from_fork+0x58/0x90
  165. Apr 11 10:18:57 drbd9-02 kernel: [<ffffffff810971a0>] ? kthread_create_on_node+0x140/0x140
  166. Apr 11 10:19:05 drbd9-02 chronyd[913]: Selected source 114.70.64.57
  167. Apr 11 10:20:02 drbd9-02 systemd: Starting Session 3 of user root.
  168. Apr 11 10:20:02 drbd9-02 systemd: Started Session 3 of user root.
  169. Apr 11 10:20:57 drbd9-02 kernel: INFO: task kworker/u256:0:6 blocked for more than 120 seconds.
  170. Apr 11 10:20:57 drbd9-02 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  171. Apr 11 10:20:57 drbd9-02 kernel: kworker/u256:0 D ffff88003d613680 0 6 2 0x00000000
  172. Apr 11 10:20:57 drbd9-02 kernel: Workqueue: writeback bdi_writeback_workfn (flush-147:1)
  173. Apr 11 10:20:57 drbd9-02 kernel: ffff88003cfff948 0000000000000046 ffff88003cee38e0 ffff88003cffffd8
  174. Apr 11 10:20:57 drbd9-02 kernel: ffff88003cffffd8 ffff88003cffffd8 ffff88003cee38e0 ffff88002a743c00
  175. Apr 11 10:20:57 drbd9-02 kernel: ffff88002a743d50 ffff88002d595000 000000010000c0e1 ffff88002a743d74
  176. Apr 11 10:20:57 drbd9-02 kernel: Call Trace:
  177. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff816096a9>] schedule+0x29/0x70
  178. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffffa040759a>] drbd_make_request+0x14a/0x360 [drbd]
  179. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff812aa56f>] ? generic_make_request_checks+0x24f/0x380
  180. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff81098230>] ? wake_up_bit+0x30/0x30
  181. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff812aa782>] generic_make_request+0xe2/0x130
  182. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff812aa841>] submit_bio+0x71/0x150
  183. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffffa0519c55>] ext4_io_submit+0x25/0x50 [ext4]
  184. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffffa05164b3>] ext4_writepages+0x8d3/0xd60 [ext4]
  185. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff81162bfe>] do_writepages+0x1e/0x40
  186. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff811f0000>] __writeback_single_inode+0x40/0x220
  187. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff811f0cfe>] writeback_sb_inodes+0x25e/0x420
  188. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff811f0f5f>] __writeback_inodes_wb+0x9f/0xd0
  189. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff811f17a3>] wb_writeback+0x263/0x2f0
  190. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff811f2cdc>] bdi_writeback_workfn+0x1cc/0x460
  191. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff8108f0bb>] process_one_work+0x17b/0x470
  192. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff8108fe8b>] worker_thread+0x11b/0x400
  193. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff8108fd70>] ? rescuer_thread+0x400/0x400
  194. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff8109726f>] kthread+0xcf/0xe0
  195. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff810971a0>] ? kthread_create_on_node+0x140/0x140
  196. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff81614158>] ret_from_fork+0x58/0x90
  197. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff810971a0>] ? kthread_create_on_node+0x140/0x140
  198. Apr 11 10:20:57 drbd9-02 kernel: INFO: task pool:2916 blocked for more than 120 seconds.
  199. Apr 11 10:20:57 drbd9-02 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  200. Apr 11 10:20:57 drbd9-02 kernel: pool D ffff88003d613680 0 2916 1 0x00000080
  201. Apr 11 10:20:57 drbd9-02 kernel: ffff880027637b28 0000000000000086 ffff880014ef5b00 ffff880027637fd8
  202. Apr 11 10:20:57 drbd9-02 kernel: ffff880027637fd8 ffff880027637fd8 ffff880014ef5b00 ffff88003d613f48
  203. Apr 11 10:20:57 drbd9-02 kernel: ffff88003ffb8f50 0000000000000002 ffffffffa04e0720 ffff880027637ba0
  204. Apr 11 10:20:57 drbd9-02 kernel: Call Trace:
  205. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffffa04e0720>] ? start_this_handle+0x5d0/0x5d0 [jbd2]
  206. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff816099ad>] io_schedule+0x9d/0x130
  207. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffffa04e072e>] sleep_on_shadow_bh+0xe/0x20 [jbd2]
  208. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff81607780>] __wait_on_bit+0x60/0x90
  209. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff81012609>] ? __switch_to+0x179/0x4a0
  210. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffffa04e0720>] ? start_this_handle+0x5d0/0x5d0 [jbd2]
  211. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff81607837>] out_of_line_wait_on_bit+0x87/0xb0
  212. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff81098270>] ? autoremove_wake_function+0x40/0x40
  213. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffffa04e1c0d>] do_get_write_access+0x2ed/0x500 [jbd2]
  214. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff811f9eed>] ? __getblk+0x2d/0x2e0
  215. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffffa04e1e47>] jbd2_journal_get_write_access+0x27/0x40 [jbd2]
  216. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffffa05434cb>] __ext4_journal_get_write_access+0x3b/0x80 [ext4]
  217. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffffa05159a0>] ext4_reserve_inode_write+0x70/0xa0 [ext4]
  218. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffffa0518e30>] ? ext4_dirty_inode+0x40/0x60 [ext4]
  219. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffffa0515a23>] ext4_mark_inode_dirty+0x53/0x210 [ext4]
  220. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffffa0518e30>] ext4_dirty_inode+0x40/0x60 [ext4]
  221. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff811f069a>] __mark_inode_dirty+0xca/0x290
  222. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff811e1721>] update_time+0x81/0xd0
  223. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff811e1970>] file_update_time+0xa0/0xf0
  224. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff811f40d8>] generic_file_splice_write+0xc8/0x180
  225. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff811f4010>] ? pipe_to_file+0x170/0x170
  226. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff811f4603>] do_splice_from+0xc3/0x140
  227. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff811f619e>] SyS_splice+0x27e/0x690
  228. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff81614209>] system_call_fastpath+0x16/0x1b
  229. Apr 11 10:20:57 drbd9-02 kernel: INFO: task jbd2/drbd1-8:2900 blocked for more than 120 seconds.
  230. Apr 11 10:20:57 drbd9-02 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  231. Apr 11 10:20:57 drbd9-02 kernel: jbd2/drbd1-8 D ffff88003d613680 0 2900 2 0x00000080
  232. Apr 11 10:20:57 drbd9-02 kernel: ffff8800189c7b48 0000000000000046 ffff880014cda220 ffff8800189c7fd8
  233. Apr 11 10:20:57 drbd9-02 kernel: ffff8800189c7fd8 ffff8800189c7fd8 ffff880014cda220 ffff88002a743c00
  234. Apr 11 10:20:57 drbd9-02 kernel: ffff88002a743d50 ffff880014f86400 000000010000d287 ffff88002a743d74
  235. Apr 11 10:20:57 drbd9-02 kernel: Call Trace:
  236. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff816096a9>] schedule+0x29/0x70
  237. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffffa040759a>] drbd_make_request+0x14a/0x360 [drbd]
  238. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff812aa56f>] ? generic_make_request_checks+0x24f/0x380
  239. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff81098230>] ? wake_up_bit+0x30/0x30
  240. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff812aa782>] generic_make_request+0xe2/0x130
  241. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff812aa841>] submit_bio+0x71/0x150
  242. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff811fe98e>] ? bio_alloc_bioset+0x1be/0x2e0
  243. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff811f99b3>] _submit_bh+0x143/0x210
  244. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff811f9a90>] submit_bh+0x10/0x20
  245. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffffa04e36d0>] jbd2_journal_commit_transaction+0x6c0/0x19a0 [jbd2]
  246. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff810ad756>] ? __dequeue_entity+0x26/0x40
  247. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff810125c6>] ? __switch_to+0x136/0x4a0
  248. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffffa04e8da9>] kjournald2+0xc9/0x260 [jbd2]
  249. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff81098230>] ? wake_up_bit+0x30/0x30
  250. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffffa04e8ce0>] ? commit_timeout+0x10/0x10 [jbd2]
  251. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff8109726f>] kthread+0xcf/0xe0
  252. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff810971a0>] ? kthread_create_on_node+0x140/0x140
  253. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff81614158>] ret_from_fork+0x58/0x90
  254. Apr 11 10:20:57 drbd9-02 kernel: [<ffffffff810971a0>] ? kthread_create_on_node+0x140/0x140
  255. Apr 11 10:22:57 drbd9-02 kernel: INFO: task kworker/u256:0:6 blocked for more than 120 seconds.
  256. Apr 11 10:22:57 drbd9-02 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  257. Apr 11 10:22:57 drbd9-02 kernel: kworker/u256:0 D ffff88003d613680 0 6 2 0x00000000
  258. Apr 11 10:22:57 drbd9-02 kernel: Workqueue: writeback bdi_writeback_workfn (flush-147:1)
  259. Apr 11 10:22:57 drbd9-02 kernel: ffff88003cfff948 0000000000000046 ffff88003cee38e0 ffff88003cffffd8
  260. Apr 11 10:22:57 drbd9-02 kernel: ffff88003cffffd8 ffff88003cffffd8 ffff88003cee38e0 ffff88002a743c00
  261. Apr 11 10:22:57 drbd9-02 kernel: ffff88002a743d50 ffff88002d595000 000000010000c0e1 ffff88002a743d74
  262. Apr 11 10:22:57 drbd9-02 kernel: Call Trace:
  263. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff816096a9>] schedule+0x29/0x70
  264. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffffa040759a>] drbd_make_request+0x14a/0x360 [drbd]
  265. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff812aa56f>] ? generic_make_request_checks+0x24f/0x380
  266. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff81098230>] ? wake_up_bit+0x30/0x30
  267. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff812aa782>] generic_make_request+0xe2/0x130
  268. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff812aa841>] submit_bio+0x71/0x150
  269. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffffa0519c55>] ext4_io_submit+0x25/0x50 [ext4]
  270. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffffa05164b3>] ext4_writepages+0x8d3/0xd60 [ext4]
  271. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff81162bfe>] do_writepages+0x1e/0x40
  272. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff811f0000>] __writeback_single_inode+0x40/0x220
  273. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff811f0cfe>] writeback_sb_inodes+0x25e/0x420
  274. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff811f0f5f>] __writeback_inodes_wb+0x9f/0xd0
  275. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff811f17a3>] wb_writeback+0x263/0x2f0
  276. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff811f2cdc>] bdi_writeback_workfn+0x1cc/0x460
  277. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff8108f0bb>] process_one_work+0x17b/0x470
  278. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff8108fe8b>] worker_thread+0x11b/0x400
  279. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff8108fd70>] ? rescuer_thread+0x400/0x400
  280. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff8109726f>] kthread+0xcf/0xe0
  281. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff810971a0>] ? kthread_create_on_node+0x140/0x140
  282. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff81614158>] ret_from_fork+0x58/0x90
  283. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff810971a0>] ? kthread_create_on_node+0x140/0x140
  284. Apr 11 10:22:57 drbd9-02 kernel: INFO: task pool:2916 blocked for more than 120 seconds.
  285. Apr 11 10:22:57 drbd9-02 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  286. Apr 11 10:22:57 drbd9-02 kernel: pool D ffff88003d613680 0 2916 1 0x00000080
  287. Apr 11 10:22:57 drbd9-02 kernel: ffff880027637b28 0000000000000086 ffff880014ef5b00 ffff880027637fd8
  288. Apr 11 10:22:57 drbd9-02 kernel: ffff880027637fd8 ffff880027637fd8 ffff880014ef5b00 ffff88003d613f48
  289. Apr 11 10:22:57 drbd9-02 kernel: ffff88003ffb8f50 0000000000000002 ffffffffa04e0720 ffff880027637ba0
  290. Apr 11 10:22:57 drbd9-02 kernel: Call Trace:
  291. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffffa04e0720>] ? start_this_handle+0x5d0/0x5d0 [jbd2]
  292. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff816099ad>] io_schedule+0x9d/0x130
  293. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffffa04e072e>] sleep_on_shadow_bh+0xe/0x20 [jbd2]
  294. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff81607780>] __wait_on_bit+0x60/0x90
  295. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff81012609>] ? __switch_to+0x179/0x4a0
  296. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffffa04e0720>] ? start_this_handle+0x5d0/0x5d0 [jbd2]
  297. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff81607837>] out_of_line_wait_on_bit+0x87/0xb0
  298. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff81098270>] ? autoremove_wake_function+0x40/0x40
  299. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffffa04e1c0d>] do_get_write_access+0x2ed/0x500 [jbd2]
  300. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff811f9eed>] ? __getblk+0x2d/0x2e0
  301. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffffa04e1e47>] jbd2_journal_get_write_access+0x27/0x40 [jbd2]
  302. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffffa05434cb>] __ext4_journal_get_write_access+0x3b/0x80 [ext4]
  303. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffffa05159a0>] ext4_reserve_inode_write+0x70/0xa0 [ext4]
  304. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffffa0518e30>] ? ext4_dirty_inode+0x40/0x60 [ext4]
  305. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffffa0515a23>] ext4_mark_inode_dirty+0x53/0x210 [ext4]
  306. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffffa0518e30>] ext4_dirty_inode+0x40/0x60 [ext4]
  307. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff811f069a>] __mark_inode_dirty+0xca/0x290
  308. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff811e1721>] update_time+0x81/0xd0
  309. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff811e1970>] file_update_time+0xa0/0xf0
  310. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff811f40d8>] generic_file_splice_write+0xc8/0x180
  311. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff811f4010>] ? pipe_to_file+0x170/0x170
  312. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff811f4603>] do_splice_from+0xc3/0x140
  313. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff811f619e>] SyS_splice+0x27e/0x690
  314. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff81614209>] system_call_fastpath+0x16/0x1b
  315. Apr 11 10:22:57 drbd9-02 kernel: INFO: task jbd2/drbd1-8:2900 blocked for more than 120 seconds.
  316. Apr 11 10:22:57 drbd9-02 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  317. Apr 11 10:22:57 drbd9-02 kernel: jbd2/drbd1-8 D ffff88003d613680 0 2900 2 0x00000080
  318. Apr 11 10:22:57 drbd9-02 kernel: ffff8800189c7b48 0000000000000046 ffff880014cda220 ffff8800189c7fd8
  319. Apr 11 10:22:57 drbd9-02 kernel: ffff8800189c7fd8 ffff8800189c7fd8 ffff880014cda220 ffff88002a743c00
  320. Apr 11 10:22:57 drbd9-02 kernel: ffff88002a743d50 ffff880014f86400 000000010000d287 ffff88002a743d74
  321. Apr 11 10:22:57 drbd9-02 kernel: Call Trace:
  322. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff816096a9>] schedule+0x29/0x70
  323. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffffa040759a>] drbd_make_request+0x14a/0x360 [drbd]
  324. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff812aa56f>] ? generic_make_request_checks+0x24f/0x380
  325. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff81098230>] ? wake_up_bit+0x30/0x30
  326. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff812aa782>] generic_make_request+0xe2/0x130
  327. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff812aa841>] submit_bio+0x71/0x150
  328. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff811fe98e>] ? bio_alloc_bioset+0x1be/0x2e0
  329. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff811f99b3>] _submit_bh+0x143/0x210
  330. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff811f9a90>] submit_bh+0x10/0x20
  331. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffffa04e36d0>] jbd2_journal_commit_transaction+0x6c0/0x19a0 [jbd2]
  332. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff810ad756>] ? __dequeue_entity+0x26/0x40
  333. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff810125c6>] ? __switch_to+0x136/0x4a0
  334. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffffa04e8da9>] kjournald2+0xc9/0x260 [jbd2]
  335. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff81098230>] ? wake_up_bit+0x30/0x30
  336. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffffa04e8ce0>] ? commit_timeout+0x10/0x10 [jbd2]
  337. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff8109726f>] kthread+0xcf/0xe0
  338. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff810971a0>] ? kthread_create_on_node+0x140/0x140
  339. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff81614158>] ret_from_fork+0x58/0x90
  340. Apr 11 10:22:57 drbd9-02 kernel: [<ffffffff810971a0>] ? kthread_create_on_node+0x140/0x140
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement