API tools faq
paste
Advertisement
Guest User

[CASE-41] ahead_primary.txt

a guest
Apr 20th, 2016
109
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 39.88 KB | None | 0 0
raw download clone embed print report
  1. version: 9.0.2-1 (api:2/proto:86-111)
  2. GIT-hash: bdcc2a765a9a80c8b263c011a6508cb6e0c3e4d2 build by root@drbd9-04, 2016-04-20 17:38:59
  3. Transports (api:14): tcp (1.0.0)
  4.  
  5.  
  6. [root@drbd9-03 ~]# tail -f /var/log/messages | grep "drbd"
  7. Apr 20 18:14:54 drbd9-03 kernel: sd 4:0:0:0: [sdd] Write Protect is off
  8. Apr 20 18:14:54 drbd9-03 kernel: sd 4:0:0:0: [sdd] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
  9. Apr 20 18:14:54 drbd9-03 kernel: sdd: unknown partition table
  10. Apr 20 18:14:54 drbd9-03 kernel: sd 4:0:0:0: [sdd] Attached SCSI removable disk
  11. Apr 20 18:15:33 drbd9-03 kernel: drbd r0: Preparing cluster-wide state change 1124476236 (0->-1 3/1)
  12. Apr 20 18:15:33 drbd9-03 kernel: drbd r0: State change 1124476236: primary_nodes=1, weak_nodes=FFFFFFFFFFFFFFFC
  13. Apr 20 18:15:33 drbd9-03 kernel: drbd r0: Committing cluster-wide state change 1124476236 (1ms)
  14. Apr 20 18:15:33 drbd9-03 kernel: drbd r0: role( Secondary -> Primary )
  15. Apr 20 18:15:58 drbd9-03 kernel: EXT4-fs (drbd1): mounted filesystem with ordered data mode. Opts: (null)
  16. Apr 20 18:16:38 drbd9-03 chronyd[906]: System clock wrong by 5.200831 seconds, adjustment started
  17. Apr 20 18:17:42 drbd9-03 chronyd[906]: Selected source 211.233.40.78
  18. Apr 20 18:17:42 drbd9-03 chronyd[906]: System clock wrong by 5.357265 seconds, adjustment started
  19. Apr 20 18:17:43 drbd9-03 chronyd[906]: Selected source 114.207.245.166
  20. Apr 20 18:18:47 drbd9-03 chronyd[906]: System clock wrong by 4.938931 seconds, adjustment started
  21. Apr 20 18:19:52 drbd9-03 chronyd[906]: System clock wrong by 4.800627 seconds, adjustment started
  22. Apr 20 18:20:06 drbd9-03 systemd: Starting Session 3 of user root.
  23. Apr 20 18:20:06 drbd9-03 systemd: Started Session 3 of user root.
  24. Apr 20 18:20:50 drbd9-03 systemd: Got automount request for /proc/sys/fs/binfmt_misc, triggered by 3064 (nautilus-search)
  25. Apr 20 18:20:50 drbd9-03 systemd: Mounting Arbitrary Executable File Formats File System...
  26. Apr 20 18:20:50 drbd9-03 systemd: Mounted Arbitrary Executable File Formats File System.
  27. Apr 20 18:20:57 drbd9-03 chronyd[906]: Selected source 211.233.40.78
  28. Apr 20 18:20:57 drbd9-03 chronyd[906]: System clock wrong by -20.464500 seconds, adjustment started
  29. Apr 20 18:21:31 drbd9-03 kernel: drbd r0/0 drbd1: Congestion-fill threshold reached
  30. Apr 20 18:21:31 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: pdsk( UpToDate -> Consistent ) repl( Established -> Ahead )
  31. Apr 20 18:21:31 drbd9-03 gnome-session: (gnome-settings-daemon:2123): color-plugin-WARNING **: unable to get EDID for xrandr-Virtual1: unable to get EDID for output
  32. Apr 20 18:21:31 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: pdsk( Consistent -> Outdated )
  33. Apr 20 18:21:32 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: helper command: /sbin/drbdadm before-resync-source
  34. Apr 20 18:21:32 drbd9-03 kernel: drbd r0/0 drbd1: new current UUID: D569D49A8856B905 weak: FFFFFFFFFFFFFFFE
  35. Apr 20 18:21:33 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: helper command: /sbin/drbdadm before-resync-source exit code 0 (0x0)
  36. Apr 20 18:21:33 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: pdsk( Outdated -> Inconsistent ) repl( Ahead -> SyncSource )
  37. Apr 20 18:21:33 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: Began resync as SyncSource (will sync 78272 KB [19568 bits set]).
  38. Apr 20 18:21:33 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: updated UUIDs D569D49A8856B905:0000000000000000:919E3AB15C33EA8C:0000000000000000
  39. Apr 20 18:21:33 drbd9-03 kernel: drbd r0/0 drbd1: bitmap WRITE of 1 pages took 10 ms
  40. Apr 20 18:21:33 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: SyncSource still sees bits set!! FIXME
  41. Apr 20 18:21:33 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: Resync done (total 1 sec; paused 0 sec; 78272 K/sec)
  42. Apr 20 18:21:33 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: ASSERTION (n_oos - peer_device->rs_failed) == 0 FAILED in drbd_resync_finished
  43. Apr 20 18:21:33 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: pdsk( Inconsistent -> UpToDate ) repl( SyncSource -> Established )
  44. Apr 20 18:21:33 drbd9-03 kernel: drbd r0/0 drbd1: Congestion-fill threshold reached
  45. Apr 20 18:21:33 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: pdsk( UpToDate -> Consistent ) repl( Established -> Ahead )
  46. Apr 20 18:21:33 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: pdsk( Consistent -> Outdated )
  47. Apr 20 18:21:33 drbd9-03 kernel: drbd r0/0 drbd1: new current UUID: 4603B73BD0E1AF69 weak: FFFFFFFFFFFFFFFE
  48. Apr 20 18:21:34 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: helper command: /sbin/drbdadm before-resync-source
  49. Apr 20 18:21:34 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: helper command: /sbin/drbdadm before-resync-source exit code 0 (0x0)
  50. Apr 20 18:21:34 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: pdsk( Outdated -> Inconsistent ) repl( Ahead -> SyncSource )
  51. Apr 20 18:21:34 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: Began resync as SyncSource (will sync 149952 KB [37488 bits set]).
  52. Apr 20 18:21:34 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: updated UUIDs 4603B73BD0E1AF69:0000000000000000:D569D49A8856B904:919E3AB15C33EA8C
  53. Apr 20 18:21:34 drbd9-03 kernel: drbd r0/0 drbd1: bitmap WRITE of 2 pages took 10 ms
  54. Apr 20 18:21:34 drbd9-03 kernel: drbd r0/0 drbd1: Congestion-fill threshold reached
  55. Apr 20 18:21:34 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: repl( SyncSource -> Ahead )
  56. Apr 20 18:21:35 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: helper command: /sbin/drbdadm before-resync-source
  57. Apr 20 18:21:35 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: helper command: /sbin/drbdadm before-resync-source exit code 0 (0x0)
  58. Apr 20 18:21:35 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: repl( Ahead -> SyncSource )
  59. Apr 20 18:21:35 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: Began resync as SyncSource (will sync 162212 KB [40553 bits set]).
  60. Apr 20 18:21:35 drbd9-03 kernel: drbd r0/0 drbd1: Congestion-fill threshold reached
  61. Apr 20 18:21:35 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: repl( SyncSource -> Ahead )
  62. Apr 20 18:21:36 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: helper command: /sbin/drbdadm before-resync-source
  63. Apr 20 18:21:36 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: helper command: /sbin/drbdadm before-resync-source exit code 0 (0x0)
  64. Apr 20 18:21:36 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: repl( Ahead -> SyncSource )
  65. Apr 20 18:21:36 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: Began resync as SyncSource (will sync 202376 KB [50594 bits set]).
  66. Apr 20 18:21:36 drbd9-03 kernel: drbd r0/0 drbd1: Congestion-fill threshold reached
  67. Apr 20 18:21:36 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: repl( SyncSource -> Ahead )
  68. Apr 20 18:21:38 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: helper command: /sbin/drbdadm before-resync-source
  69. Apr 20 18:21:38 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: helper command: /sbin/drbdadm before-resync-source exit code 0 (0x0)
  70. Apr 20 18:21:38 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: repl( Ahead -> SyncSource )
  71. Apr 20 18:21:38 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: Began resync as SyncSource (will sync 341372 KB [85343 bits set]).
  72. Apr 20 18:21:38 drbd9-03 kernel: drbd r0/0 drbd1: Congestion-fill threshold reached
  73. Apr 20 18:21:38 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: repl( SyncSource -> Ahead )
  74. Apr 20 18:21:39 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: helper command: /sbin/drbdadm before-resync-source
  75. Apr 20 18:21:39 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: helper command: /sbin/drbdadm before-resync-source exit code 0 (0x0)
  76. Apr 20 18:21:39 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: repl( Ahead -> SyncSource )
  77. Apr 20 18:21:39 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: Began resync as SyncSource (will sync 356416 KB [89104 bits set]).
  78. Apr 20 18:21:39 drbd9-03 kernel: drbd r0/0 drbd1: Congestion-fill threshold reached
  79. Apr 20 18:21:39 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: repl( SyncSource -> Ahead )
  80. Apr 20 18:21:40 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: helper command: /sbin/drbdadm before-resync-source
  81. Apr 20 18:21:40 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: helper command: /sbin/drbdadm before-resync-source exit code 0 (0x0)
  82. Apr 20 18:21:40 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: repl( Ahead -> SyncSource )
  83. Apr 20 18:21:40 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: Began resync as SyncSource (will sync 396068 KB [99017 bits set]).
  84. Apr 20 18:21:40 drbd9-03 kernel: drbd r0/0 drbd1: Congestion-fill threshold reached
  85. Apr 20 18:21:40 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: repl( SyncSource -> Ahead )
  86. Apr 20 18:21:41 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: helper command: /sbin/drbdadm before-resync-source
  87. Apr 20 18:21:41 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: helper command: /sbin/drbdadm before-resync-source exit code 0 (0x0)
  88. Apr 20 18:21:41 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: repl( Ahead -> SyncSource )
  89. Apr 20 18:21:41 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: Began resync as SyncSource (will sync 489480 KB [122370 bits set]).
  90. Apr 20 18:21:41 drbd9-03 kernel: drbd r0/0 drbd1: Congestion-fill threshold reached
  91. Apr 20 18:21:41 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: repl( SyncSource -> Ahead )
  92. Apr 20 18:21:56 drbd9-03 kernel: drbd r0 drbd9-04: meta connection shut down by peer.
  93. Apr 20 18:21:56 drbd9-03 rtkit-daemon[928]: The canary thread is apparently starving. Taking action.
  94. Apr 20 18:21:56 drbd9-03 rtkit-daemon[928]: Demoting known real-time threads.
  95. Apr 20 18:21:56 drbd9-03 rtkit-daemon[928]: Demoted 0 threads.
  96. Apr 20 18:21:56 drbd9-03 kernel: drbd r0 drbd9-04: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  97. Apr 20 18:21:56 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: repl( Ahead -> Off )
  98. Apr 20 18:21:56 drbd9-03 kernel: drbd r0 drbd9-04: ack_receiver terminated
  99. Apr 20 18:21:56 drbd9-03 kernel: drbd r0 drbd9-04: Terminating ack_recv thread
  100. Apr 20 18:21:56 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: postponing start_resync ...
  101. Apr 20 18:21:56 drbd9-03 kernel: drbd r0 drbd9-04: sock was shut down by peer
  102. Apr 20 18:21:56 drbd9-03 kernel: drbd r0 drbd9-04: Connection closed
  103. Apr 20 18:21:56 drbd9-03 kernel: drbd r0 drbd9-04: conn( NetworkFailure -> Unconnected )
  104. Apr 20 18:21:56 drbd9-03 kernel: drbd r0 drbd9-04: Restarting receiver thread
  105. Apr 20 18:21:56 drbd9-03 kernel: drbd r0 drbd9-04: conn( Unconnected -> Connecting )
  106. Apr 20 18:21:56 drbd9-03 kernel: drbd r0 drbd9-04: Handshake successful: Agreed network protocol version 111
  107. Apr 20 18:21:56 drbd9-03 kernel: drbd r0 drbd9-04: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
  108. Apr 20 18:21:56 drbd9-03 kernel: drbd r0 drbd9-04: Starting ack_recv thread (from drbd_r_r0 [1697])
  109. Apr 20 18:21:56 drbd9-03 kernel: drbd r0: Preparing cluster-wide state change 3573631356 (0->1 499/145)
  110. Apr 20 18:21:56 drbd9-03 kernel: drbd r0: State change 3573631356: primary_nodes=1, weak_nodes=FFFFFFFFFFFFFFFC
  111. Apr 20 18:21:56 drbd9-03 kernel: drbd r0: Committing cluster-wide state change 3573631356 (13ms)
  112. Apr 20 18:21:56 drbd9-03 kernel: drbd r0 drbd9-04: conn( Connecting -> Connected ) peer( Unknown -> Secondary )
  113. Apr 20 18:21:56 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: drbd_sync_handshake:
  114. Apr 20 18:21:56 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: self 4603B73BD0E1AF69:0000000000000000:D569D49A8856B904:919E3AB15C33EA8C bits:168335 flags:0
  115. Apr 20 18:21:56 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: peer 4603B73BD0E1AF68:0000000000000000:D569D49A8856B904:1C0061DB90E19FE0 bits:138759 flags:104
  116. Apr 20 18:21:56 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: uuid_compare()=0 by rule 38
  117. Apr 20 18:21:56 drbd9-03 kernel: drbd r0/0 drbd1: Becoming sync source due to disk states.
  118. Apr 20 18:21:56 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: repl( Off -> WFBitMapS )
  119. Apr 20 18:23:06 drbd9-03 chronyd[906]: Selected source 114.207.245.166
  120. Apr 20 18:25:16 drbd9-03 kernel: INFO: task pool:3092 blocked for more than 120 seconds.
  121. Apr 20 18:25:16 drbd9-03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  122. Apr 20 18:25:16 drbd9-03 kernel: pool D ffff88003d613680 0 3092 1 0x00000080
  123. Apr 20 18:25:16 drbd9-03 kernel: ffff8800138cfb28 0000000000000082 ffff8800376238e0 ffff8800138cffd8
  124. Apr 20 18:25:16 drbd9-03 kernel: ffff8800138cffd8 ffff8800138cffd8 ffff8800376238e0 ffff88003d613f48
  125. Apr 20 18:25:16 drbd9-03 kernel: ffff88003ffba750 0000000000000002 ffffffffa0505720 ffff8800138cfba0
  126. Apr 20 18:25:16 drbd9-03 kernel: Call Trace:
  127. Apr 20 18:25:16 drbd9-03 kernel: [<ffffffffa0505720>] ? start_this_handle+0x5d0/0x5d0 [jbd2]
  128. Apr 20 18:25:16 drbd9-03 kernel: [<ffffffff816099ad>] io_schedule+0x9d/0x130
  129. Apr 20 18:25:16 drbd9-03 kernel: [<ffffffffa050572e>] sleep_on_shadow_bh+0xe/0x20 [jbd2]
  130. Apr 20 18:25:16 drbd9-03 kernel: [<ffffffff81607780>] __wait_on_bit+0x60/0x90
  131. Apr 20 18:25:16 drbd9-03 kernel: [<ffffffff81012609>] ? __switch_to+0x179/0x4a0
  132. Apr 20 18:25:16 drbd9-03 kernel: [<ffffffffa0505720>] ? start_this_handle+0x5d0/0x5d0 [jbd2]
  133. Apr 20 18:25:16 drbd9-03 kernel: [<ffffffff81607837>] out_of_line_wait_on_bit+0x87/0xb0
  134. Apr 20 18:25:16 drbd9-03 kernel: [<ffffffff81098270>] ? autoremove_wake_function+0x40/0x40
  135. Apr 20 18:25:16 drbd9-03 kernel: [<ffffffffa0506c0d>] do_get_write_access+0x2ed/0x500 [jbd2]
  136. Apr 20 18:25:16 drbd9-03 kernel: [<ffffffff811f9eed>] ? __getblk+0x2d/0x2e0
  137. Apr 20 18:25:16 drbd9-03 kernel: [<ffffffffa0506e47>] jbd2_journal_get_write_access+0x27/0x40 [jbd2]
  138. Apr 20 18:25:16 drbd9-03 kernel: [<ffffffffa05594cb>] __ext4_journal_get_write_access+0x3b/0x80 [ext4]
  139. Apr 20 18:25:16 drbd9-03 kernel: [<ffffffffa052b9a0>] ext4_reserve_inode_write+0x70/0xa0 [ext4]
  140. Apr 20 18:25:16 drbd9-03 kernel: [<ffffffffa052ee30>] ? ext4_dirty_inode+0x40/0x60 [ext4]
  141. Apr 20 18:25:16 drbd9-03 kernel: [<ffffffffa052ba23>] ext4_mark_inode_dirty+0x53/0x210 [ext4]
  142. Apr 20 18:25:16 drbd9-03 kernel: [<ffffffffa052ee30>] ext4_dirty_inode+0x40/0x60 [ext4]
  143. Apr 20 18:25:16 drbd9-03 kernel: [<ffffffff811f069a>] __mark_inode_dirty+0xca/0x290
  144. Apr 20 18:25:16 drbd9-03 kernel: [<ffffffff811e1721>] update_time+0x81/0xd0
  145. Apr 20 18:25:16 drbd9-03 kernel: [<ffffffff811e1970>] file_update_time+0xa0/0xf0
  146. Apr 20 18:25:16 drbd9-03 kernel: [<ffffffff811f40d8>] generic_file_splice_write+0xc8/0x180
  147. Apr 20 18:25:16 drbd9-03 kernel: [<ffffffff811f4010>] ? pipe_to_file+0x170/0x170
  148. Apr 20 18:25:16 drbd9-03 kernel: [<ffffffff811f4603>] do_splice_from+0xc3/0x140
  149. Apr 20 18:25:16 drbd9-03 kernel: [<ffffffff811f619e>] SyS_splice+0x27e/0x690
  150. Apr 20 18:25:16 drbd9-03 kernel: [<ffffffff81614209>] system_call_fastpath+0x16/0x1b
  151. ^AApr 20 18:27:16 drbd9-03 kernel: INFO: task kworker/u256:1:293 blocked for more than 120 seconds.
  152. Apr 20 18:27:16 drbd9-03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  153. Apr 20 18:27:16 drbd9-03 kernel: kworker/u256:1 D ffff88003d613680 0 293 2 0x00000000
  154. Apr 20 18:27:16 drbd9-03 kernel: Workqueue: writeback bdi_writeback_workfn (flush-147:1)
  155. Apr 20 18:27:16 drbd9-03 kernel: ffff880036817948 0000000000000046 ffff880038fc8b60 ffff880036817fd8
  156. Apr 20 18:27:16 drbd9-03 kernel: ffff880036817fd8 ffff880036817fd8 ffff880038fc8b60 ffff88002dc74000
  157. Apr 20 18:27:16 drbd9-03 kernel: ffff88002dc74150 ffff88002dc74174 ffff880015606400 0000000100035e39
  158. Apr 20 18:27:16 drbd9-03 kernel: Call Trace:
  159. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff816096a9>] schedule+0x29/0x70
  160. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffffa04165ca>] drbd_make_request+0x15a/0x380 [drbd]
  161. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff812aa56f>] ? generic_make_request_checks+0x24f/0x380
  162. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff81098230>] ? wake_up_bit+0x30/0x30
  163. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff812aa782>] generic_make_request+0xe2/0x130
  164. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff812aa841>] submit_bio+0x71/0x150
  165. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffffa052fc55>] ext4_io_submit+0x25/0x50 [ext4]
  166. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffffa052c4b3>] ext4_writepages+0x8d3/0xd60 [ext4]
  167. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff81162bfe>] do_writepages+0x1e/0x40
  168. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff811f0000>] __writeback_single_inode+0x40/0x220
  169. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff811f0cfe>] writeback_sb_inodes+0x25e/0x420
  170. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff811f0f5f>] __writeback_inodes_wb+0x9f/0xd0
  171. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff811f17a3>] wb_writeback+0x263/0x2f0
  172. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff811f2cdc>] bdi_writeback_workfn+0x1cc/0x460
  173. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff8108f0bb>] process_one_work+0x17b/0x470
  174. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff8108fe8b>] worker_thread+0x11b/0x400
  175. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff8108fd70>] ? rescuer_thread+0x400/0x400
  176. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff8109726f>] kthread+0xcf/0xe0
  177. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff810971a0>] ? kthread_create_on_node+0x140/0x140
  178. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff81614158>] ret_from_fork+0x58/0x90
  179. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff810971a0>] ? kthread_create_on_node+0x140/0x140
  180. Apr 20 18:27:16 drbd9-03 kernel: INFO: task pool:3092 blocked for more than 120 seconds.
  181. Apr 20 18:27:16 drbd9-03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  182. Apr 20 18:27:16 drbd9-03 kernel: pool D ffff88003d613680 0 3092 1 0x00000080
  183. Apr 20 18:27:16 drbd9-03 kernel: ffff8800138cfb28 0000000000000082 ffff8800376238e0 ffff8800138cffd8
  184. Apr 20 18:27:16 drbd9-03 kernel: ffff8800138cffd8 ffff8800138cffd8 ffff8800376238e0 ffff88003d613f48
  185. Apr 20 18:27:16 drbd9-03 kernel: ffff88003ffba750 0000000000000002 ffffffffa0505720 ffff8800138cfba0
  186. Apr 20 18:27:16 drbd9-03 kernel: Call Trace:
  187. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffffa0505720>] ? start_this_handle+0x5d0/0x5d0 [jbd2]
  188. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff816099ad>] io_schedule+0x9d/0x130
  189. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffffa050572e>] sleep_on_shadow_bh+0xe/0x20 [jbd2]
  190. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff81607780>] __wait_on_bit+0x60/0x90
  191. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff81012609>] ? __switch_to+0x179/0x4a0
  192. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffffa0505720>] ? start_this_handle+0x5d0/0x5d0 [jbd2]
  193. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff81607837>] out_of_line_wait_on_bit+0x87/0xb0
  194. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff81098270>] ? autoremove_wake_function+0x40/0x40
  195. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffffa0506c0d>] do_get_write_access+0x2ed/0x500 [jbd2]
  196. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff811f9eed>] ? __getblk+0x2d/0x2e0
  197. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffffa0506e47>] jbd2_journal_get_write_access+0x27/0x40 [jbd2]
  198. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffffa05594cb>] __ext4_journal_get_write_access+0x3b/0x80 [ext4]
  199. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffffa052b9a0>] ext4_reserve_inode_write+0x70/0xa0 [ext4]
  200. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffffa052ee30>] ? ext4_dirty_inode+0x40/0x60 [ext4]
  201. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffffa052ba23>] ext4_mark_inode_dirty+0x53/0x210 [ext4]
  202. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffffa052ee30>] ext4_dirty_inode+0x40/0x60 [ext4]
  203. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff811f069a>] __mark_inode_dirty+0xca/0x290
  204. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff811e1721>] update_time+0x81/0xd0
  205. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff811e1970>] file_update_time+0xa0/0xf0
  206. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff811f40d8>] generic_file_splice_write+0xc8/0x180
  207. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff811f4010>] ? pipe_to_file+0x170/0x170
  208. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff811f4603>] do_splice_from+0xc3/0x140
  209. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff811f619e>] SyS_splice+0x27e/0x690
  210. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff81614209>] system_call_fastpath+0x16/0x1b
  211. Apr 20 18:27:16 drbd9-03 kernel: INFO: task jbd2/drbd1-8:2922 blocked for more than 120 seconds.
  212. Apr 20 18:27:16 drbd9-03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  213. Apr 20 18:27:16 drbd9-03 kernel: jbd2/drbd1-8 D ffff88003d613680 0 2922 2 0x00000080
  214. Apr 20 18:27:16 drbd9-03 kernel: ffff88001869bb48 0000000000000046 ffff88002b70e660 ffff88001869bfd8
  215. Apr 20 18:27:16 drbd9-03 kernel: ffff88001869bfd8 ffff88001869bfd8 ffff88002b70e660 ffff88002dc74000
  216. Apr 20 18:27:16 drbd9-03 kernel: ffff88002dc74150 ffff88002dc74174 ffff880010a52b00 0000000100036f63
  217. Apr 20 18:27:16 drbd9-03 kernel: Call Trace:
  218. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff816096a9>] schedule+0x29/0x70
  219. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffffa04165ca>] drbd_make_request+0x15a/0x380 [drbd]
  220. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff812aa56f>] ? generic_make_request_checks+0x24f/0x380
  221. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff81098230>] ? wake_up_bit+0x30/0x30
  222. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff812aa782>] generic_make_request+0xe2/0x130
  223. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff812aa841>] submit_bio+0x71/0x150
  224. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff811fe98e>] ? bio_alloc_bioset+0x1be/0x2e0
  225. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff811f99b3>] _submit_bh+0x143/0x210
  226. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff811f9a90>] submit_bh+0x10/0x20
  227. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffffa05086d0>] jbd2_journal_commit_transaction+0x6c0/0x19a0 [jbd2]
  228. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff810ad756>] ? __dequeue_entity+0x26/0x40
  229. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff810125c6>] ? __switch_to+0x136/0x4a0
  230. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffffa050dda9>] kjournald2+0xc9/0x260 [jbd2]
  231. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff81098230>] ? wake_up_bit+0x30/0x30
  232. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffffa050dce0>] ? commit_timeout+0x10/0x10 [jbd2]
  233. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff8109726f>] kthread+0xcf/0xe0
  234. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff810971a0>] ? kthread_create_on_node+0x140/0x140
  235. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff81614158>] ret_from_fork+0x58/0x90
  236. Apr 20 18:27:16 drbd9-03 kernel: [<ffffffff810971a0>] ? kthread_create_on_node+0x140/0x140
  237. Apr 20 18:28:16 drbd9-03 systemd: Starting Cleanup of Temporary Directories...
  238. Apr 20 18:28:16 drbd9-03 systemd: Started Cleanup of Temporary Directories.
  239. Apr 20 18:29:16 drbd9-03 kernel: INFO: task kworker/u256:1:293 blocked for more than 120 seconds.
  240. Apr 20 18:29:16 drbd9-03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  241. Apr 20 18:29:16 drbd9-03 kernel: kworker/u256:1 D ffff88003d613680 0 293 2 0x00000000
  242. Apr 20 18:29:16 drbd9-03 kernel: Workqueue: writeback bdi_writeback_workfn (flush-147:1)
  243. Apr 20 18:29:16 drbd9-03 kernel: ffff880036817948 0000000000000046 ffff880038fc8b60 ffff880036817fd8
  244. Apr 20 18:29:16 drbd9-03 kernel: ffff880036817fd8 ffff880036817fd8 ffff880038fc8b60 ffff88002dc74000
  245. Apr 20 18:29:16 drbd9-03 kernel: ffff88002dc74150 ffff88002dc74174 ffff880015606400 0000000100035e39
  246. Apr 20 18:29:16 drbd9-03 kernel: Call Trace:
  247. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff816096a9>] schedule+0x29/0x70
  248. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffffa04165ca>] drbd_make_request+0x15a/0x380 [drbd]
  249. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff812aa56f>] ? generic_make_request_checks+0x24f/0x380
  250. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff81098230>] ? wake_up_bit+0x30/0x30
  251. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff812aa782>] generic_make_request+0xe2/0x130
  252. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff812aa841>] submit_bio+0x71/0x150
  253. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffffa052fc55>] ext4_io_submit+0x25/0x50 [ext4]
  254. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffffa052c4b3>] ext4_writepages+0x8d3/0xd60 [ext4]
  255. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff81162bfe>] do_writepages+0x1e/0x40
  256. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff811f0000>] __writeback_single_inode+0x40/0x220
  257. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff811f0cfe>] writeback_sb_inodes+0x25e/0x420
  258. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff811f0f5f>] __writeback_inodes_wb+0x9f/0xd0
  259. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff811f17a3>] wb_writeback+0x263/0x2f0
  260. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff811f2cdc>] bdi_writeback_workfn+0x1cc/0x460
  261. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff8108f0bb>] process_one_work+0x17b/0x470
  262. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff8108fe8b>] worker_thread+0x11b/0x400
  263. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff8108fd70>] ? rescuer_thread+0x400/0x400
  264. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff8109726f>] kthread+0xcf/0xe0
  265. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff810971a0>] ? kthread_create_on_node+0x140/0x140
  266. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff81614158>] ret_from_fork+0x58/0x90
  267. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff810971a0>] ? kthread_create_on_node+0x140/0x140
  268. Apr 20 18:29:16 drbd9-03 kernel: INFO: task pool:3092 blocked for more than 120 seconds.
  269. Apr 20 18:29:16 drbd9-03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  270. Apr 20 18:29:16 drbd9-03 kernel: pool D ffff88003d613680 0 3092 1 0x00000080
  271. Apr 20 18:29:16 drbd9-03 kernel: ffff8800138cfb28 0000000000000082 ffff8800376238e0 ffff8800138cffd8
  272. Apr 20 18:29:16 drbd9-03 kernel: ffff8800138cffd8 ffff8800138cffd8 ffff8800376238e0 ffff88003d613f48
  273. Apr 20 18:29:16 drbd9-03 kernel: ffff88003ffba750 0000000000000002 ffffffffa0505720 ffff8800138cfba0
  274. Apr 20 18:29:16 drbd9-03 kernel: Call Trace:
  275. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffffa0505720>] ? start_this_handle+0x5d0/0x5d0 [jbd2]
  276. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff816099ad>] io_schedule+0x9d/0x130
  277. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffffa050572e>] sleep_on_shadow_bh+0xe/0x20 [jbd2]
  278. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff81607780>] __wait_on_bit+0x60/0x90
  279. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff81012609>] ? __switch_to+0x179/0x4a0
  280. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffffa0505720>] ? start_this_handle+0x5d0/0x5d0 [jbd2]
  281. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff81607837>] out_of_line_wait_on_bit+0x87/0xb0
  282. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff81098270>] ? autoremove_wake_function+0x40/0x40
  283. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffffa0506c0d>] do_get_write_access+0x2ed/0x500 [jbd2]
  284. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff811f9eed>] ? __getblk+0x2d/0x2e0
  285. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffffa0506e47>] jbd2_journal_get_write_access+0x27/0x40 [jbd2]
  286. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffffa05594cb>] __ext4_journal_get_write_access+0x3b/0x80 [ext4]
  287. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffffa052b9a0>] ext4_reserve_inode_write+0x70/0xa0 [ext4]
  288. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffffa052ee30>] ? ext4_dirty_inode+0x40/0x60 [ext4]
  289. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffffa052ba23>] ext4_mark_inode_dirty+0x53/0x210 [ext4]
  290. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffffa052ee30>] ext4_dirty_inode+0x40/0x60 [ext4]
  291. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff811f069a>] __mark_inode_dirty+0xca/0x290
  292. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff811e1721>] update_time+0x81/0xd0
  293. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff811e1970>] file_update_time+0xa0/0xf0
  294. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff811f40d8>] generic_file_splice_write+0xc8/0x180
  295. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff811f4010>] ? pipe_to_file+0x170/0x170
  296. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff811f4603>] do_splice_from+0xc3/0x140
  297. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff811f619e>] SyS_splice+0x27e/0x690
  298. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff81614209>] system_call_fastpath+0x16/0x1b
  299. Apr 20 18:29:16 drbd9-03 kernel: INFO: task jbd2/drbd1-8:2922 blocked for more than 120 seconds.
  300. Apr 20 18:29:16 drbd9-03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  301. Apr 20 18:29:16 drbd9-03 kernel: jbd2/drbd1-8 D ffff88003d613680 0 2922 2 0x00000080
  302. Apr 20 18:29:16 drbd9-03 kernel: ffff88001869bb48 0000000000000046 ffff88002b70e660 ffff88001869bfd8
  303. Apr 20 18:29:16 drbd9-03 kernel: ffff88001869bfd8 ffff88001869bfd8 ffff88002b70e660 ffff88002dc74000
  304. Apr 20 18:29:16 drbd9-03 kernel: ffff88002dc74150 ffff88002dc74174 ffff880010a52b00 0000000100036f63
  305. Apr 20 18:29:16 drbd9-03 kernel: Call Trace:
  306. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff816096a9>] schedule+0x29/0x70
  307. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffffa04165ca>] drbd_make_request+0x15a/0x380 [drbd]
  308. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff812aa56f>] ? generic_make_request_checks+0x24f/0x380
  309. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff81098230>] ? wake_up_bit+0x30/0x30
  310. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff812aa782>] generic_make_request+0xe2/0x130
  311. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff812aa841>] submit_bio+0x71/0x150
  312. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff811fe98e>] ? bio_alloc_bioset+0x1be/0x2e0
  313. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff811f99b3>] _submit_bh+0x143/0x210
  314. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff811f9a90>] submit_bh+0x10/0x20
  315. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffffa05086d0>] jbd2_journal_commit_transaction+0x6c0/0x19a0 [jbd2]
  316. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff810ad756>] ? __dequeue_entity+0x26/0x40
  317. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff810125c6>] ? __switch_to+0x136/0x4a0
  318. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffffa050dda9>] kjournald2+0xc9/0x260 [jbd2]
  319. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff81098230>] ? wake_up_bit+0x30/0x30
  320. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffffa050dce0>] ? commit_timeout+0x10/0x10 [jbd2]
  321. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff8109726f>] kthread+0xcf/0xe0
  322. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff810971a0>] ? kthread_create_on_node+0x140/0x140
  323. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff81614158>] ret_from_fork+0x58/0x90
  324. Apr 20 18:29:16 drbd9-03 kernel: [<ffffffff810971a0>] ? kthread_create_on_node+0x140/0x140
  325. Apr 20 18:30:01 drbd9-03 systemd: Starting Session 4 of user root.
  326. Apr 20 18:30:01 drbd9-03 systemd: Started Session 4 of user root.
  327. Apr 20 18:30:50 drbd9-03 kernel: drbd r0 drbd9-04: Preparing remote state change 1677017691 (primary_nodes=1, weak_nodes=FFFFFFFFFFFFFFFE)
  328. Apr 20 18:30:50 drbd9-03 kernel: drbd r0 drbd9-04: Committing remote state change 1677017691
  329. Apr 20 18:30:50 drbd9-03 kernel: drbd r0 drbd9-04: conn( Connected -> TearDown ) peer( Secondary -> Unknown )
  330. Apr 20 18:30:50 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: repl( WFBitMapS -> Off )
  331. Apr 20 18:30:50 drbd9-03 kernel: drbd r0 drbd9-04: ack_receiver terminated
  332. Apr 20 18:30:50 drbd9-03 kernel: drbd r0 drbd9-04: Terminating ack_recv thread
  333. Apr 20 18:30:50 drbd9-03 kernel: drbd r0/0 drbd1: drbd_w_r0[1661] going to 'send_bitmap (WFBitMapS)' but bitmap already locked for 'write from disconnected' by drbd_r_r0[1697]
  334. Apr 20 18:30:50 drbd9-03 kernel: drbd r0 drbd9-04: Connection closed
  335. Apr 20 18:30:50 drbd9-03 kernel: drbd r0 drbd9-04: conn( TearDown -> Unconnected )
  336. Apr 20 18:30:50 drbd9-03 kernel: drbd r0 drbd9-04: Restarting receiver thread
  337. Apr 20 18:30:50 drbd9-03 kernel: drbd r0 drbd9-04: conn( Unconnected -> Connecting )
  338. Apr 20 18:31:02 drbd9-03 kernel: drbd r0 drbd9-04: Handshake successful: Agreed network protocol version 111
  339. Apr 20 18:31:02 drbd9-03 kernel: drbd r0 drbd9-04: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
  340. Apr 20 18:31:02 drbd9-03 kernel: drbd r0 drbd9-04: Starting ack_recv thread (from drbd_r_r0 [1697])
  341. Apr 20 18:31:03 drbd9-03 kernel: drbd r0: Preparing cluster-wide state change 3341379820 (0->1 499/145)
  342. Apr 20 18:31:03 drbd9-03 kernel: drbd r0: State change 3341379820: primary_nodes=1, weak_nodes=FFFFFFFFFFFFFFFC
  343. Apr 20 18:31:03 drbd9-03 kernel: drbd r0: Committing cluster-wide state change 3341379820 (1ms)
  344. Apr 20 18:31:03 drbd9-03 kernel: drbd r0 drbd9-04: conn( Connecting -> Connected ) peer( Unknown -> Secondary )
  345. Apr 20 18:31:03 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: drbd_sync_handshake:
  346. Apr 20 18:31:03 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: self 4603B73BD0E1AF69:0000000000000000:D569D49A8856B904:919E3AB15C33EA8C bits:642367 flags:0
  347. Apr 20 18:31:03 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: peer 4603B73BD0E1AF68:0000000000000000:D569D49A8856B904:1C0061DB90E19FE0 bits:138759 flags:4
  348. Apr 20 18:31:03 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: uuid_compare()=0 by rule 38
  349. Apr 20 18:31:03 drbd9-03 kernel: drbd r0/0 drbd1: Becoming sync source due to disk states.
  350. Apr 20 18:31:03 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: repl( Off -> WFBitMapS )
  351. Apr 20 18:31:03 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 117(1), total 117; compression: 100.0%
  352. Apr 20 18:31:03 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 117(1), total 117; compression: 100.0%
  353. Apr 20 18:31:03 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: helper command: /sbin/drbdadm before-resync-source
  354. Apr 20 18:31:03 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: helper command: /sbin/drbdadm before-resync-source exit code 0 (0x0)
  355. Apr 20 18:31:03 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: repl( WFBitMapS -> SyncSource )
  356. Apr 20 18:31:03 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: Began resync as SyncSource (will sync 2585084 KB [646271 bits set]).
  357. Apr 20 18:31:03 drbd9-03 kernel: drbd r0/0 drbd1: Congestion-fill threshold reached
  358. Apr 20 18:31:03 drbd9-03 kernel: drbd r0/0 drbd1 drbd9-04: repl( SyncSource -> Ahead )
  359. lApr 20 18:33:16 drbd9-03 kernel: INFO: task pool:3092 blocked for more than 120 seconds.
  360. Apr 20 18:33:16 drbd9-03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  361. Apr 20 18:33:16 drbd9-03 kernel: pool D ffff88003d613680 0 3092 1 0x00000080
  362. Apr 20 18:33:16 drbd9-03 kernel: ffff8800138cfbd8 0000000000000082 ffff8800376238e0 ffff8800138cffd8
  363. Apr 20 18:33:16 drbd9-03 kernel: ffff8800138cffd8 ffff8800138cffd8 ffff8800376238e0 ffff8800377e9000
  364. Apr 20 18:33:16 drbd9-03 kernel: ffff8800377e9078 0000000000000016 ffff8800189a0100 ffff8800364f3030
  365. Apr 20 18:33:16 drbd9-03 kernel: Call Trace:
  366. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffff816096a9>] schedule+0x29/0x70
  367. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffffa0505085>] wait_transaction_locked+0x85/0xd0 [jbd2]
  368. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffff81098230>] ? wake_up_bit+0x30/0x30
  369. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffffa0505400>] start_this_handle+0x2b0/0x5d0 [jbd2]
  370. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffff811ab72a>] ? kmem_cache_alloc+0x1ba/0x1d0
  371. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffffa0505933>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
  372. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffffa052ee1a>] ? ext4_dirty_inode+0x2a/0x60 [ext4]
  373. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffffa0559299>] __ext4_journal_start_sb+0x69/0xe0 [ext4]
  374. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffffa052ee1a>] ext4_dirty_inode+0x2a/0x60 [ext4]
  375. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffff811f069a>] __mark_inode_dirty+0xca/0x290
  376. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffff811e1721>] update_time+0x81/0xd0
  377. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffff811e1970>] file_update_time+0xa0/0xf0
  378. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffff811f40d8>] generic_file_splice_write+0xc8/0x180
  379. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffff811f4010>] ? pipe_to_file+0x170/0x170
  380. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffff811f4603>] do_splice_from+0xc3/0x140
  381. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffff811f619e>] SyS_splice+0x27e/0x690
  382. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffff81614209>] system_call_fastpath+0x16/0x1b
  383. Apr 20 18:33:16 drbd9-03 kernel: INFO: task jbd2/drbd1-8:2922 blocked for more than 120 seconds.
  384. Apr 20 18:33:16 drbd9-03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  385. Apr 20 18:33:16 drbd9-03 kernel: jbd2/drbd1-8 D ffff88003d613680 0 2922 2 0x00000080
  386. Apr 20 18:33:16 drbd9-03 kernel: ffff88001869bc90 0000000000000046 ffff88002b70e660 ffff88001869bfd8
  387. Apr 20 18:33:16 drbd9-03 kernel: ffff88001869bfd8 ffff88001869bfd8 ffff88002b70e660 ffff88001869bda0
  388. Apr 20 18:33:16 drbd9-03 kernel: ffff8800377e90c0 ffff88002b70e660 ffff88001869bd88 ffff8800189a0100
  389. Apr 20 18:33:16 drbd9-03 kernel: Call Trace:
  390. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffff816096a9>] schedule+0x29/0x70
  391. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffffa0508258>] jbd2_journal_commit_transaction+0x248/0x19a0 [jbd2]
  392. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffff810abe95>] ? sched_clock_cpu+0xb5/0x100
  393. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffff810a2d9a>] ? update_rq_clock.part.74+0x1a/0x150
  394. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffff810af874>] ? update_blocked_averages+0x2a4/0x740
  395. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffff810125c6>] ? __switch_to+0x136/0x4a0
  396. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffff81098230>] ? wake_up_bit+0x30/0x30
  397. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffff8108007e>] ? try_to_del_timer_sync+0x5e/0x90
  398. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffffa050dda9>] kjournald2+0xc9/0x260 [jbd2]
  399. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffff81098230>] ? wake_up_bit+0x30/0x30
  400. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffffa050dce0>] ? commit_timeout+0x10/0x10 [jbd2]
  401. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffff8109726f>] kthread+0xcf/0xe0
  402. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffff810971a0>] ? kthread_create_on_node+0x140/0x140
  403. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffff81614158>] ret_from_fork+0x58/0x90
  404. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffff810971a0>] ? kthread_create_on_node+0x140/0x140
  405. Apr 20 18:33:16 drbd9-03 kernel: INFO: task kworker/u256:0:3230 blocked for more than 120 seconds.
  406. Apr 20 18:33:16 drbd9-03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  407. Apr 20 18:33:16 drbd9-03 kernel: kworker/u256:0 D ffff88003d613680 0 3230 2 0x00000080
  408. Apr 20 18:33:16 drbd9-03 kernel: Workqueue: writeback bdi_writeback_workfn (flush-147:1)
  409. Apr 20 18:33:16 drbd9-03 kernel: ffff88001b667800 0000000000000046 ffff8800162b5b00 ffff88001b667fd8
  410. Apr 20 18:33:16 drbd9-03 kernel: ffff88001b667fd8 ffff88001b667fd8 ffff8800162b5b00 ffff88002dc74000
  411. Apr 20 18:33:16 drbd9-03 kernel: ffff88002dc74150 ffff88002dc74174 ffff88003a798200 00000001000bb494
  412. Apr 20 18:33:16 drbd9-03 kernel: Call Trace:
  413. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffff816096a9>] schedule+0x29/0x70
  414. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffffa04165ca>] drbd_make_request+0x15a/0x380 [drbd]
  415. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffff812aa56f>] ? generic_make_request_checks+0x24f/0x380
  416. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffff81098230>] ? wake_up_bit+0x30/0x30
  417. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffff812aa782>] generic_make_request+0xe2/0x130
  418. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffff812aa841>] submit_bio+0x71/0x150
  419. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffffa052fc55>] ext4_io_submit+0x25/0x50 [ext4]
  420. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffffa052fe09>] ext4_bio_write_page+0x159/0x2e0 [ext4]
  421. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffffa0526f6d>] mpage_submit_page+0x5d/0x80 [ext4]
  422. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffffa0527232>] mpage_map_and_submit_buffers+0x172/0x2a0 [ext4]
  423. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffffa052c313>] ext4_writepages+0x733/0xd60 [ext4]
  424. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffff81162bfe>] do_writepages+0x1e/0x40
  425. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffff811f0000>] __writeback_single_inode+0x40/0x220
  426. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffff811f0cfe>] writeback_sb_inodes+0x25e/0x420
  427. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffff811f0f5f>] __writeback_inodes_wb+0x9f/0xd0
  428. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffff811f17a3>] wb_writeback+0x263/0x2f0
  429. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffff811f2cdc>] bdi_writeback_workfn+0x1cc/0x460
  430. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffff8108f0bb>] process_one_work+0x17b/0x470
  431. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffff8108fe8b>] worker_thread+0x11b/0x400
  432. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffff8108fd70>] ? rescuer_thread+0x400/0x400
  433. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffff8109726f>] kthread+0xcf/0xe0
  434. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffff810971a0>] ? kthread_create_on_node+0x140/0x140
  435. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffff81614158>] ret_from_fork+0x58/0x90
  436. Apr 20 18:33:16 drbd9-03 kernel: [<ffffffff810971a0>] ? kthread_create_on_node+0x140/0x140
  437. Apr 20 18:37:08 drbd9-03 dbus-daemon: dbus[929]: [system] Activating via systemd: service name='org.freedesktop.hostname1' unit='dbus-org.freedesktop.hostname1.service'
  438. Apr 20 18:37:08 drbd9-03 dbus[929]: [system] Activating via systemd: service name='org.freedesktop.hostname1' unit='dbus-org.freedesktop.hostname1.service'
  439. Apr 20 18:37:08 drbd9-03 systemd: Starting Hostname Service...
  440. Apr 20 18:37:08 drbd9-03 dbus-daemon: dbus[929]: [system] Successfully activated service 'org.freedesktop.hostname1'
  441. Apr 20 18:37:08 drbd9-03 dbus[929]: [system] Successfully activated service 'org.freedesktop.hostname1'
  442. Apr 20 18:37:08 drbd9-03 systemd: Started Hostname Service.
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!