API tools faq
paste
Advertisement
AleksZi

dmesg_storage-load-test-1.log

AleksZi
Mar 30th, 2025
16
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 199.11 KB | None | 0 0
raw download clone embed print report
  1. Mar 31 01:05:02 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Starting worker thread (node-id 1)
  2. Mar 31 01:05:02 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Starting sender thread (peer-node-id 0)
  3. Mar 31 01:05:02 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Starting sender thread (peer-node-id 2)
  4. Mar 31 01:05:02 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: meta-data IO uses: blk-bio
  5. Mar 31 01:05:02 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( no -> quorum ) [attach]
  6. Mar 31 01:05:02 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Diskless -> Attaching ) [attach]
  7. Mar 31 01:05:02 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: Maximum number of peer devices = 7
  8. Mar 31 01:05:02 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Method to ensure write ordering: flush
  9. Mar 31 01:05:02 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: drbd_bm_resize called with capacity == 2104816
  10. Mar 31 01:05:02 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: resync bitmap: bits=263102 words=28777 pages=57
  11. Mar 31 01:05:02 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: size = 1028 MB (1052408 KB)
  12. Mar 31 01:05:02 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Attaching -> Inconsistent ) [attach]
  13. Mar 31 01:05:02 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: attached to current UUID: 0000000000000004
  14. Mar 31 01:05:02 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( StandAlone -> Unconnected ) [connect]
  15. Mar 31 01:05:02 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( StandAlone -> Unconnected ) [connect]
  16. Mar 31 01:05:02 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Starting receiver thread (peer-node-id 0)
  17. Mar 31 01:05:02 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Unconnected -> Connecting ) [connecting]
  18. Mar 31 01:05:02 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Starting receiver thread (peer-node-id 2)
  19. Mar 31 01:05:02 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Unconnected -> Connecting ) [connecting]
  20. Mar 31 01:05:02 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Handshake to peer 2 successful: Agreed network protocol version 122
  21. Mar 31 01:05:02 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  22. Mar 31 01:05:02 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Peer authenticated using 20 bytes HMAC
  23. Mar 31 01:05:02 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 924098904: 1->2 role( Secondary ) conn( Connected )
  24. Mar 31 01:05:02 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: drbd_sync_handshake:
  25. Mar 31 01:05:02 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: self 0000000000000004:0000000000000000:0000000000000000:0000000000000000 bits:0 flags:24
  26. Mar 31 01:05:02 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: peer 0000000000000004:0000000000000000:0000000000000000:0000000000000000 bits:0 flags:24
  27. Mar 31 01:05:02 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: uuid_compare()=no-sync by rule=just-created-both
  28. Mar 31 01:05:02 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 924098904: primary_nodes=0, weak_nodes=0
  29. Mar 31 01:05:02 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 924098904 (28ms)
  30. Mar 31 01:05:02 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [connected]
  31. Mar 31 01:05:02 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( DUnknown -> Inconsistent ) repl( Off -> Established ) [connected]
  32. Mar 31 01:05:03 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3072324464: 0->2 role( Secondary ) conn( Connected )
  33. Mar 31 01:05:03 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Handshake to peer 0 successful: Agreed network protocol version 122
  34. Mar 31 01:05:03 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  35. Mar 31 01:05:03 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Peer authenticated using 20 bytes HMAC
  36. Mar 31 01:05:03 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 3072324464 (primary_nodes=0)
  37. Mar 31 01:05:03 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: resync-susp( no -> peer ) [peer-state]
  38. Mar 31 01:05:03 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2377433559: 0->1 role( Secondary ) conn( Connected )
  39. Mar 31 01:05:03 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: drbd_sync_handshake:
  40. Mar 31 01:05:03 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: self 0000000000000004:0000000000000000:28D31133FEBC993A:0000000000000000 bits:0 flags:24
  41. Mar 31 01:05:03 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: peer 52F5EAB2DCF34652:28D31133FEBC993B:0000000000000000:0000000000000000 bits:263102 flags:1020
  42. Mar 31 01:05:03 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: uuid_compare()=target-set-bitmap by rule=just-created-self
  43. Mar 31 01:05:03 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: Setting and writing the whole bitmap, fresh node
  44. Mar 31 01:05:03 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 2377433559 (primary_nodes=0)
  45. Mar 31 01:05:03 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  46. Mar 31 01:05:03 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( DUnknown -> UpToDate ) repl( Off -> WFBitMapT ) [remote]
  47. Mar 31 01:05:03 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: Setting exposed data uuid: 52F5EAB2DCF34652
  48. Mar 31 01:05:03 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: persisting effective size = 1028 MB (1052408 KB)
  49. Mar 31 01:05:03 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 21(1), total 21; compression: 100.0%
  50. Mar 31 01:05:03 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 21(1), total 21; compression: 100.0%
  51. Mar 31 01:05:03 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: setting UUIDs to 28D31133FEBC993A:0000000000000000:28D31133FEBC993A:0000000000000000
  52. Mar 31 01:05:03 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( WFBitMapT -> SyncTarget ) [receive-bitmap]
  53. Mar 31 01:05:03 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: Setting exposed data uuid: 52F5EAB2DCF34652
  54. Mar 31 01:05:03 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: Began resync as SyncTarget (will sync 1052408 KB [263102 bits set]).
  55. Mar 31 01:05:14 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: Resync done (total 10 sec; paused 0 sec; 105240 K/sec)
  56. Mar 31 01:05:14 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: updated UUIDs 52F5EAB2DCF34652:0000000000000000:28D31133FEBC993A:0000000000000000
  57. Mar 31 01:05:14 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( quorum -> no ) [resync-finished]
  58. Mar 31 01:05:14 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Inconsistent -> UpToDate ) quorum( no -> yes ) [resync-finished]
  59. Mar 31 01:05:14 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( SyncTarget -> Established ) [resync-finished]
  60. Mar 31 01:05:14 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: updated UUIDs 52F5EAB2DCF34652:0000000000000000:28D31133FEBC993A:0000000000000000
  61. Mar 31 01:05:14 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( Inconsistent -> UpToDate ) resync-susp( peer -> no ) [peer-state]
  62. Mar 31 01:05:15 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3028870165: 0->all role( Primary )
  63. Mar 31 01:05:15 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 3028870165 (primary_nodes=1)
  64. Mar 31 01:05:15 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: peer( Secondary -> Primary ) [remote]
  65. Mar 31 01:07:54 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2893496555: 0->all role( Secondary )
  66. Mar 31 01:07:54 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 2893496555 (primary_nodes=0)
  67. Mar 31 01:07:54 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: peer( Primary -> Secondary ) [remote]
  68. Mar 31 01:07:58 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Starting sender thread (peer-node-id 3)
  69. Mar 31 01:07:58 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( StandAlone -> Unconnected ) [connect]
  70. Mar 31 01:07:58 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Starting receiver thread (peer-node-id 3)
  71. Mar 31 01:07:58 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Unconnected -> Connecting ) [connecting]
  72. Mar 31 01:07:59 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Handshake to peer 3 successful: Agreed network protocol version 122
  73. Mar 31 01:07:59 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  74. Mar 31 01:07:59 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Peer authenticated using 20 bytes HMAC
  75. Mar 31 01:07:59 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4112650981: 2->3 role( Secondary ) conn( Connected )
  76. Mar 31 01:07:59 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 4112650981 (primary_nodes=0)
  77. Mar 31 01:07:59 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1056996341: 1->3 role( Secondary ) conn( Connected )
  78. Mar 31 01:07:59 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: self 52F5EAB2DCF34652:28D31133FEBC993A:28D31133FEBC993A:0000000000000000 bits:0 flags:0
  79. Mar 31 01:07:59 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: peer's exposed UUID: 52F5EAB2DCF34652
  80. Mar 31 01:07:59 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 1056996341: primary_nodes=0, weak_nodes=0
  81. Mar 31 01:07:59 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 1056996341 (20ms)
  82. Mar 31 01:07:59 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [connected]
  83. Mar 31 01:07:59 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( DUnknown -> Diskless ) repl( Off -> Established ) [connected]
  84. Mar 31 01:07:59 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1852736253: 0->3 role( Secondary ) conn( Connected )
  85. Mar 31 01:07:59 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 1852736253 (primary_nodes=0)
  86. Mar 31 01:08:04 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 229711114: 3->all role( Primary )
  87. Mar 31 01:08:04 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 229711114 (primary_nodes=8)
  88. Mar 31 01:08:04 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: peer( Secondary -> Primary ) [remote]
  89. Mar 31 01:08:04 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: received new current UUID: 3B36CC1B2C2CCBAD weak_nodes=FFFFFFFFFFFFFFF8
  90. Mar 31 01:08:40 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3427757018: 0->all empty
  91. Mar 31 01:08:41 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: PingAck did not arrive in time.
  92. Mar 31 01:08:41 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  93. Mar 31 01:08:41 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
  94. Mar 31 01:08:41 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Terminating sender thread
  95. Mar 31 01:08:41 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Starting sender thread (peer-node-id 2)
  96. Mar 31 01:08:41 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Aborting remote state change 3427757018
  97. Mar 31 01:08:41 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 780112742: 1->all empty
  98. Mar 31 01:08:41 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Connection closed
  99. Mar 31 01:08:41 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( NetworkFailure -> Unconnected ) [disconnected]
  100. Mar 31 01:08:41 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Restarting receiver thread
  101. Mar 31 01:08:41 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Unconnected -> Connecting ) [connecting]
  102. Mar 31 01:08:41 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 252ms
  103. Mar 31 01:08:41 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1468276786: 3->all empty
  104. Mar 31 01:08:41 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Rejecting concurrent remote state change 457764969 because of state change 1468276786
  105. Mar 31 01:08:41 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Aborting remote state change 1468276786
  106. Mar 31 01:08:41 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 999804101: 3->all empty
  107. Mar 31 01:08:41 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 999804101 (primary_nodes=8)
  108. Mar 31 01:08:41 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: received new current UUID: ADB250ED33B11983 weak_nodes=FFFFFFFFFFFFFFFC
  109. Mar 31 01:08:41 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1825593975: 1->all empty
  110. Mar 31 01:08:41 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 1825593975: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF4
  111. Mar 31 01:08:41 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 1825593975 (0ms)
  112. Mar 31 01:08:42 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3548852179: 0->all empty
  113. Mar 31 01:08:42 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 3548852179 (primary_nodes=8)
  114. Mar 31 01:09:24 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 841577807: 0->2 role( Secondary ) conn( Connected )
  115. Mar 31 01:09:24 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 841577807 (primary_nodes=8)
  116. Mar 31 01:09:25 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Handshake to peer 2 successful: Agreed network protocol version 122
  117. Mar 31 01:09:25 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  118. Mar 31 01:09:25 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Peer authenticated using 20 bytes HMAC
  119. Mar 31 01:09:25 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3079894093: 1->2 role( Secondary ) conn( Connected )
  120. Mar 31 01:09:25 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: drbd_sync_handshake:
  121. Mar 31 01:09:25 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: self ADB250ED33B11982:3B36CC1B2C2CCBAC:52F5EAB2DCF34652:28D31133FEBC993A bits:0 flags:100
  122. Mar 31 01:09:25 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: peer 3B36CC1B2C2CCBAC:3B36CC1B2C2CCBAC:52F5EAB2DCF34652:28D31133FEBC993A bits:0 flags:1124
  123. Mar 31 01:09:25 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: uuid_compare()=source-use-bitmap by rule=bitmap-self
  124. Mar 31 01:09:25 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 3079894093: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF4
  125. Mar 31 01:09:25 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 3079894093 (32ms)
  126. Mar 31 01:09:25 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [connected]
  127. Mar 31 01:09:25 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( DUnknown -> Inconsistent ) repl( Off -> WFBitMapS ) [connected]
  128. Mar 31 01:09:25 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 21(1), total 21; compression: 100.0%
  129. Mar 31 01:09:25 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 21(1), total 21; compression: 100.0%
  130. Mar 31 01:09:25 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( WFBitMapS -> SyncSource ) [receive-bitmap]
  131. Mar 31 01:09:25 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: Began resync as SyncSource (will sync 0 KB [0 bits set]).
  132. Mar 31 01:09:25 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: Resync done (total 1 sec; paused 0 sec; 0 K/sec)
  133. Mar 31 01:09:25 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( SyncSource -> Established ) [resync-finished]
  134. Mar 31 01:09:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 97363440: 2->3 role( Secondary ) conn( Connected )
  135. Mar 31 01:09:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 97363440 (primary_nodes=8)
  136. Mar 31 01:09:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: resync-susp( no -> peer ) [peer-state]
  137. Mar 31 01:09:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: updated UUIDs ADB250ED33B11982:0000000000000000:3B36CC1B2C2CCBAC:52F5EAB2DCF34652
  138. Mar 31 01:09:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( Inconsistent -> UpToDate ) resync-susp( peer -> no ) [peer-state]
  139. Mar 31 01:09:46 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: PingAck did not arrive in time.
  140. Mar 31 01:09:46 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  141. Mar 31 01:09:46 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
  142. Mar 31 01:09:46 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1665238630: 1->all empty
  143. Mar 31 01:09:46 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Terminating sender thread
  144. Mar 31 01:09:46 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Starting sender thread (peer-node-id 0)
  145. Mar 31 01:09:46 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Connection closed
  146. Mar 31 01:09:46 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( NetworkFailure -> Unconnected ) [disconnected]
  147. Mar 31 01:09:46 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Restarting receiver thread
  148. Mar 31 01:09:46 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Unconnected -> Connecting ) [connecting]
  149. Mar 31 01:09:46 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: PingAck did not arrive in time.
  150. Mar 31 01:09:46 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )
  151. Mar 31 01:09:46 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( UpToDate -> Consistent )
  152. Mar 31 01:09:46 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
  153. Mar 31 01:09:46 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: PingAck did not arrive in time.
  154. Mar 31 01:09:46 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( no -> quorum )
  155. Mar 31 01:09:46 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  156. Mar 31 01:09:46 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( yes -> no )
  157. Mar 31 01:09:46 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
  158. Mar 31 01:09:46 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 72ms
  159. Mar 31 01:09:46 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Terminating sender thread
  160. Mar 31 01:09:46 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Starting sender thread (peer-node-id 2)
  161. Mar 31 01:09:46 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Terminating sender thread
  162. Mar 31 01:09:46 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Starting sender thread (peer-node-id 3)
  163. Mar 31 01:09:46 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Connection closed
  164. Mar 31 01:09:46 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( NetworkFailure -> Unconnected ) [disconnected]
  165. Mar 31 01:09:46 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Restarting receiver thread
  166. Mar 31 01:09:46 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Unconnected -> Connecting ) [connecting]
  167. Mar 31 01:09:46 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Connection closed
  168. Mar 31 01:09:46 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( NetworkFailure -> Unconnected ) [disconnected]
  169. Mar 31 01:09:46 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Restarting receiver thread
  170. Mar 31 01:09:46 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Unconnected -> Connecting ) [connecting]
  171. Mar 31 01:09:46 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1582917899: 1->all empty
  172. Mar 31 01:09:46 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 1582917899 (0ms)
  173. Mar 31 01:09:46 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Consistent -> UpToDate ) [lost-peer]
  174. Mar 31 01:10:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Handshake to peer 2 successful: Agreed network protocol version 122
  175. Mar 31 01:10:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  176. Mar 31 01:10:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Peer authenticated using 20 bytes HMAC
  177. Mar 31 01:10:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 4229506866: 1->2 role( Secondary ) conn( Connected )
  178. Mar 31 01:10:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: drbd_sync_handshake:
  179. Mar 31 01:10:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: self ADB250ED33B11982:0000000000000000:3B36CC1B2C2CCBAC:52F5EAB2DCF34652 bits:75 flags:120
  180. Mar 31 01:10:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: peer DD95BBD64DA6BD62:ADB250ED33B11982:52F5EAB2DCF34652:28D31133FEBC993A bits:230 flags:1100
  181. Mar 31 01:10:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: uuid_compare()=target-use-bitmap by rule=bitmap-peer
  182. Mar 31 01:10:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 4229506866: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF0
  183. Mar 31 01:10:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 4229506866 (44ms)
  184. Mar 31 01:10:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [connected]
  185. Mar 31 01:10:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( UpToDate -> Outdated ) [connected]
  186. Mar 31 01:10:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( DUnknown -> UpToDate ) repl( Off -> WFBitMapT ) [connected]
  187. Mar 31 01:10:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: Setting exposed data uuid: DD95BBD64DA6BD62
  188. Mar 31 01:10:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 37(1), total 37; compression: 99.9%
  189. Mar 31 01:10:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 37(1), total 37; compression: 99.9%
  190. Mar 31 01:10:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Outdated -> Inconsistent ) [receive-bitmap]
  191. Mar 31 01:10:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: resync-susp( no -> connection dependency ) [receive-bitmap]
  192. Mar 31 01:10:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( WFBitMapT -> SyncTarget ) [receive-bitmap]
  193. Mar 31 01:10:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: resync-susp( no -> connection dependency ) [receive-bitmap]
  194. Mar 31 01:10:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: Began resync as SyncTarget (will sync 1068 KB [267 bits set]).
  195. Mar 31 01:10:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: Resync done (total 1 sec; paused 0 sec; 1068 K/sec)
  196. Mar 31 01:10:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: Peer was unstable during resync
  197. Mar 31 01:10:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: resync-susp( connection dependency -> no ) [resync-finished]
  198. Mar 31 01:10:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( SyncTarget -> Established ) [resync-finished]
  199. Mar 31 01:10:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: resync-susp( connection dependency -> no ) [resync-finished]
  200. Mar 31 01:10:15 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Handshake to peer 0 successful: Agreed network protocol version 122
  201. Mar 31 01:10:15 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  202. Mar 31 01:10:15 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Peer authenticated using 20 bytes HMAC
  203. Mar 31 01:10:15 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 461079150: 0->1 role( Secondary ) conn( Connected )
  204. Mar 31 01:10:15 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: drbd_sync_handshake:
  205. Mar 31 01:10:15 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: self ADB250ED33B11982:ADB250ED33B11982:3B36CC1B2C2CCBAC:52F5EAB2DCF34652 bits:267 flags:124
  206. Mar 31 01:10:15 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: peer DD95BBD64DA6BD62:ADB250ED33B11982:3B36CC1B2C2CCBAC:52F5EAB2DCF34652 bits:192 flags:1100
  207. Mar 31 01:10:15 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: uuid_compare()=target-use-bitmap by rule=bitmap-peer
  208. Mar 31 01:10:15 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 461079150 (primary_nodes=8)
  209. Mar 31 01:10:15 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( quorum -> no ) [remote]
  210. Mar 31 01:10:15 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  211. Mar 31 01:10:15 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( no -> yes ) [remote]
  212. Mar 31 01:10:15 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( DUnknown -> UpToDate ) repl( Off -> WFBitMapT ) [remote]
  213. Mar 31 01:10:15 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 32(1), total 32; compression: 100.0%
  214. Mar 31 01:10:15 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 37(1), total 37; compression: 99.9%
  215. Mar 31 01:10:15 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: resync-susp( no -> connection dependency ) [receive-bitmap]
  216. Mar 31 01:10:15 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: resync-susp( no -> connection dependency ) [receive-bitmap]
  217. Mar 31 01:10:15 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( WFBitMapT -> SyncTarget ) [receive-bitmap]
  218. Mar 31 01:10:15 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: Began resync as SyncTarget (will sync 1068 KB [267 bits set]).
  219. Mar 31 01:10:15 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: Resync done (total 1 sec; paused 0 sec; 1068 K/sec)
  220. Mar 31 01:10:15 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: Peer was unstable during resync
  221. Mar 31 01:10:15 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: resync-susp( connection dependency -> no ) [resync-finished]
  222. Mar 31 01:10:15 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: resync-susp( connection dependency -> no ) [resync-finished]
  223. Mar 31 01:10:15 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( SyncTarget -> Established ) [resync-finished]
  224. Mar 31 01:10:16 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Handshake to peer 3 successful: Agreed network protocol version 122
  225. Mar 31 01:10:16 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  226. Mar 31 01:10:16 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Peer authenticated using 20 bytes HMAC
  227. Mar 31 01:10:16 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 4185765531: 1->3 role( Secondary ) conn( Connected )
  228. Mar 31 01:10:16 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: self ADB250ED33B11982:28D31133FEBC993A:3B36CC1B2C2CCBAC:52F5EAB2DCF34652 bits:0 flags:0
  229. Mar 31 01:10:16 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: peer's exposed UUID: DD95BBD64DA6BD63
  230. Mar 31 01:10:16 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 4185765531: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF0
  231. Mar 31 01:10:16 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 4185765531 (32ms)
  232. Mar 31 01:10:16 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connecting -> Connected ) peer( Unknown -> Primary ) [connected]
  233. Mar 31 01:10:16 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( DUnknown -> Diskless ) repl( Off -> Established ) [connected]
  234. Mar 31 01:10:16 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: strategy = target-use-bitmap
  235. Mar 31 01:10:16 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: strategy = target-use-bitmap
  236. Mar 31 01:10:16 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: drbd_sync_handshake:
  237. Mar 31 01:10:16 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: self ADB250ED33B11982:0000000000000000:3B36CC1B2C2CCBAC:52F5EAB2DCF34652 bits:267 flags:104
  238. Mar 31 01:10:16 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: peer DD95BBD64DA6BD62:ADB250ED33B11982:52F5EAB2DCF34652:28D31133FEBC993A bits:230 flags:1100
  239. Mar 31 01:10:16 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: uuid_compare()=target-use-bitmap by rule=bitmap-peer
  240. Mar 31 01:10:16 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: Becoming WFBitMapT because primary is diskless
  241. Mar 31 01:10:16 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( Established -> WFBitMapT ) [diskless-primary]
  242. Mar 31 01:10:16 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 21(1), total 21; compression: 100.0%
  243. Mar 31 01:10:16 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 37(1), total 37; compression: 99.9%
  244. Mar 31 01:10:16 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: resync-susp( no -> connection dependency ) [receive-bitmap]
  245. Mar 31 01:10:16 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( WFBitMapT -> SyncTarget ) [receive-bitmap]
  246. Mar 31 01:10:16 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: resync-susp( no -> connection dependency ) [receive-bitmap]
  247. Mar 31 01:10:16 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: Began resync as SyncTarget (will sync 1068 KB [267 bits set]).
  248. Mar 31 01:10:16 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: Resync done (total 1 sec; paused 0 sec; 1068 K/sec)
  249. Mar 31 01:10:16 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: updated UUIDs DD95BBD64DA6BD62:0000000000000000:3B36CC1B2C2CCBAC:52F5EAB2DCF34652
  250. Mar 31 01:10:16 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Inconsistent -> UpToDate ) [resync-finished]
  251. Mar 31 01:10:16 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: resync-susp( connection dependency -> no ) [resync-finished]
  252. Mar 31 01:10:16 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( SyncTarget -> Established ) [resync-finished]
  253. Mar 31 01:10:16 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: resync-susp( connection dependency -> no ) [resync-finished]
  254. Mar 31 01:10:42 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4191436868: 0->all empty
  255. Mar 31 01:10:42 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( UpToDate -> Consistent ) [peer-state]
  256. Mar 31 01:10:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: PingAck did not arrive in time.
  257. Mar 31 01:10:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )
  258. Mar 31 01:10:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( UpToDate -> Consistent )
  259. Mar 31 01:10:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
  260. Mar 31 01:10:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: My uuid_flags changed from 0x100 to 0x120 during handshake.
  261. Mar 31 01:10:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Terminating sender thread
  262. Mar 31 01:10:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Starting sender thread (peer-node-id 3)
  263. Mar 31 01:10:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Connection closed
  264. Mar 31 01:10:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( no -> quorum ) [disconnected]
  265. Mar 31 01:10:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( NetworkFailure -> Unconnected ) [disconnected]
  266. Mar 31 01:10:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( yes -> no ) [disconnected]
  267. Mar 31 01:10:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Restarting receiver thread
  268. Mar 31 01:10:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Unconnected -> Connecting ) [connecting]
  269. Mar 31 01:10:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: No reconciliation resync even though 'storage-load-test-0' disappeared. (o=0)
  270. Mar 31 01:10:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Aborting remote state change 4191436868
  271. Mar 31 01:10:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 595626168: 1->all empty
  272. Mar 31 01:10:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( UpToDate -> Consistent ) [peer-state]
  273. Mar 31 01:10:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( Established -> WFBitMapS ) [receive-peer-dagtag]
  274. Mar 31 01:10:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Reconciliation resync because 'storage-load-test-0' disappeared. (o=16)
  275. Mar 31 01:10:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 30(1), total 30; compression: 100.0%
  276. Mar 31 01:10:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Rejecting concurrent remote state change 1771109539 because of state change 595626168
  277. Mar 31 01:10:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 595626168: primary_nodes=0, weak_nodes=0
  278. Mar 31 01:10:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 595626168 (56ms)
  279. Mar 31 01:10:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Consistent -> UpToDate ) [lost-peer]
  280. Mar 31 01:10:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 871012577: 1->all empty
  281. Mar 31 01:10:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( quorum -> no ) [peer-state]
  282. Mar 31 01:10:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( no -> yes ) [peer-state]
  283. Mar 31 01:10:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( Consistent -> UpToDate ) [peer-state]
  284. Mar 31 01:10:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( Consistent -> Outdated ) [peer-state]
  285. Mar 31 01:10:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 871012577: primary_nodes=0, weak_nodes=0
  286. Mar 31 01:10:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 871012577 (16ms)
  287. Mar 31 01:10:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 30(1), total 30; compression: 100.0%
  288. Mar 31 01:10:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( Outdated -> Inconsistent ) repl( WFBitMapS -> SyncSource ) [receive-bitmap]
  289. Mar 31 01:10:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: Began resync as SyncSource (will sync 148 KB [37 bits set]).
  290. Mar 31 01:10:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: Resync done (total 1 sec; paused 0 sec; 148 K/sec)
  291. Mar 31 01:10:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( Inconsistent -> UpToDate ) repl( SyncSource -> Established ) [resync-finished]
  292. Mar 31 01:10:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3701694474: 0->all empty
  293. Mar 31 01:10:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 3701694474 (primary_nodes=0)
  294. Mar 31 01:10:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1033669755: 0->all empty
  295. Mar 31 01:10:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 1033669755 (primary_nodes=0)
  296. Mar 31 01:10:44 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1413252624: 2->all empty
  297. Mar 31 01:10:44 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 1413252624 (primary_nodes=0)
  298. Mar 31 01:10:44 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2834901056: 2->all empty
  299. Mar 31 01:10:44 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 2834901056 (primary_nodes=0)
  300. Mar 31 01:11:12 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Handshake to peer 3 successful: Agreed network protocol version 122
  301. Mar 31 01:11:12 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  302. Mar 31 01:11:12 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Peer authenticated using 20 bytes HMAC
  303. Mar 31 01:11:12 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1914803476: 1->3 role( Secondary ) conn( Connected )
  304. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: self DD95BBD64DA6BD62:28D31133FEBC993A:3B36CC1B2C2CCBAC:52F5EAB2DCF34652 bits:0 flags:0
  305. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: peer's exposed UUID: DD95BBD64DA6BD63
  306. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 1914803476: primary_nodes=8, weak_nodes=0
  307. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 1914803476 (28ms)
  308. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connecting -> Connected ) peer( Unknown -> Primary ) [connected]
  309. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( DUnknown -> Diskless ) repl( Off -> Established ) [connected]
  310. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: received new current UUID: DD95BBD64DA6BD63 weak_nodes=FFFFFFFFFFFFFFFD
  311. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3587873295: 3->all empty
  312. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 3587873295 (primary_nodes=8)
  313. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: peer request with dagtag 9344 not found
  314. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )
  315. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( UpToDate -> Consistent )
  316. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
  317. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3330819532: 1->all empty
  318. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 3330819532: primary_nodes=0, weak_nodes=0
  319. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( no -> quorum ) [peer-state]
  320. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 3330819532 (0ms)
  321. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( yes -> no ) [peer-state]
  322. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( UpToDate -> Outdated ) [peer-state]
  323. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( quorum -> no ) [lost-peer]
  324. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Consistent -> UpToDate ) quorum( no -> yes ) [lost-peer]
  325. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( no -> quorum ) [peer-state]
  326. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( yes -> no ) [peer-state]
  327. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( UpToDate -> Outdated ) [peer-state]
  328. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1043972389: 1->all empty
  329. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Terminating sender thread
  330. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Starting sender thread (peer-node-id 3)
  331. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: updated UUIDs DD95BBD64DA6BD62:0000000000000000:3B36CC1B2C2CCBAC:52F5EAB2DCF34652
  332. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( Outdated -> Consistent ) [peer-state]
  333. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Connection closed
  334. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( NetworkFailure -> Unconnected ) [disconnected]
  335. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Restarting receiver thread
  336. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Unconnected -> Connecting ) [connecting]
  337. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: My bitmap UUID changed during handshake. 0xDD95BBD64DA6BD62 to 0x0
  338. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( Outdated -> Consistent ) [peer-state]
  339. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 1043972389: primary_nodes=0, weak_nodes=0
  340. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 1043972389 (40ms)
  341. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( quorum -> no ) [peer-state]
  342. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( no -> yes ) [peer-state]
  343. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( Consistent -> UpToDate ) [peer-state]
  344. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( Consistent -> UpToDate ) [peer-state]
  345. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( UpToDate -> Outdated ) [peer-state]
  346. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( no -> quorum ) [peer-state]
  347. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( yes -> no ) [peer-state]
  348. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( UpToDate -> Outdated ) [peer-state]
  349. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Handshake to peer 3 successful: Agreed network protocol version 122
  350. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  351. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Peer authenticated using 20 bytes HMAC
  352. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1142912854: 1->3 role( Secondary ) conn( Connected )
  353. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: self DD95BBD64DA6BD62:28D31133FEBC993A:3B36CC1B2C2CCBAC:52F5EAB2DCF34652 bits:0 flags:0
  354. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: peer's exposed UUID: DD95BBD64DA6BD63
  355. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 1142912854: primary_nodes=8, weak_nodes=0
  356. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 1142912854 (40ms)
  357. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connecting -> Connected ) peer( Unknown -> Primary ) [connected]
  358. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( DUnknown -> Diskless ) repl( Off -> Established ) [connected]
  359. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: received new current UUID: DD95BBD64DA6BD63 weak_nodes=FFFFFFFFFFFFFFFD
  360. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1970256621: 3->all empty
  361. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 1970256621 (primary_nodes=8)
  362. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1547400645: 2->3 role( Secondary ) conn( Connected )
  363. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 1547400645 (primary_nodes=8)
  364. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: clearing bitmap UUID and content (5 bits) for node 0 (storage-load-test-2)(slot 1)
  365. Mar 31 01:11:13 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: updated UUIDs DD95BBD64DA6BD62:0000000000000000:3B36CC1B2C2CCBAC:52F5EAB2DCF34652
  366. Mar 31 01:11:16 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1713475212: 0->3 role( Secondary ) conn( Connected )
  367. Mar 31 01:11:16 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 1713475212 (primary_nodes=8)
  368. Mar 31 01:11:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: meta connection shut down by peer.
  369. Mar 31 01:11:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  370. Mar 31 01:11:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( Established -> Off )
  371. Mar 31 01:11:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1652681402: 0->all empty
  372. Mar 31 01:11:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Terminating sender thread
  373. Mar 31 01:11:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Starting sender thread (peer-node-id 0)
  374. Mar 31 01:11:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Aborting remote state change 1652681402
  375. Mar 31 01:11:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4152775671: 2->all empty
  376. Mar 31 01:11:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Connection closed
  377. Mar 31 01:11:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( NetworkFailure -> Unconnected ) [disconnected]
  378. Mar 31 01:11:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Restarting receiver thread
  379. Mar 31 01:11:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Unconnected -> Connecting ) [connecting]
  380. Mar 31 01:11:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 4152775671 (primary_nodes=8)
  381. Mar 31 01:11:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 797125860: 1->all empty
  382. Mar 31 01:11:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Rejecting concurrent remote state change 1902223197 because of state change 797125860
  383. Mar 31 01:11:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 797125860: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF1
  384. Mar 31 01:11:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 797125860 (0ms)
  385. Mar 31 01:11:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3924326760: 3->all empty
  386. Mar 31 01:11:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 3924326760 (primary_nodes=8)
  387. Mar 31 01:12:03 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2217267134: 0->2 role( Secondary ) conn( Connected )
  388. Mar 31 01:12:03 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 2217267134 (primary_nodes=8)
  389. Mar 31 01:12:08 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2130346017: 0->3 role( Secondary ) conn( Connected )
  390. Mar 31 01:12:08 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Duplicate prepare for remote state change 2130346017
  391. Mar 31 01:12:08 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 2130346017 (primary_nodes=8)
  392. Mar 31 01:12:11 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Handshake to peer 0 successful: Agreed network protocol version 122
  393. Mar 31 01:12:11 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  394. Mar 31 01:12:11 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Peer authenticated using 20 bytes HMAC
  395. Mar 31 01:12:11 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1527465761: 0->1 role( Secondary ) conn( Connected )
  396. Mar 31 01:12:11 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: drbd_sync_handshake:
  397. Mar 31 01:12:11 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: self DD95BBD64DA6BD62:0000000000000000:3B36CC1B2C2CCBAC:52F5EAB2DCF34652 bits:0 flags:100
  398. Mar 31 01:12:11 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: peer DD95BBD64DA6BD62:DD95BBD64DA6BD63:ADB250ED33B11982:3B36CC1B2C2CCBAC bits:0 flags:1100
  399. Mar 31 01:12:11 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: uuid_compare()=no-sync by rule=reconnected
  400. Mar 31 01:12:11 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 1527465761 (primary_nodes=8)
  401. Mar 31 01:12:11 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( quorum -> no ) [remote]
  402. Mar 31 01:12:11 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  403. Mar 31 01:12:11 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( no -> yes ) [remote]
  404. Mar 31 01:12:11 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( Outdated -> UpToDate ) repl( Off -> Established ) [remote]
  405. Mar 31 01:12:11 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: cleared bm UUID and bitmap DD95BBD64DA6BD62:0000000000000000:3B36CC1B2C2CCBAC:52F5EAB2DCF34652
  406. Mar 31 01:12:11 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: bitmap WRITE of 2 pages took 8 ms
  407. Mar 31 01:12:24 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1389696837: 3->all empty
  408. Mar 31 01:12:24 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Aborting remote state change 1389696837
  409. Mar 31 01:12:24 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2149326564: 3->all empty
  410. Mar 31 01:12:24 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Aborting remote state change 2149326564
  411. Mar 31 01:12:25 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: meta connection shut down by peer.
  412. Mar 31 01:12:25 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  413. Mar 31 01:12:25 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( Established -> Off )
  414. Mar 31 01:12:25 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3714550546: 1->all empty
  415. Mar 31 01:12:25 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Terminating sender thread
  416. Mar 31 01:12:25 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Starting sender thread (peer-node-id 2)
  417. Mar 31 01:12:25 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Connection closed
  418. Mar 31 01:12:25 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( NetworkFailure -> Unconnected ) [disconnected]
  419. Mar 31 01:12:25 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Restarting receiver thread
  420. Mar 31 01:12:25 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Unconnected -> Connecting ) [connecting]
  421. Mar 31 01:12:25 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 268ms
  422. Mar 31 01:12:25 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2301440963: 0->all empty
  423. Mar 31 01:12:25 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 2301440963 (primary_nodes=8)
  424. Mar 31 01:12:25 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3306662247: 1->all empty
  425. Mar 31 01:12:25 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 3306662247: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF4
  426. Mar 31 01:12:25 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 3306662247 (0ms)
  427. Mar 31 01:12:25 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4059018192: 3->all empty
  428. Mar 31 01:12:25 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 4059018192 (primary_nodes=8)
  429. Mar 31 01:12:47 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4282827581: 2->3 role( Secondary ) conn( Connected )
  430. Mar 31 01:12:47 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 4282827581 (primary_nodes=8)
  431. Mar 31 01:12:47 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3429861694: 0->2 role( Secondary ) conn( Connected )
  432. Mar 31 01:12:47 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Handshake to peer 2 successful: Agreed network protocol version 122
  433. Mar 31 01:12:47 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  434. Mar 31 01:12:47 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Peer authenticated using 20 bytes HMAC
  435. Mar 31 01:12:47 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 3429861694 (primary_nodes=8)
  436. Mar 31 01:12:47 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1152771459: 1->2 role( Secondary ) conn( Connected )
  437. Mar 31 01:12:47 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: drbd_sync_handshake:
  438. Mar 31 01:12:47 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: self DD95BBD64DA6BD62:0000000000000000:3B36CC1B2C2CCBAC:52F5EAB2DCF34652 bits:315 flags:100
  439. Mar 31 01:12:47 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: peer DD95BBD64DA6BD62:DD95BBD64DA6BD63:3B36CC1B2C2CCBAC:ADB250ED33B11982 bits:0 flags:1100
  440. Mar 31 01:12:47 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: uuid_compare()=no-sync by rule=reconnected
  441. Mar 31 01:12:47 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 1152771459: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF0
  442. Mar 31 01:12:47 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 1152771459 (60ms)
  443. Mar 31 01:12:47 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [connected]
  444. Mar 31 01:12:47 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( Outdated -> UpToDate ) repl( Off -> Established ) [connected]
  445. Mar 31 01:12:47 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: cleared bm UUID and bitmap DD95BBD64DA6BD62:0000000000000000:3B36CC1B2C2CCBAC:52F5EAB2DCF34652
  446. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3292241375: 0->all empty
  447. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( UpToDate -> Consistent ) [peer-state]
  448. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: PingAck did not arrive in time.
  449. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )
  450. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( UpToDate -> Consistent )
  451. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
  452. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: My uuid_flags changed from 0x100 to 0x120 during handshake.
  453. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Terminating sender thread
  454. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Starting sender thread (peer-node-id 3)
  455. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Connection closed
  456. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( no -> quorum ) [disconnected]
  457. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( NetworkFailure -> Unconnected ) [disconnected]
  458. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( yes -> no ) [disconnected]
  459. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Restarting receiver thread
  460. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Unconnected -> Connecting ) [connecting]
  461. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( Established -> WFBitMapS ) [receive-peer-dagtag]
  462. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Reconciliation resync because 'storage-load-test-0' disappeared. (o=208)
  463. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( UpToDate -> Consistent ) [peer-state]
  464. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 30(1), total 30; compression: 100.0%
  465. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Aborting remote state change 3292241375
  466. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( Consistent -> Outdated ) [peer-state]
  467. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 4119964504: 1->all empty
  468. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 30(1), total 30; compression: 100.0%
  469. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( Outdated -> Inconsistent ) repl( WFBitMapS -> SyncSource ) [receive-bitmap]
  470. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( Established -> WFBitMapS ) [receive-peer-dagtag]
  471. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Reconciliation resync because 'storage-load-test-0' disappeared. (o=208)
  472. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: Began resync as SyncSource (will sync 132 KB [33 bits set]).
  473. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 30(1), total 30; compression: 100.0%
  474. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( Consistent -> Outdated ) [peer-state]
  475. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 4119964504: primary_nodes=0, weak_nodes=0
  476. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 4119964504 (40ms)
  477. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Consistent -> UpToDate ) [lost-peer]
  478. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: Resync done (total 1 sec; paused 0 sec; 132 K/sec)
  479. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( Inconsistent -> Consistent ) repl( SyncSource -> Established ) [resync-finished]
  480. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4075443310: 2->all empty
  481. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Aborting remote state change 4075443310
  482. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 30(1), total 30; compression: 100.0%
  483. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( Outdated -> Inconsistent ) repl( WFBitMapS -> SyncSource ) [receive-bitmap]
  484. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: Began resync as SyncSource (will sync 132 KB [33 bits set]).
  485. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( quorum -> no ) [peer-state]
  486. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( no -> yes ) [peer-state]
  487. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( Consistent -> UpToDate ) [peer-state]
  488. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: Resync done (total 1 sec; paused 0 sec; 132 K/sec)
  489. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( Inconsistent -> UpToDate ) repl( SyncSource -> Established ) [resync-finished]
  490. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3079372999: 2->all empty
  491. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 3079372999 (primary_nodes=0)
  492. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2335269304: 2->all empty
  493. Mar 31 01:13:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 2335269304 (primary_nodes=0)
  494. Mar 31 01:13:06 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1444156437: 0->all empty
  495. Mar 31 01:13:06 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 1444156437 (primary_nodes=0)
  496. Mar 31 01:13:06 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3261655757: 0->all empty
  497. Mar 31 01:13:06 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 3261655757 (primary_nodes=0)
  498. Mar 31 01:13:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Handshake to peer 3 successful: Agreed network protocol version 122
  499. Mar 31 01:13:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  500. Mar 31 01:13:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Peer authenticated using 20 bytes HMAC
  501. Mar 31 01:13:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3125836916: 1->3 role( Secondary ) conn( Connected )
  502. Mar 31 01:13:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: self DD95BBD64DA6BD62:28D31133FEBC993A:3B36CC1B2C2CCBAC:52F5EAB2DCF34652 bits:0 flags:0
  503. Mar 31 01:13:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: peer's exposed UUID: DD95BBD64DA6BD63
  504. Mar 31 01:13:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 3125836916: primary_nodes=8, weak_nodes=0
  505. Mar 31 01:13:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 3125836916 (36ms)
  506. Mar 31 01:13:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connecting -> Connected ) peer( Unknown -> Primary ) [connected]
  507. Mar 31 01:13:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( DUnknown -> Diskless ) repl( Off -> Established ) [connected]
  508. Mar 31 01:13:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: received new current UUID: DD95BBD64DA6BD63 weak_nodes=FFFFFFFFFFFFFFFD
  509. Mar 31 01:13:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2402789986: 3->all empty
  510. Mar 31 01:13:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 2402789986 (primary_nodes=8)
  511. Mar 31 01:13:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( UpToDate -> Outdated ) [peer-state]
  512. Mar 31 01:13:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( no -> quorum ) [peer-state]
  513. Mar 31 01:13:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( yes -> no ) [peer-state]
  514. Mar 31 01:13:43 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( UpToDate -> Outdated ) [peer-state]
  515. Mar 31 01:13:46 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1570761256: 0->3 role( Secondary ) conn( Connected )
  516. Mar 31 01:13:46 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Aborting remote state change 1570761256
  517. Mar 31 01:13:47 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3138477124: 0->3 role( Secondary ) conn( Connected )
  518. Mar 31 01:13:47 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Aborting remote state change 3138477124
  519. Mar 31 01:13:48 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3535667735: 0->3 role( Secondary ) conn( Connected )
  520. Mar 31 01:13:48 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Aborting remote state change 3535667735
  521. Mar 31 01:13:48 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2072727487: 2->3 role( Secondary ) conn( Connected )
  522. Mar 31 01:13:48 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Aborting remote state change 2072727487
  523. Mar 31 01:13:49 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3417612: 0->3 role( Secondary ) conn( Connected )
  524. Mar 31 01:13:49 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Aborting remote state change 3417612
  525. Mar 31 01:13:49 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 204289545: 2->3 role( Secondary ) conn( Connected )
  526. Mar 31 01:13:49 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Aborting remote state change 204289545
  527. Mar 31 01:13:50 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2615676757: 0->3 role( Secondary ) conn( Connected )
  528. Mar 31 01:13:50 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Aborting remote state change 2615676757
  529. Mar 31 01:13:50 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2561739627: 2->3 role( Secondary ) conn( Connected )
  530. Mar 31 01:13:50 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Aborting remote state change 2561739627
  531. Mar 31 01:13:52 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: meta connection shut down by peer.
  532. Mar 31 01:13:52 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  533. Mar 31 01:13:52 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( Established -> Off )
  534. Mar 31 01:13:52 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3246052411: 1->all empty
  535. Mar 31 01:13:52 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Terminating sender thread
  536. Mar 31 01:13:52 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Starting sender thread (peer-node-id 0)
  537. Mar 31 01:13:52 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 100ms
  538. Mar 31 01:13:52 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Connection closed
  539. Mar 31 01:13:52 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( NetworkFailure -> Unconnected ) [disconnected]
  540. Mar 31 01:13:52 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Restarting receiver thread
  541. Mar 31 01:13:52 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Unconnected -> Connecting ) [connecting]
  542. Mar 31 01:13:52 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3276519458: 1->all empty
  543. Mar 31 01:13:52 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 1128ms
  544. Mar 31 01:13:52 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3115004085: 0->3 role( Secondary ) conn( Connected )
  545. Mar 31 01:13:52 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: meta connection shut down by peer.
  546. Mar 31 01:13:52 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  547. Mar 31 01:13:52 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( Established -> Off )
  548. Mar 31 01:13:52 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Terminating sender thread
  549. Mar 31 01:13:52 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Starting sender thread (peer-node-id 2)
  550. Mar 31 01:13:52 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Aborting remote state change 3115004085
  551. Mar 31 01:13:52 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Connection closed
  552. Mar 31 01:13:52 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( NetworkFailure -> Unconnected ) [disconnected]
  553. Mar 31 01:13:52 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Restarting receiver thread
  554. Mar 31 01:13:52 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Unconnected -> Connecting ) [connecting]
  555. Mar 31 01:13:52 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: PingAck did not arrive in time.
  556. Mar 31 01:13:52 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )
  557. Mar 31 01:13:52 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( UpToDate -> Consistent )
  558. Mar 31 01:13:52 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
  559. Mar 31 01:13:52 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Terminating sender thread
  560. Mar 31 01:13:52 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Starting sender thread (peer-node-id 3)
  561. Mar 31 01:13:52 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Connection closed
  562. Mar 31 01:13:52 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( NetworkFailure -> Unconnected ) [disconnected]
  563. Mar 31 01:13:52 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Restarting receiver thread
  564. Mar 31 01:13:52 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Unconnected -> Connecting ) [connecting]
  565. Mar 31 01:13:53 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2989697482: 1->all empty
  566. Mar 31 01:13:53 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 2989697482 (0ms)
  567. Mar 31 01:13:53 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Consistent -> UpToDate ) [lost-peer]
  568. Mar 31 01:14:34 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Handshake to peer 3 successful: Agreed network protocol version 122
  569. Mar 31 01:14:34 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  570. Mar 31 01:14:34 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Peer authenticated using 20 bytes HMAC
  571. Mar 31 01:14:35 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3446771404: 1->3 role( Secondary ) conn( Connected )
  572. Mar 31 01:14:35 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: self DD95BBD64DA6BD62:28D31133FEBC993A:3B36CC1B2C2CCBAC:52F5EAB2DCF34652 bits:0 flags:0
  573. Mar 31 01:14:35 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: peer's exposed UUID: DD95BBD64DA6BD63
  574. Mar 31 01:14:35 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 3446771404: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF0
  575. Mar 31 01:14:35 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 3446771404 (24ms)
  576. Mar 31 01:14:35 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connecting -> Connected ) peer( Unknown -> Primary ) [connected]
  577. Mar 31 01:14:35 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( DUnknown -> Diskless ) repl( Off -> Established ) [connected]
  578. Mar 31 01:14:35 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: received new current UUID: DD95BBD64DA6BD63 weak_nodes=FFFFFFFFFFFFFFF8
  579. Mar 31 01:14:36 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Handshake to peer 0 successful: Agreed network protocol version 122
  580. Mar 31 01:14:36 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  581. Mar 31 01:14:36 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Peer authenticated using 20 bytes HMAC
  582. Mar 31 01:14:36 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 864936807: 0->1 role( Secondary ) conn( Connected )
  583. Mar 31 01:14:36 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: drbd_sync_handshake:
  584. Mar 31 01:14:36 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: self DD95BBD64DA6BD62:DD95BBD64DA6BD63:3B36CC1B2C2CCBAC:52F5EAB2DCF34652 bits:50 flags:100
  585. Mar 31 01:14:36 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: peer DD95BBD64DA6BD62:DD95BBD64DA6BD62:ADB250ED33B11982:3B36CC1B2C2CCBAC bits:342 flags:1100
  586. Mar 31 01:14:36 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: uuid_compare()=no-sync by rule=reconnected
  587. Mar 31 01:14:36 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 864936807 (primary_nodes=8)
  588. Mar 31 01:14:36 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( quorum -> no ) [remote]
  589. Mar 31 01:14:36 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  590. Mar 31 01:14:36 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( no -> yes ) [remote]
  591. Mar 31 01:14:36 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( Outdated -> UpToDate ) repl( Off -> Established ) [remote]
  592. Mar 31 01:14:36 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: cleared bm UUID and bitmap DD95BBD64DA6BD62:0000000000000000:3B36CC1B2C2CCBAC:52F5EAB2DCF34652
  593. Mar 31 01:14:36 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: bitmap WRITE of 3 pages took 8 ms
  594. Mar 31 01:14:38 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Handshake to peer 2 successful: Agreed network protocol version 122
  595. Mar 31 01:14:38 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  596. Mar 31 01:14:38 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Peer authenticated using 20 bytes HMAC
  597. Mar 31 01:14:38 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1444235534: 1->2 role( Secondary ) conn( Connected )
  598. Mar 31 01:14:38 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: drbd_sync_handshake:
  599. Mar 31 01:14:38 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: self DD95BBD64DA6BD62:DD95BBD64DA6BD63:3B36CC1B2C2CCBAC:52F5EAB2DCF34652 bits:50 flags:100
  600. Mar 31 01:14:38 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: peer DD95BBD64DA6BD62:0000000000000000:3B36CC1B2C2CCBAC:ADB250ED33B11982 bits:342 flags:1100
  601. Mar 31 01:14:38 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: uuid_compare()=no-sync by rule=reconnected
  602. Mar 31 01:14:38 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 1444235534: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF0
  603. Mar 31 01:14:38 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 1444235534 (44ms)
  604. Mar 31 01:14:38 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [connected]
  605. Mar 31 01:14:38 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( Outdated -> UpToDate ) repl( Off -> Established ) [connected]
  606. Mar 31 01:14:38 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: cleared bm UUID and bitmap DD95BBD64DA6BD62:0000000000000000:3B36CC1B2C2CCBAC:52F5EAB2DCF34652
  607. Mar 31 01:14:38 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: bitmap WRITE of 3 pages took 8 ms
  608. Mar 31 01:14:56 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3111857493: 2->all empty
  609. Mar 31 01:14:56 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 3111857493 (primary_nodes=8)
  610. Mar 31 01:14:57 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1421634337: 3->all empty
  611. Mar 31 01:14:57 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: meta connection shut down by peer.
  612. Mar 31 01:14:57 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  613. Mar 31 01:14:57 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
  614. Mar 31 01:14:57 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Terminating sender thread
  615. Mar 31 01:14:57 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Aborting remote state change 1421634337
  616. Mar 31 01:14:57 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3701488114: 1->all empty
  617. Mar 31 01:14:57 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Starting sender thread (peer-node-id 0)
  618. Mar 31 01:14:57 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 3701488114: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF1
  619. Mar 31 01:14:57 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 3701488114 (0ms)
  620. Mar 31 01:14:57 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Connection closed
  621. Mar 31 01:14:57 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( NetworkFailure -> Unconnected ) [disconnected]
  622. Mar 31 01:14:57 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Restarting receiver thread
  623. Mar 31 01:14:57 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Unconnected -> Connecting ) [connecting]
  624. Mar 31 01:14:57 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 216853111: 3->all empty
  625. Mar 31 01:14:57 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 216853111 (primary_nodes=8)
  626. Mar 31 01:14:59 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: received new current UUID: D8796B7E2A90EF4D weak_nodes=FFFFFFFFFFFFFFF9
  627. Mar 31 01:15:16 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1653285823: 0->3 role( Secondary ) conn( Connected )
  628. Mar 31 01:15:16 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 1653285823 (primary_nodes=8)
  629. Mar 31 01:15:16 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Handshake to peer 0 successful: Agreed network protocol version 122
  630. Mar 31 01:15:16 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  631. Mar 31 01:15:16 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Peer authenticated using 20 bytes HMAC
  632. Mar 31 01:15:16 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3579449599: 0->1 role( Secondary ) conn( Connected )
  633. Mar 31 01:15:16 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: drbd_sync_handshake:
  634. Mar 31 01:15:16 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: self D8796B7E2A90EF4C:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC:52F5EAB2DCF34652 bits:152 flags:100
  635. Mar 31 01:15:16 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: peer DD95BBD64DA6BD62:0000000000000000:ADB250ED33B11982:3B36CC1B2C2CCBAC bits:0 flags:1100
  636. Mar 31 01:15:16 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: uuid_compare()=source-use-bitmap by rule=bitmap-self
  637. Mar 31 01:15:17 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 3579449599 (primary_nodes=8)
  638. Mar 31 01:15:17 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  639. Mar 31 01:15:17 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( DUnknown -> Outdated ) repl( Off -> WFBitMapS ) [remote]
  640. Mar 31 01:15:17 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 31(1), total 31; compression: 100.0%
  641. Mar 31 01:15:17 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 31(1), total 31; compression: 100.0%
  642. Mar 31 01:15:17 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( Outdated -> Inconsistent ) repl( WFBitMapS -> SyncSource ) [receive-bitmap]
  643. Mar 31 01:15:17 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: Began resync as SyncSource (will sync 608 KB [152 bits set]).
  644. Mar 31 01:15:17 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: updated UUIDs D8796B7E2A90EF4C:0000000000000000:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC
  645. Mar 31 01:15:17 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: Resync done (total 1 sec; paused 0 sec; 608 K/sec)
  646. Mar 31 01:15:17 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( Inconsistent -> UpToDate ) repl( SyncSource -> Established ) [resync-finished]
  647. Mar 31 01:15:19 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2130924816: 0->2 role( Secondary ) conn( Connected )
  648. Mar 31 01:15:19 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 2130924816 (primary_nodes=8)
  649. Mar 31 01:15:29 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4158750261: 0->all empty
  650. Mar 31 01:15:30 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: PingAck did not arrive in time.
  651. Mar 31 01:15:30 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  652. Mar 31 01:15:30 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
  653. Mar 31 01:15:30 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Terminating sender thread
  654. Mar 31 01:15:30 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Starting sender thread (peer-node-id 2)
  655. Mar 31 01:15:30 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Aborting remote state change 4158750261
  656. Mar 31 01:15:30 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2259277806: 1->all empty
  657. Mar 31 01:15:30 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 2259277806: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF4
  658. Mar 31 01:15:30 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 2259277806 (0ms)
  659. Mar 31 01:15:30 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2173774204: 3->all empty
  660. Mar 31 01:15:30 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Aborting remote state change 2173774204
  661. Mar 31 01:15:30 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Connection closed
  662. Mar 31 01:15:30 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( NetworkFailure -> Unconnected ) [disconnected]
  663. Mar 31 01:15:30 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Restarting receiver thread
  664. Mar 31 01:15:30 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Unconnected -> Connecting ) [connecting]
  665. Mar 31 01:15:30 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: received new current UUID: 752996FB4D06CB51 weak_nodes=FFFFFFFFFFFFFFFC
  666. Mar 31 01:15:30 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 567398661: 0->all empty
  667. Mar 31 01:15:30 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 567398661 (primary_nodes=8)
  668. Mar 31 01:15:30 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3559347635: 3->all empty
  669. Mar 31 01:15:30 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 3559347635 (primary_nodes=8)
  670. Mar 31 01:16:01 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1631962529: 2->3 role( Secondary ) conn( Connected )
  671. Mar 31 01:16:02 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 1631962529 (primary_nodes=8)
  672. Mar 31 01:16:02 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3369737057: 0->2 role( Secondary ) conn( Connected )
  673. Mar 31 01:16:02 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 3369737057 (primary_nodes=8)
  674. Mar 31 01:16:02 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: clearing bitmap UUID and content (77 bits) for node 2 (storage-load-test-3)(slot 0)
  675. Mar 31 01:16:04 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Handshake to peer 2 successful: Agreed network protocol version 122
  676. Mar 31 01:16:04 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  677. Mar 31 01:16:04 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Peer authenticated using 20 bytes HMAC
  678. Mar 31 01:16:04 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2284281406: 1->2 role( Secondary ) conn( Connected )
  679. Mar 31 01:16:04 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: drbd_sync_handshake:
  680. Mar 31 01:16:04 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: self 752996FB4D06CB50:0000000000000000:D8796B7E2A90EF4C:DD95BBD64DA6BD62 bits:0 flags:100
  681. Mar 31 01:16:04 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: peer 752996FB4D06CB50:0000000000000000:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC bits:0 flags:1100
  682. Mar 31 01:16:04 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: uuid_compare()=no-sync by rule=reconnected
  683. Mar 31 01:16:04 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 2284281406: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF0
  684. Mar 31 01:16:04 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 2284281406 (56ms)
  685. Mar 31 01:16:04 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [connected]
  686. Mar 31 01:16:04 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( DUnknown -> UpToDate ) repl( Off -> Established ) [connected]
  687. Mar 31 01:16:04 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: cleared bm UUID and bitmap 752996FB4D06CB50:0000000000000000:D8796B7E2A90EF4C:DD95BBD64DA6BD62
  688. Mar 31 01:16:26 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1388789927: 0->all empty
  689. Mar 31 01:16:26 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( UpToDate -> Consistent ) [peer-state]
  690. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: PingAck did not arrive in time.
  691. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )
  692. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( UpToDate -> Consistent )
  693. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
  694. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: error receiving P_DATA, e: -512 l: 106496!
  695. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: My uuid_flags changed from 0x100 to 0x120 during handshake.
  696. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Terminating sender thread
  697. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Starting sender thread (peer-node-id 3)
  698. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( no -> quorum ) [peer-state]
  699. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( yes -> no ) [peer-state]
  700. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( UpToDate -> Consistent ) [peer-state]
  701. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Connection closed
  702. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( NetworkFailure -> Unconnected ) [disconnected]
  703. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Restarting receiver thread
  704. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Unconnected -> Connecting ) [connecting]
  705. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Consistent -> Outdated ) [receive-peer-dagtag]
  706. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( Established -> WFBitMapT ) [receive-peer-dagtag]
  707. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Reconciliation resync because 'storage-load-test-0' disappeared. (o=-208)
  708. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Aborting remote state change 1388789927
  709. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3626300921: 1->all empty
  710. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 30(1), total 30; compression: 100.0%
  711. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 30(1), total 30; compression: 100.0%
  712. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Outdated -> Inconsistent ) [receive-bitmap]
  713. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: resync-susp( no -> connection dependency ) [receive-bitmap]
  714. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: resync-susp( no -> connection dependency ) [receive-bitmap]
  715. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( WFBitMapT -> SyncTarget ) [receive-bitmap]
  716. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: Began resync as SyncTarget (will sync 132 KB [33 bits set]).
  717. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 3626300921: primary_nodes=0, weak_nodes=0
  718. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 3626300921 (16ms)
  719. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 30(1), total 30; compression: 100.0%
  720. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: unexpected repl_state (Established) in receive_bitmap
  721. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2248437662: 2->all empty
  722. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Aborting remote state change 2248437662
  723. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: Resync done (total 1 sec; paused 0 sec; 132 K/sec)
  724. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: updated UUIDs 752996FB4D06CB50:0000000000000000:ADB250ED33B11982:D8796B7E2A90EF4C
  725. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Inconsistent -> Consistent ) [resync-finished]
  726. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: resync-susp( connection dependency -> no ) [resync-finished]
  727. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: resync-susp( connection dependency -> no ) [resync-finished]
  728. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( SyncTarget -> Established ) [resync-finished]
  729. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 111167260: 2->all empty
  730. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 111167260 (primary_nodes=0)
  731. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 194935479: 2->all empty
  732. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( quorum -> no ) [peer-state]
  733. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Consistent -> UpToDate ) quorum( no -> yes ) [peer-state]
  734. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( Consistent -> UpToDate ) [peer-state]
  735. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( Consistent -> UpToDate ) [peer-state]
  736. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 194935479 (primary_nodes=0)
  737. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2038869718: 0->all empty
  738. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 2038869718 (primary_nodes=0)
  739. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2081815640: 0->all empty
  740. Mar 31 01:16:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 2081815640 (primary_nodes=0)
  741. Mar 31 01:16:37 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3768514093: 0->3 role( Secondary ) conn( Connected )
  742. Mar 31 01:16:37 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 3768514093 (primary_nodes=8)
  743. Mar 31 01:16:37 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 88211161: 0->all empty
  744. Mar 31 01:16:37 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( UpToDate -> Consistent ) [peer-state]
  745. Mar 31 01:16:37 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 88211161 (primary_nodes=0)
  746. Mar 31 01:16:37 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( Consistent -> UpToDate ) [peer-state]
  747. Mar 31 01:16:37 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 718664504: 0->all empty
  748. Mar 31 01:16:37 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 718664504 (primary_nodes=0)
  749. Mar 31 01:16:37 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( UpToDate -> Outdated ) [receive-peer-dagtag]
  750. Mar 31 01:16:37 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( Established -> WFBitMapT ) [receive-peer-dagtag]
  751. Mar 31 01:16:37 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Reconciliation resync because 'storage-load-test-0' disappeared. (o=-32)
  752. Mar 31 01:16:37 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( no -> quorum ) [peer-state]
  753. Mar 31 01:16:37 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( yes -> no ) [peer-state]
  754. Mar 31 01:16:37 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( UpToDate -> Outdated ) [peer-state]
  755. Mar 31 01:16:37 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Handshake to peer 3 successful: Agreed network protocol version 122
  756. Mar 31 01:16:37 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  757. Mar 31 01:16:37 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Peer authenticated using 20 bytes HMAC
  758. Mar 31 01:16:37 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 778947052: 1->3 role( Secondary ) conn( Connected )
  759. Mar 31 01:16:37 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: self 752996FB4D06CB50:28D31133FEBC993A:ADB250ED33B11982:D8796B7E2A90EF4C bits:0 flags:0
  760. Mar 31 01:16:37 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: peer's exposed UUID: 28983A7814F88E65
  761. Mar 31 01:16:37 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 778947052: primary_nodes=8, weak_nodes=0
  762. Mar 31 01:16:37 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 778947052 (28ms)
  763. Mar 31 01:16:37 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connecting -> Connected ) peer( Unknown -> Primary ) [connected]
  764. Mar 31 01:16:37 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( DUnknown -> Diskless ) repl( Off -> Established ) [connected]
  765. Mar 31 01:16:37 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: strategy = no-sync
  766. Mar 31 01:16:38 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2467925668: 2->3 role( Secondary ) conn( Connected )
  767. Mar 31 01:16:38 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 2467925668 (primary_nodes=8)
  768. Mar 31 01:16:38 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 433044289: 3->all empty
  769. Mar 31 01:16:38 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 433044289 (primary_nodes=8)
  770. Mar 31 01:16:38 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 722555851: 3->all empty
  771. Mar 31 01:16:38 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 722555851 (primary_nodes=8)
  772. Mar 31 01:16:38 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2386366568: 0->3 role( Secondary ) conn( Connected )
  773. Mar 31 01:16:38 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 2386366568 (primary_nodes=8)
  774. Mar 31 01:16:38 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( UpToDate -> Outdated ) [peer-state]
  775. Mar 31 01:16:38 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1209087830: 3->all empty
  776. Mar 31 01:16:38 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Aborting remote state change 1209087830
  777. Mar 31 01:16:38 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3472239397: 0->all empty
  778. Mar 31 01:16:38 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 3472239397 (primary_nodes=8)
  779. Mar 31 01:16:38 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2761915777: 3->all empty
  780. Mar 31 01:16:38 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 2761915777 (primary_nodes=8)
  781. Mar 31 01:16:38 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1779301481: 3->all empty
  782. Mar 31 01:16:38 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 1779301481 (primary_nodes=8)
  783. Mar 31 01:16:58 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: meta connection shut down by peer.
  784. Mar 31 01:16:58 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  785. Mar 31 01:16:58 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( Established -> Off )
  786. Mar 31 01:16:58 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1400865519: 2->all empty
  787. Mar 31 01:16:58 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Terminating sender thread
  788. Mar 31 01:16:58 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Starting sender thread (peer-node-id 2)
  789. Mar 31 01:16:58 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Connection closed
  790. Mar 31 01:16:58 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( NetworkFailure -> Unconnected ) [disconnected]
  791. Mar 31 01:16:58 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Restarting receiver thread
  792. Mar 31 01:16:58 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Unconnected -> Connecting ) [connecting]
  793. Mar 31 01:16:59 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: PingAck did not arrive in time.
  794. Mar 31 01:16:59 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )
  795. Mar 31 01:16:59 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
  796. Mar 31 01:16:59 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Terminating sender thread
  797. Mar 31 01:16:59 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Starting sender thread (peer-node-id 3)
  798. Mar 31 01:16:59 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: meta connection shut down by peer.
  799. Mar 31 01:16:59 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  800. Mar 31 01:16:59 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( WFBitMapT -> Off )
  801. Mar 31 01:16:59 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Terminating sender thread
  802. Mar 31 01:16:59 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Starting sender thread (peer-node-id 0)
  803. Mar 31 01:16:59 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Aborting remote state change 1400865519 commit not possible
  804. Mar 31 01:16:59 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2807157035: 1->all empty
  805. Mar 31 01:16:59 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 2807157035 (0ms)
  806. Mar 31 01:16:59 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Connection closed
  807. Mar 31 01:16:59 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( NetworkFailure -> Unconnected ) [disconnected]
  808. Mar 31 01:16:59 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Restarting receiver thread
  809. Mar 31 01:16:59 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Unconnected -> Connecting ) [connecting]
  810. Mar 31 01:16:59 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Connection closed
  811. Mar 31 01:16:59 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( NetworkFailure -> Unconnected ) [disconnected]
  812. Mar 31 01:16:59 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Restarting receiver thread
  813. Mar 31 01:16:59 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Unconnected -> Connecting ) [connecting]
  814. Mar 31 01:17:28 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Handshake to peer 2 successful: Agreed network protocol version 122
  815. Mar 31 01:17:28 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  816. Mar 31 01:17:28 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Peer authenticated using 20 bytes HMAC
  817. Mar 31 01:17:28 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 332174034: 1->2 role( Secondary ) conn( Connected )
  818. Mar 31 01:17:28 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: drbd_sync_handshake:
  819. Mar 31 01:17:28 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: self 752996FB4D06CB50:0000000000000000:ADB250ED33B11982:D8796B7E2A90EF4C bits:0 flags:120
  820. Mar 31 01:17:28 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: peer 752996FB4D06CB50:0000000000000000:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC bits:0 flags:1100
  821. Mar 31 01:17:28 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: uuid_compare()=no-sync by rule=reconnected
  822. Mar 31 01:17:28 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 332174034: primary_nodes=8, weak_nodes=0
  823. Mar 31 01:17:28 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 332174034 (48ms)
  824. Mar 31 01:17:28 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [connected]
  825. Mar 31 01:17:28 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( Off -> Established ) [connected]
  826. Mar 31 01:17:28 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: cleared bm UUID and bitmap 752996FB4D06CB50:0000000000000000:ADB250ED33B11982:D8796B7E2A90EF4C
  827. Mar 31 01:17:28 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Handshake to peer 0 successful: Agreed network protocol version 122
  828. Mar 31 01:17:28 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  829. Mar 31 01:17:28 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Peer authenticated using 20 bytes HMAC
  830. Mar 31 01:17:28 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3438731863: 0->1 role( Secondary ) conn( Connected )
  831. Mar 31 01:17:28 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: drbd_sync_handshake:
  832. Mar 31 01:17:28 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: self 752996FB4D06CB50:0000000000000000:ADB250ED33B11982:D8796B7E2A90EF4C bits:0 flags:120
  833. Mar 31 01:17:28 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: peer 752996FB4D06CB50:0000000000000000:D8796B7E2A90EF4C:ADB250ED33B11982 bits:1 flags:1120
  834. Mar 31 01:17:28 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: uuid_compare()=no-sync by rule=reconnected
  835. Mar 31 01:17:28 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 3438731863 (primary_nodes=8)
  836. Mar 31 01:17:28 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  837. Mar 31 01:17:28 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( Off -> Established ) [remote]
  838. Mar 31 01:17:28 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: cleared bm UUID and bitmap 752996FB4D06CB50:0000000000000000:ADB250ED33B11982:D8796B7E2A90EF4C
  839. Mar 31 01:17:28 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Handshake to peer 3 successful: Agreed network protocol version 122
  840. Mar 31 01:17:28 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  841. Mar 31 01:17:28 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Peer authenticated using 20 bytes HMAC
  842. Mar 31 01:17:28 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 369871165: 1->3 role( Secondary ) conn( Connected )
  843. Mar 31 01:17:28 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: self 752996FB4D06CB50:28D31133FEBC993A:ADB250ED33B11982:D8796B7E2A90EF4C bits:0 flags:0
  844. Mar 31 01:17:28 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: peer's exposed UUID: 28983A7814F88E65
  845. Mar 31 01:17:28 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 369871165: primary_nodes=8, weak_nodes=0
  846. Mar 31 01:17:28 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 369871165 (52ms)
  847. Mar 31 01:17:28 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connecting -> Connected ) peer( Unknown -> Primary ) [connected]
  848. Mar 31 01:17:28 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( DUnknown -> Diskless ) repl( Off -> Established ) [connected]
  849. Mar 31 01:17:49 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: PingAck did not arrive in time.
  850. Mar 31 01:17:49 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  851. Mar 31 01:17:49 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( Established -> Off )
  852. Mar 31 01:17:49 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 581687972: 1->all empty
  853. Mar 31 01:17:49 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Terminating sender thread
  854. Mar 31 01:17:49 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Starting sender thread (peer-node-id 0)
  855. Mar 31 01:17:49 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Connection closed
  856. Mar 31 01:17:49 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( NetworkFailure -> Unconnected ) [disconnected]
  857. Mar 31 01:17:49 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Restarting receiver thread
  858. Mar 31 01:17:49 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Unconnected -> Connecting ) [connecting]
  859. Mar 31 01:17:50 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 276ms
  860. Mar 31 01:17:50 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3634414869: 2->all empty
  861. Mar 31 01:17:50 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 3634414869 (primary_nodes=8)
  862. Mar 31 01:17:50 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1347114405: 1->all empty
  863. Mar 31 01:17:50 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 1347114405: primary_nodes=8, weak_nodes=0
  864. Mar 31 01:17:50 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 1347114405 (0ms)
  865. Mar 31 01:18:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1603385043: 0->2 role( Secondary ) conn( Connected )
  866. Mar 31 01:18:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 1603385043 (primary_nodes=8)
  867. Mar 31 01:18:29 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Handshake to peer 0 successful: Agreed network protocol version 122
  868. Mar 31 01:18:29 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  869. Mar 31 01:18:29 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Peer authenticated using 20 bytes HMAC
  870. Mar 31 01:18:29 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2458447738: 0->1 role( Secondary ) conn( Connected )
  871. Mar 31 01:18:29 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: drbd_sync_handshake:
  872. Mar 31 01:18:29 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: self 752996FB4D06CB50:0000000000000000:ADB250ED33B11982:D8796B7E2A90EF4C bits:0 flags:100
  873. Mar 31 01:18:29 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: peer 752996FB4D06CB50:0000000000000000:D8796B7E2A90EF4C:ADB250ED33B11982 bits:0 flags:1120
  874. Mar 31 01:18:29 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: uuid_compare()=no-sync by rule=reconnected
  875. Mar 31 01:18:29 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 2458447738 (primary_nodes=8)
  876. Mar 31 01:18:29 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  877. Mar 31 01:18:29 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( Off -> Established ) [remote]
  878. Mar 31 01:18:29 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: cleared bm UUID and bitmap 752996FB4D06CB50:0000000000000000:ADB250ED33B11982:D8796B7E2A90EF4C
  879. Mar 31 01:18:39 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2710131752: 0->all empty
  880. Mar 31 01:18:39 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 2710131752 (primary_nodes=8)
  881. Mar 31 01:18:39 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1561974972: 3->all empty
  882. Mar 31 01:18:39 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 1561974972 (primary_nodes=8)
  883. Mar 31 01:18:40 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: PingAck did not arrive in time.
  884. Mar 31 01:18:40 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  885. Mar 31 01:18:40 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( Established -> Off )
  886. Mar 31 01:18:40 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Terminating sender thread
  887. Mar 31 01:18:40 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Starting sender thread (peer-node-id 2)
  888. Mar 31 01:18:40 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 307708718: 1->all empty
  889. Mar 31 01:18:40 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 307708718: primary_nodes=8, weak_nodes=0
  890. Mar 31 01:18:40 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 307708718 (0ms)
  891. Mar 31 01:18:40 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Connection closed
  892. Mar 31 01:18:40 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( NetworkFailure -> Unconnected ) [disconnected]
  893. Mar 31 01:18:40 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Restarting receiver thread
  894. Mar 31 01:18:40 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Unconnected -> Connecting ) [connecting]
  895. Mar 31 01:19:04 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4181492047: 0->2 role( Secondary ) conn( Connected )
  896. Mar 31 01:19:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 4181492047 (primary_nodes=8)
  897. Mar 31 01:19:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2159242530: 2->3 role( Secondary ) conn( Connected )
  898. Mar 31 01:19:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Duplicate prepare for remote state change 2159242530
  899. Mar 31 01:19:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 2159242530 (primary_nodes=8)
  900. Mar 31 01:19:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Handshake to peer 2 successful: Agreed network protocol version 122
  901. Mar 31 01:19:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  902. Mar 31 01:19:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Peer authenticated using 20 bytes HMAC
  903. Mar 31 01:19:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3871126125: 1->2 role( Secondary ) conn( Connected )
  904. Mar 31 01:19:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: drbd_sync_handshake:
  905. Mar 31 01:19:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: self 752996FB4D06CB50:0000000000000000:ADB250ED33B11982:D8796B7E2A90EF4C bits:0 flags:100
  906. Mar 31 01:19:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: peer 752996FB4D06CB50:0000000000000000:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC bits:0 flags:1100
  907. Mar 31 01:19:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: uuid_compare()=no-sync by rule=reconnected
  908. Mar 31 01:19:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 3871126125: primary_nodes=8, weak_nodes=0
  909. Mar 31 01:19:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 3871126125 (32ms)
  910. Mar 31 01:19:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [connected]
  911. Mar 31 01:19:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( Off -> Established ) [connected]
  912. Mar 31 01:19:05 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: cleared bm UUID and bitmap 752996FB4D06CB50:0000000000000000:ADB250ED33B11982:D8796B7E2A90EF4C
  913. Mar 31 01:19:25 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2693247151: 2->all empty
  914. Mar 31 01:19:26 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Rejecting concurrent remote state change 732383613 because of state change 2693247151
  915. Mar 31 01:19:26 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Rejecting concurrent remote state change 3213501202 because of state change 2693247151
  916. Mar 31 01:19:26 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: PingAck did not arrive in time.
  917. Mar 31 01:19:26 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )
  918. Mar 31 01:19:26 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
  919. Mar 31 01:19:26 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Terminating sender thread
  920. Mar 31 01:19:26 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Starting sender thread (peer-node-id 3)
  921. Mar 31 01:19:26 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Aborting remote state change 2693247151
  922. Mar 31 01:19:26 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 595089147: 1->all empty
  923. Mar 31 01:19:26 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 595089147: primary_nodes=0, weak_nodes=0
  924. Mar 31 01:19:26 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 595089147 (0ms)
  925. Mar 31 01:19:26 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Connection closed
  926. Mar 31 01:19:26 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( NetworkFailure -> Unconnected ) [disconnected]
  927. Mar 31 01:19:26 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Restarting receiver thread
  928. Mar 31 01:19:26 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Unconnected -> Connecting ) [connecting]
  929. Mar 31 01:19:26 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 73580424: 2->all empty
  930. Mar 31 01:19:26 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 73580424 (primary_nodes=0)
  931. Mar 31 01:20:00 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1789966072: 2->3 role( Secondary ) conn( Connected )
  932. Mar 31 01:20:00 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 1789966072 (primary_nodes=8)
  933. Mar 31 01:20:07 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Handshake to peer 3 successful: Agreed network protocol version 122
  934. Mar 31 01:20:07 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  935. Mar 31 01:20:07 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Peer authenticated using 20 bytes HMAC
  936. Mar 31 01:20:07 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 178417790: 1->3 role( Secondary ) conn( Connected )
  937. Mar 31 01:20:07 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: self 752996FB4D06CB50:28D31133FEBC993A:ADB250ED33B11982:D8796B7E2A90EF4C bits:0 flags:0
  938. Mar 31 01:20:07 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: peer's exposed UUID: 28983A7814F88E65
  939. Mar 31 01:20:07 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 178417790: primary_nodes=8, weak_nodes=0
  940. Mar 31 01:20:07 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 178417790 (32ms)
  941. Mar 31 01:20:07 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connecting -> Connected ) peer( Unknown -> Primary ) [connected]
  942. Mar 31 01:20:07 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( DUnknown -> Diskless ) repl( Off -> Established ) [connected]
  943. Mar 31 01:20:18 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: meta connection shut down by peer.
  944. Mar 31 01:20:18 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  945. Mar 31 01:20:18 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( Established -> Off )
  946. Mar 31 01:20:18 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1409325880: 0->all empty
  947. Mar 31 01:20:18 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Terminating sender thread
  948. Mar 31 01:20:18 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Starting sender thread (peer-node-id 0)
  949. Mar 31 01:20:18 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Aborting remote state change 1409325880
  950. Mar 31 01:20:18 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3320924655: 1->all empty
  951. Mar 31 01:20:18 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Connection closed
  952. Mar 31 01:20:18 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( NetworkFailure -> Unconnected ) [disconnected]
  953. Mar 31 01:20:18 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Restarting receiver thread
  954. Mar 31 01:20:18 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Unconnected -> Connecting ) [connecting]
  955. Mar 31 01:20:18 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 104ms
  956. Mar 31 01:20:18 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3305118617: 2->all empty
  957. Mar 31 01:20:18 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 3305118617 (primary_nodes=8)
  958. Mar 31 01:20:19 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3950734662: 1->all empty
  959. Mar 31 01:20:19 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 3950734662: primary_nodes=8, weak_nodes=0
  960. Mar 31 01:20:19 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 3950734662 (0ms)
  961. Mar 31 01:20:56 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Handshake to peer 0 successful: Agreed network protocol version 122
  962. Mar 31 01:20:56 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  963. Mar 31 01:20:56 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Peer authenticated using 20 bytes HMAC
  964. Mar 31 01:20:56 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 26180170: 0->1 role( Secondary ) conn( Connected )
  965. Mar 31 01:20:56 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: drbd_sync_handshake:
  966. Mar 31 01:20:56 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: self 752996FB4D06CB50:0000000000000000:ADB250ED33B11982:D8796B7E2A90EF4C bits:0 flags:100
  967. Mar 31 01:20:56 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: peer 752996FB4D06CB50:0000000000000000:D8796B7E2A90EF4C:ADB250ED33B11982 bits:0 flags:1120
  968. Mar 31 01:20:56 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: uuid_compare()=no-sync by rule=reconnected
  969. Mar 31 01:20:56 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 26180170 (primary_nodes=8)
  970. Mar 31 01:20:56 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  971. Mar 31 01:20:56 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( Off -> Established ) [remote]
  972. Mar 31 01:20:56 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: cleared bm UUID and bitmap 752996FB4D06CB50:0000000000000000:ADB250ED33B11982:D8796B7E2A90EF4C
  973. Mar 31 01:21:00 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 113751511: 0->2 role( Secondary ) conn( Connected )
  974. Mar 31 01:21:00 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 113751511 (primary_nodes=8)
  975. Mar 31 01:21:07 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: PingAck did not arrive in time.
  976. Mar 31 01:21:07 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  977. Mar 31 01:21:07 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( Established -> Off )
  978. Mar 31 01:21:07 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Terminating sender thread
  979. Mar 31 01:21:07 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Starting sender thread (peer-node-id 2)
  980. Mar 31 01:21:07 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3394754177: 2->all empty
  981. Mar 31 01:21:07 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Connection closed
  982. Mar 31 01:21:07 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( NetworkFailure -> Unconnected ) [disconnected]
  983. Mar 31 01:21:07 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Restarting receiver thread
  984. Mar 31 01:21:07 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Unconnected -> Connecting ) [connecting]
  985. Mar 31 01:21:07 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: meta connection shut down by peer.
  986. Mar 31 01:21:07 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  987. Mar 31 01:21:07 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( Established -> Off )
  988. Mar 31 01:21:07 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Terminating sender thread
  989. Mar 31 01:21:07 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Starting sender thread (peer-node-id 0)
  990. Mar 31 01:21:07 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: sock was shut down by peer
  991. Mar 31 01:21:07 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connected -> BrokenPipe ) peer( Primary -> Unknown )
  992. Mar 31 01:21:07 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
  993. Mar 31 01:21:07 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Terminating sender thread
  994. Mar 31 01:21:07 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Starting sender thread (peer-node-id 3)
  995. Mar 31 01:21:07 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Connection closed
  996. Mar 31 01:21:07 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( NetworkFailure -> Unconnected ) [disconnected]
  997. Mar 31 01:21:07 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Restarting receiver thread
  998. Mar 31 01:21:07 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Unconnected -> Connecting ) [connecting]
  999. Mar 31 01:21:08 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Aborting remote state change 3394754177 commit not possible
  1000. Mar 31 01:21:08 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2793902138: 1->all empty
  1001. Mar 31 01:21:08 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 2793902138 (0ms)
  1002. Mar 31 01:21:08 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Connection closed
  1003. Mar 31 01:21:08 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( BrokenPipe -> Unconnected ) [disconnected]
  1004. Mar 31 01:21:08 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Restarting receiver thread
  1005. Mar 31 01:21:08 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Unconnected -> Connecting ) [connecting]
  1006. Mar 31 01:21:32 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Handshake to peer 2 successful: Agreed network protocol version 122
  1007. Mar 31 01:21:32 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  1008. Mar 31 01:21:32 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Peer authenticated using 20 bytes HMAC
  1009. Mar 31 01:21:32 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1146113168: 1->2 role( Secondary ) conn( Connected )
  1010. Mar 31 01:21:32 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: drbd_sync_handshake:
  1011. Mar 31 01:21:32 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: self 752996FB4D06CB50:0000000000000000:ADB250ED33B11982:D8796B7E2A90EF4C bits:0 flags:120
  1012. Mar 31 01:21:32 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: peer 752996FB4D06CB50:0000000000000000:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC bits:0 flags:1100
  1013. Mar 31 01:21:32 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: uuid_compare()=no-sync by rule=reconnected
  1014. Mar 31 01:21:32 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 1146113168: primary_nodes=8, weak_nodes=0
  1015. Mar 31 01:21:32 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 1146113168 (24ms)
  1016. Mar 31 01:21:32 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [connected]
  1017. Mar 31 01:21:32 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( Off -> Established ) [connected]
  1018. Mar 31 01:21:32 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: cleared bm UUID and bitmap 752996FB4D06CB50:0000000000000000:ADB250ED33B11982:D8796B7E2A90EF4C
  1019. Mar 31 01:21:34 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Handshake to peer 3 successful: Agreed network protocol version 122
  1020. Mar 31 01:21:34 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  1021. Mar 31 01:21:34 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Peer authenticated using 20 bytes HMAC
  1022. Mar 31 01:21:34 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 4164491876: 1->3 role( Secondary ) conn( Connected )
  1023. Mar 31 01:21:34 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: self 752996FB4D06CB50:28D31133FEBC993A:ADB250ED33B11982:D8796B7E2A90EF4C bits:0 flags:0
  1024. Mar 31 01:21:34 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: peer's exposed UUID: 28983A7814F88E65
  1025. Mar 31 01:21:34 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 4164491876: primary_nodes=8, weak_nodes=0
  1026. Mar 31 01:21:34 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 4164491876 (48ms)
  1027. Mar 31 01:21:34 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connecting -> Connected ) peer( Unknown -> Primary ) [connected]
  1028. Mar 31 01:21:34 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( DUnknown -> Diskless ) repl( Off -> Established ) [connected]
  1029. Mar 31 01:21:34 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Handshake to peer 0 successful: Agreed network protocol version 122
  1030. Mar 31 01:21:34 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  1031. Mar 31 01:21:34 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Peer authenticated using 20 bytes HMAC
  1032. Mar 31 01:21:34 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1675388973: 0->1 role( Secondary ) conn( Connected )
  1033. Mar 31 01:21:34 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: drbd_sync_handshake:
  1034. Mar 31 01:21:34 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: self 752996FB4D06CB50:0000000000000000:ADB250ED33B11982:D8796B7E2A90EF4C bits:0 flags:100
  1035. Mar 31 01:21:34 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: peer 752996FB4D06CB50:0000000000000000:D8796B7E2A90EF4C:ADB250ED33B11982 bits:0 flags:1120
  1036. Mar 31 01:21:34 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: uuid_compare()=no-sync by rule=reconnected
  1037. Mar 31 01:21:34 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 1675388973 (primary_nodes=8)
  1038. Mar 31 01:21:34 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  1039. Mar 31 01:21:34 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( Off -> Established ) [remote]
  1040. Mar 31 01:21:34 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: cleared bm UUID and bitmap 752996FB4D06CB50:0000000000000000:ADB250ED33B11982:D8796B7E2A90EF4C
  1041. Mar 31 01:21:55 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2583071922: 2->all empty
  1042. Mar 31 01:21:55 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: sock was shut down by peer
  1043. Mar 31 01:21:55 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connected -> BrokenPipe ) peer( Primary -> Unknown )
  1044. Mar 31 01:21:55 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
  1045. Mar 31 01:21:55 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Terminating sender thread
  1046. Mar 31 01:21:55 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Starting sender thread (peer-node-id 3)
  1047. Mar 31 01:21:55 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Aborting remote state change 2583071922
  1048. Mar 31 01:21:55 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 937972097: 1->all empty
  1049. Mar 31 01:21:55 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 937972097: primary_nodes=0, weak_nodes=0
  1050. Mar 31 01:21:55 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 937972097 (0ms)
  1051. Mar 31 01:21:55 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Connection closed
  1052. Mar 31 01:21:55 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( BrokenPipe -> Unconnected ) [disconnected]
  1053. Mar 31 01:21:55 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Restarting receiver thread
  1054. Mar 31 01:21:55 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Unconnected -> Connecting ) [connecting]
  1055. Mar 31 01:21:55 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3407140815: 2->all empty
  1056. Mar 31 01:21:55 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 3407140815 (primary_nodes=0)
  1057. Mar 31 01:22:33 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3835743607: 2->3 role( Secondary ) conn( Connected )
  1058. Mar 31 01:22:33 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Handshake to peer 3 successful: Agreed network protocol version 122
  1059. Mar 31 01:22:33 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  1060. Mar 31 01:22:33 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Peer authenticated using 20 bytes HMAC
  1061. Mar 31 01:22:33 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 3835743607 (primary_nodes=8)
  1062. Mar 31 01:22:33 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1248691193: 1->3 role( Secondary ) conn( Connected )
  1063. Mar 31 01:22:33 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: self 752996FB4D06CB50:28D31133FEBC993A:ADB250ED33B11982:D8796B7E2A90EF4C bits:0 flags:0
  1064. Mar 31 01:22:33 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: peer's exposed UUID: 28983A7814F88E65
  1065. Mar 31 01:22:33 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 1248691193: primary_nodes=8, weak_nodes=0
  1066. Mar 31 01:22:33 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 1248691193 (32ms)
  1067. Mar 31 01:22:33 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connecting -> Connected ) peer( Unknown -> Primary ) [connected]
  1068. Mar 31 01:22:33 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( DUnknown -> Diskless ) repl( Off -> Established ) [connected]
  1069. Mar 31 01:22:54 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: PingAck did not arrive in time.
  1070. Mar 31 01:22:54 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  1071. Mar 31 01:22:54 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( Established -> Off )
  1072. Mar 31 01:22:54 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 449856471: 1->all empty
  1073. Mar 31 01:22:54 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Terminating sender thread
  1074. Mar 31 01:22:54 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Starting sender thread (peer-node-id 0)
  1075. Mar 31 01:22:54 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Connection closed
  1076. Mar 31 01:22:54 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( NetworkFailure -> Unconnected ) [disconnected]
  1077. Mar 31 01:22:54 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Restarting receiver thread
  1078. Mar 31 01:22:54 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Unconnected -> Connecting ) [connecting]
  1079. Mar 31 01:22:55 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 228ms
  1080. Mar 31 01:22:55 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4094485826: 2->all empty
  1081. Mar 31 01:22:55 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 4094485826 (primary_nodes=8)
  1082. Mar 31 01:22:55 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1321179860: 1->all empty
  1083. Mar 31 01:22:55 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 1321179860: primary_nodes=8, weak_nodes=0
  1084. Mar 31 01:22:55 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 1321179860 (0ms)
  1085. Mar 31 01:23:21 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Handshake to peer 0 successful: Agreed network protocol version 122
  1086. Mar 31 01:23:21 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  1087. Mar 31 01:23:21 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Peer authenticated using 20 bytes HMAC
  1088. Mar 31 01:23:21 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4104775491: 0->1 role( Secondary ) conn( Connected )
  1089. Mar 31 01:23:21 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: drbd_sync_handshake:
  1090. Mar 31 01:23:21 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: self 752996FB4D06CB50:0000000000000000:ADB250ED33B11982:D8796B7E2A90EF4C bits:0 flags:100
  1091. Mar 31 01:23:21 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: peer 752996FB4D06CB50:0000000000000000:D8796B7E2A90EF4C:ADB250ED33B11982 bits:0 flags:1120
  1092. Mar 31 01:23:21 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: uuid_compare()=no-sync by rule=reconnected
  1093. Mar 31 01:23:21 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 4104775491 (primary_nodes=8)
  1094. Mar 31 01:23:21 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  1095. Mar 31 01:23:21 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( Off -> Established ) [remote]
  1096. Mar 31 01:23:21 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: cleared bm UUID and bitmap 752996FB4D06CB50:0000000000000000:ADB250ED33B11982:D8796B7E2A90EF4C
  1097. Mar 31 01:23:21 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 8244816: 0->2 role( Secondary ) conn( Connected )
  1098. Mar 31 01:23:21 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 8244816 (primary_nodes=8)
  1099. Mar 31 01:23:31 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: meta connection shut down by peer.
  1100. Mar 31 01:23:31 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  1101. Mar 31 01:23:31 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( Established -> Off )
  1102. Mar 31 01:23:31 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2945318694: 1->all empty
  1103. Mar 31 01:23:31 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Terminating sender thread
  1104. Mar 31 01:23:31 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Starting sender thread (peer-node-id 2)
  1105. Mar 31 01:23:31 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 2945318694: primary_nodes=8, weak_nodes=0
  1106. Mar 31 01:23:31 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 2945318694 (4ms)
  1107. Mar 31 01:23:31 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Connection closed
  1108. Mar 31 01:23:31 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( NetworkFailure -> Unconnected ) [disconnected]
  1109. Mar 31 01:23:31 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Restarting receiver thread
  1110. Mar 31 01:23:31 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Unconnected -> Connecting ) [connecting]
  1111. Mar 31 01:23:32 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2793566032: 3->all empty
  1112. Mar 31 01:23:32 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Aborting remote state change 2793566032
  1113. Mar 31 01:23:32 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 532498931: 0->all empty
  1114. Mar 31 01:23:32 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 532498931 (primary_nodes=8)
  1115. Mar 31 01:23:32 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2699842262: 3->all empty
  1116. Mar 31 01:23:32 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 2699842262 (primary_nodes=8)
  1117. Mar 31 01:24:17 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 664435784: 2->3 role( Secondary ) conn( Connected )
  1118. Mar 31 01:24:17 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 664435784 (primary_nodes=8)
  1119. Mar 31 01:24:18 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 426807218: 0->2 role( Secondary ) conn( Connected )
  1120. Mar 31 01:24:18 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 426807218 (primary_nodes=8)
  1121. Mar 31 01:24:20 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Handshake to peer 2 successful: Agreed network protocol version 122
  1122. Mar 31 01:24:20 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  1123. Mar 31 01:24:20 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Peer authenticated using 20 bytes HMAC
  1124. Mar 31 01:24:20 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2001482756: 1->2 role( Secondary ) conn( Connected )
  1125. Mar 31 01:24:20 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: drbd_sync_handshake:
  1126. Mar 31 01:24:20 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: self 752996FB4D06CB50:0000000000000000:ADB250ED33B11982:D8796B7E2A90EF4C bits:0 flags:100
  1127. Mar 31 01:24:20 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: peer 752996FB4D06CB50:0000000000000000:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC bits:0 flags:1100
  1128. Mar 31 01:24:20 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: uuid_compare()=no-sync by rule=reconnected
  1129. Mar 31 01:24:20 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 2001482756: primary_nodes=8, weak_nodes=0
  1130. Mar 31 01:24:20 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 2001482756 (28ms)
  1131. Mar 31 01:24:20 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [connected]
  1132. Mar 31 01:24:20 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( Off -> Established ) [connected]
  1133. Mar 31 01:24:20 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: cleared bm UUID and bitmap 752996FB4D06CB50:0000000000000000:ADB250ED33B11982:D8796B7E2A90EF4C
  1134. Mar 31 01:24:30 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: meta connection shut down by peer.
  1135. Mar 31 01:24:30 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  1136. Mar 31 01:24:30 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( Established -> Off )
  1137. Mar 31 01:24:30 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 4088068711: 1->all empty
  1138. Mar 31 01:24:30 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Terminating sender thread
  1139. Mar 31 01:24:30 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Starting sender thread (peer-node-id 0)
  1140. Mar 31 01:24:30 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Connection closed
  1141. Mar 31 01:24:30 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( NetworkFailure -> Unconnected ) [disconnected]
  1142. Mar 31 01:24:30 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Restarting receiver thread
  1143. Mar 31 01:24:30 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Unconnected -> Connecting ) [connecting]
  1144. Mar 31 01:24:30 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: sock was shut down by peer
  1145. Mar 31 01:24:30 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connected -> BrokenPipe ) peer( Primary -> Unknown )
  1146. Mar 31 01:24:30 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
  1147. Mar 31 01:24:30 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 364ms
  1148. Mar 31 01:24:30 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Terminating sender thread
  1149. Mar 31 01:24:30 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Starting sender thread (peer-node-id 3)
  1150. Mar 31 01:24:31 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Connection closed
  1151. Mar 31 01:24:31 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( BrokenPipe -> Unconnected ) [disconnected]
  1152. Mar 31 01:24:31 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Restarting receiver thread
  1153. Mar 31 01:24:31 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Unconnected -> Connecting ) [connecting]
  1154. Mar 31 01:24:31 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1087692431: 1->all empty
  1155. Mar 31 01:24:31 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: meta connection shut down by peer.
  1156. Mar 31 01:24:31 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  1157. Mar 31 01:24:31 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( Established -> Off )
  1158. Mar 31 01:24:31 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 512ms
  1159. Mar 31 01:24:31 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Terminating sender thread
  1160. Mar 31 01:24:31 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Starting sender thread (peer-node-id 2)
  1161. Mar 31 01:24:31 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Connection closed
  1162. Mar 31 01:24:31 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( NetworkFailure -> Unconnected ) [disconnected]
  1163. Mar 31 01:24:31 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Restarting receiver thread
  1164. Mar 31 01:24:31 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Unconnected -> Connecting ) [connecting]
  1165. Mar 31 01:24:31 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2308417964: 1->all empty
  1166. Mar 31 01:24:31 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 2308417964 (0ms)
  1167. Mar 31 01:25:22 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Handshake to peer 2 successful: Agreed network protocol version 122
  1168. Mar 31 01:25:22 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  1169. Mar 31 01:25:22 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Peer authenticated using 20 bytes HMAC
  1170. Mar 31 01:25:22 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 4181019255: 1->2 role( Secondary ) conn( Connected )
  1171. Mar 31 01:25:22 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: drbd_sync_handshake:
  1172. Mar 31 01:25:22 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: self 752996FB4D06CB50:0000000000000000:ADB250ED33B11982:D8796B7E2A90EF4C bits:0 flags:120
  1173. Mar 31 01:25:22 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: peer 752996FB4D06CB50:0000000000000000:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC bits:0 flags:1100
  1174. Mar 31 01:25:22 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: uuid_compare()=no-sync by rule=reconnected
  1175. Mar 31 01:25:22 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 4181019255: primary_nodes=8, weak_nodes=0
  1176. Mar 31 01:25:22 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 4181019255 (36ms)
  1177. Mar 31 01:25:22 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [connected]
  1178. Mar 31 01:25:22 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( Off -> Established ) [connected]
  1179. Mar 31 01:25:22 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: cleared bm UUID and bitmap 752996FB4D06CB50:0000000000000000:ADB250ED33B11982:D8796B7E2A90EF4C
  1180. Mar 31 01:25:22 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Handshake to peer 3 successful: Agreed network protocol version 122
  1181. Mar 31 01:25:22 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  1182. Mar 31 01:25:22 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Peer authenticated using 20 bytes HMAC
  1183. Mar 31 01:25:22 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 283636919: 1->3 role( Secondary ) conn( Connected )
  1184. Mar 31 01:25:22 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: self 752996FB4D06CB50:28D31133FEBC993A:ADB250ED33B11982:D8796B7E2A90EF4C bits:0 flags:0
  1185. Mar 31 01:25:22 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: peer's exposed UUID: 28983A7814F88E65
  1186. Mar 31 01:25:22 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 283636919: primary_nodes=8, weak_nodes=0
  1187. Mar 31 01:25:22 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 283636919 (28ms)
  1188. Mar 31 01:25:22 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connecting -> Connected ) peer( Unknown -> Primary ) [connected]
  1189. Mar 31 01:25:22 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( DUnknown -> Diskless ) repl( Off -> Established ) [connected]
  1190. Mar 31 01:25:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Handshake to peer 0 successful: Agreed network protocol version 122
  1191. Mar 31 01:25:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  1192. Mar 31 01:25:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Peer authenticated using 20 bytes HMAC
  1193. Mar 31 01:25:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1804534236: 0->1 role( Secondary ) conn( Connected )
  1194. Mar 31 01:25:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: drbd_sync_handshake:
  1195. Mar 31 01:25:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: self 752996FB4D06CB50:0000000000000000:ADB250ED33B11982:D8796B7E2A90EF4C bits:0 flags:100
  1196. Mar 31 01:25:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: peer 752996FB4D06CB50:0000000000000000:D8796B7E2A90EF4C:ADB250ED33B11982 bits:0 flags:1120
  1197. Mar 31 01:25:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: uuid_compare()=no-sync by rule=reconnected
  1198. Mar 31 01:25:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 1804534236 (primary_nodes=8)
  1199. Mar 31 01:25:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  1200. Mar 31 01:25:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( Off -> Established ) [remote]
  1201. Mar 31 01:25:27 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: cleared bm UUID and bitmap 752996FB4D06CB50:0000000000000000:ADB250ED33B11982:D8796B7E2A90EF4C
  1202. Mar 31 01:45:33 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3130963472: 3->1 conn( Disconnecting ) pdsk( Outdated )
  1203. Mar 31 01:45:33 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 3130963472 (primary_nodes=8)
  1204. Mar 31 01:45:33 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connected -> TearDown ) peer( Primary -> Unknown ) [remote]
  1205. Mar 31 01:45:33 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( Diskless -> DUnknown ) repl( Established -> Off ) [remote]
  1206. Mar 31 01:45:33 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Terminating sender thread
  1207. Mar 31 01:45:33 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Starting sender thread (peer-node-id 3)
  1208. Mar 31 01:45:33 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3905303763: 3->2 conn( Disconnecting ) pdsk( Outdated )
  1209. Mar 31 01:45:33 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 3905303763 (primary_nodes=8)
  1210. Mar 31 01:45:33 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Connection closed
  1211. Mar 31 01:45:33 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( TearDown -> Unconnected ) [disconnected]
  1212. Mar 31 01:45:33 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Restarting receiver thread
  1213. Mar 31 01:45:33 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Unconnected -> Connecting ) [connecting]
  1214. Mar 31 01:45:40 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Handshake to peer 3 successful: Agreed network protocol version 122
  1215. Mar 31 01:45:40 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  1216. Mar 31 01:45:40 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Peer authenticated using 20 bytes HMAC
  1217. Mar 31 01:45:40 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3269594929: 1->3 role( Secondary ) conn( Connected )
  1218. Mar 31 01:45:40 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: self 752996FB4D06CB50:28D31133FEBC993A:ADB250ED33B11982:D8796B7E2A90EF4C bits:0 flags:0
  1219. Mar 31 01:45:40 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: peer's exposed UUID: 28983A7814F88E65
  1220. Mar 31 01:45:40 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 3269594929: primary_nodes=8, weak_nodes=0
  1221. Mar 31 01:45:40 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 3269594929 (40ms)
  1222. Mar 31 01:45:40 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connecting -> Connected ) peer( Unknown -> Primary ) [connected]
  1223. Mar 31 01:45:40 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( DUnknown -> Diskless ) repl( Off -> Established ) [connected]
  1224. Mar 31 01:45:40 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3578569625: 2->3 role( Secondary ) conn( Connected )
  1225. Mar 31 01:45:40 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 3578569625 (primary_nodes=8)
  1226. Mar 31 01:45:40 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 775278204: 0->3 role( Secondary ) conn( Connected )
  1227. Mar 31 01:45:40 storage-load-test-1 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 775278204 (primary_nodes=8)
  1228.  
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!