API tools faq
paste
Advertisement
AleksZi

dmesg_storage-load-test-3.log

AleksZi
Mar 30th, 2025
15
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 212.43 KB | None | 0 0
raw download clone embed print report
  1. Mar 31 01:05:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Starting worker thread (node-id 2)
  2. Mar 31 01:05:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Starting sender thread (peer-node-id 1)
  3. Mar 31 01:05:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Starting sender thread (peer-node-id 0)
  4. Mar 31 01:05:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: meta-data IO uses: blk-bio
  5. Mar 31 01:05:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( no -> quorum ) [attach]
  6. Mar 31 01:05:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Diskless -> Attaching ) [attach]
  7. Mar 31 01:05:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: Maximum number of peer devices = 7
  8. Mar 31 01:05:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Method to ensure write ordering: flush
  9. Mar 31 01:05:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: drbd_bm_resize called with capacity == 2104816
  10. Mar 31 01:05:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: resync bitmap: bits=263102 words=28777 pages=57
  11. Mar 31 01:05:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: size = 1028 MB (1052408 KB)
  12. Mar 31 01:05:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Attaching -> Inconsistent ) [attach]
  13. Mar 31 01:05:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: attached to current UUID: 0000000000000004
  14. Mar 31 01:05:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( StandAlone -> Unconnected ) [connect]
  15. Mar 31 01:05:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( StandAlone -> Unconnected ) [connect]
  16. Mar 31 01:05:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Starting receiver thread (peer-node-id 1)
  17. Mar 31 01:05:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Unconnected -> Connecting ) [connecting]
  18. Mar 31 01:05:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Starting receiver thread (peer-node-id 0)
  19. Mar 31 01:05:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Unconnected -> Connecting ) [connecting]
  20. Mar 31 01:05:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Handshake to peer 1 successful: Agreed network protocol version 122
  21. Mar 31 01:05:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  22. Mar 31 01:05:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Peer authenticated using 20 bytes HMAC
  23. Mar 31 01:05:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 924098904: 1->2 role( Secondary ) conn( Connected )
  24. Mar 31 01:05:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: drbd_sync_handshake:
  25. Mar 31 01:05:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: self 0000000000000004:0000000000000000:0000000000000000:0000000000000000 bits:0 flags:24
  26. Mar 31 01:05:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: peer 0000000000000004:0000000000000000:0000000000000000:0000000000000000 bits:0 flags:24
  27. Mar 31 01:05:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: uuid_compare()=no-sync by rule=just-created-both
  28. Mar 31 01:05:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 924098904 (primary_nodes=0)
  29. Mar 31 01:05:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  30. Mar 31 01:05:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( DUnknown -> Inconsistent ) repl( Off -> Established ) [remote]
  31. Mar 31 01:05:03 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Handshake to peer 0 successful: Agreed network protocol version 122
  32. Mar 31 01:05:03 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  33. Mar 31 01:05:03 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Peer authenticated using 20 bytes HMAC
  34. Mar 31 01:05:03 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3072324464: 0->2 role( Secondary ) conn( Connected )
  35. Mar 31 01:05:03 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: drbd_sync_handshake:
  36. Mar 31 01:05:03 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: self 0000000000000004:0000000000000000:28D31133FEBC993A:0000000000000000 bits:0 flags:24
  37. Mar 31 01:05:03 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: peer 52F5EAB2DCF34652:28D31133FEBC993B:0000000000000000:0000000000000000 bits:263102 flags:1020
  38. Mar 31 01:05:03 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: uuid_compare()=target-set-bitmap by rule=just-created-self
  39. Mar 31 01:05:03 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: Setting and writing the whole bitmap, fresh node
  40. Mar 31 01:05:03 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: bitmap WRITE of 57 pages took 8 ms
  41. Mar 31 01:05:03 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 3072324464 (primary_nodes=0)
  42. Mar 31 01:05:03 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  43. Mar 31 01:05:03 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( DUnknown -> UpToDate ) repl( Off -> WFBitMapT ) [remote]
  44. Mar 31 01:05:03 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: Setting exposed data uuid: 52F5EAB2DCF34652
  45. Mar 31 01:05:03 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: persisting effective size = 1028 MB (1052408 KB)
  46. Mar 31 01:05:03 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 21(1), total 21; compression: 100.0%
  47. Mar 31 01:05:03 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 21(1), total 21; compression: 100.0%
  48. Mar 31 01:05:03 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: setting UUIDs to 28D31133FEBC993A:0000000000000000:28D31133FEBC993A:0000000000000000
  49. Mar 31 01:05:03 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( WFBitMapT -> SyncTarget ) [receive-bitmap]
  50. Mar 31 01:05:03 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: resync-susp( no -> connection dependency ) [receive-bitmap]
  51. Mar 31 01:05:03 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: Setting exposed data uuid: 52F5EAB2DCF34652
  52. Mar 31 01:05:03 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: Began resync as SyncTarget (will sync 1052408 KB [263102 bits set]).
  53. Mar 31 01:05:03 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2377433559: 0->1 role( Secondary ) conn( Connected )
  54. Mar 31 01:05:03 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 2377433559 (primary_nodes=0)
  55. Mar 31 01:05:14 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( quorum -> no ) [peer-state]
  56. Mar 31 01:05:14 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( no -> yes ) [peer-state]
  57. Mar 31 01:05:14 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( Inconsistent -> UpToDate ) [peer-state]
  58. Mar 31 01:05:14 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: Multiple UpToDate peers have different current UUIDs
  59. Mar 31 01:05:14 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: storage-load-test-2: 52F5EAB2DCF34652 storage-load-test-1: 0000000000000004
  60. Mar 31 01:05:14 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: Multiple UpToDate peers have different current UUIDs
  61. Mar 31 01:05:14 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: storage-load-test-2: 52F5EAB2DCF34652 storage-load-test-1: 0000000000000004
  62. Mar 31 01:05:14 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: updated UUIDs 28D31133FEBC993A:0000000000000000:28D31133FEBC993A:0000000000000000
  63. Mar 31 01:05:14 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: Resync done (total 11 sec; paused 0 sec; 95672 K/sec)
  64. Mar 31 01:05:14 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: updated UUIDs 52F5EAB2DCF34652:0000000000000000:28D31133FEBC993A:0000000000000000
  65. Mar 31 01:05:14 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Inconsistent -> UpToDate ) [resync-finished]
  66. Mar 31 01:05:14 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( SyncTarget -> Established ) [resync-finished]
  67. Mar 31 01:05:14 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: resync-susp( connection dependency -> no ) [resync-finished]
  68. Mar 31 01:05:15 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3028870165: 0->all role( Primary )
  69. Mar 31 01:05:15 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 3028870165 (primary_nodes=1)
  70. Mar 31 01:05:15 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: peer( Secondary -> Primary ) [remote]
  71. Mar 31 01:07:54 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2893496555: 0->all role( Secondary )
  72. Mar 31 01:07:54 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 2893496555 (primary_nodes=0)
  73. Mar 31 01:07:54 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: peer( Primary -> Secondary ) [remote]
  74. Mar 31 01:07:58 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Starting sender thread (peer-node-id 3)
  75. Mar 31 01:07:58 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( StandAlone -> Unconnected ) [connect]
  76. Mar 31 01:07:58 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Starting receiver thread (peer-node-id 3)
  77. Mar 31 01:07:58 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Unconnected -> Connecting ) [connecting]
  78. Mar 31 01:07:59 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Handshake to peer 3 successful: Agreed network protocol version 122
  79. Mar 31 01:07:59 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  80. Mar 31 01:07:59 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Peer authenticated using 20 bytes HMAC
  81. Mar 31 01:07:59 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 4112650981: 2->3 role( Secondary ) conn( Connected )
  82. Mar 31 01:07:59 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: self 52F5EAB2DCF34652:28D31133FEBC993A:28D31133FEBC993A:0000000000000000 bits:0 flags:0
  83. Mar 31 01:07:59 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: peer's exposed UUID: 0000000000000000
  84. Mar 31 01:07:59 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 4112650981: primary_nodes=0, weak_nodes=0
  85. Mar 31 01:07:59 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 4112650981 (36ms)
  86. Mar 31 01:07:59 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [connected]
  87. Mar 31 01:07:59 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( DUnknown -> Diskless ) repl( Off -> Established ) [connected]
  88. Mar 31 01:07:59 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1056996341: 1->3 role( Secondary ) conn( Connected )
  89. Mar 31 01:07:59 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 1056996341 (primary_nodes=0)
  90. Mar 31 01:07:59 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1852736253: 0->3 role( Secondary ) conn( Connected )
  91. Mar 31 01:07:59 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 1852736253 (primary_nodes=0)
  92. Mar 31 01:08:04 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 229711114: 3->all role( Primary )
  93. Mar 31 01:08:04 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 229711114 (primary_nodes=8)
  94. Mar 31 01:08:04 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: peer( Secondary -> Primary ) [remote]
  95. Mar 31 01:08:04 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: received new current UUID: 3B36CC1B2C2CCBAD weak_nodes=FFFFFFFFFFFFFFF8
  96. Mar 31 01:08:40 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Duplicate prepare for remote state change 3427757018
  97. Mar 31 01:08:40 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: meta connection shut down by peer.
  98. Mar 31 01:08:40 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: sock was shut down by peer
  99. Mar 31 01:08:40 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  100. Mar 31 01:08:40 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
  101. Mar 31 01:08:40 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3427757018: 0->all empty
  102. Mar 31 01:08:40 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Terminating sender thread
  103. Mar 31 01:08:40 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Starting sender thread (peer-node-id 0)
  104. Mar 31 01:08:40 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Connection closed
  105. Mar 31 01:08:40 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( NetworkFailure -> Unconnected ) [disconnected]
  106. Mar 31 01:08:40 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Restarting receiver thread
  107. Mar 31 01:08:40 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Unconnected -> Connecting ) [connecting]
  108. Mar 31 01:08:41 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: sock was shut down by peer
  109. Mar 31 01:08:41 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( no -> quorum )
  110. Mar 31 01:08:41 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connected -> BrokenPipe ) peer( Secondary -> Unknown )
  111. Mar 31 01:08:41 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( yes -> no )
  112. Mar 31 01:08:41 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
  113. Mar 31 01:08:41 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Terminating sender thread
  114. Mar 31 01:08:41 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Starting sender thread (peer-node-id 1)
  115. Mar 31 01:08:41 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Connection closed
  116. Mar 31 01:08:41 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( BrokenPipe -> Unconnected ) [disconnected]
  117. Mar 31 01:08:41 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Restarting receiver thread
  118. Mar 31 01:08:41 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Unconnected -> Connecting ) [connecting]
  119. Mar 31 01:08:41 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Aborting remote state change 3427757018
  120. Mar 31 01:08:41 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 780112742: 1->all empty
  121. Mar 31 01:08:41 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: sock was shut down by peer
  122. Mar 31 01:08:41 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connected -> BrokenPipe ) peer( Primary -> Unknown )
  123. Mar 31 01:08:41 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( UpToDate -> Consistent )
  124. Mar 31 01:08:41 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
  125. Mar 31 01:08:41 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Terminating sender thread
  126. Mar 31 01:08:41 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Starting sender thread (peer-node-id 3)
  127. Mar 31 01:08:41 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Aborting remote state change 780112742 commit not possible
  128. Mar 31 01:08:41 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3950294474: 2->all empty
  129. Mar 31 01:08:41 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 3950294474 (0ms)
  130. Mar 31 01:08:41 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Consistent -> UpToDate ) [lost-peer]
  131. Mar 31 01:08:41 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Connection closed
  132. Mar 31 01:08:41 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( BrokenPipe -> Unconnected ) [disconnected]
  133. Mar 31 01:08:41 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Restarting receiver thread
  134. Mar 31 01:08:41 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Unconnected -> Connecting ) [connecting]
  135. Mar 31 01:09:24 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Handshake to peer 0 successful: Agreed network protocol version 122
  136. Mar 31 01:09:24 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  137. Mar 31 01:09:24 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Peer authenticated using 20 bytes HMAC
  138. Mar 31 01:09:24 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 841577807: 0->2 role( Secondary ) conn( Connected )
  139. Mar 31 01:09:24 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: drbd_sync_handshake:
  140. Mar 31 01:09:24 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: self 3B36CC1B2C2CCBAC:0000000000000000:52F5EAB2DCF34652:28D31133FEBC993A bits:0 flags:120
  141. Mar 31 01:09:24 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: peer ADB250ED33B11982:3B36CC1B2C2CCBAC:52F5EAB2DCF34652:0000000000000000 bits:343 flags:1100
  142. Mar 31 01:09:24 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: uuid_compare()=target-use-bitmap by rule=bitmap-peer
  143. Mar 31 01:09:24 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 841577807 (primary_nodes=8)
  144. Mar 31 01:09:24 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  145. Mar 31 01:09:24 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( UpToDate -> Outdated ) [remote]
  146. Mar 31 01:09:24 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( DUnknown -> UpToDate ) repl( Off -> WFBitMapT ) [remote]
  147. Mar 31 01:09:24 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: Setting exposed data uuid: ADB250ED33B11982
  148. Mar 31 01:09:24 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 39(1), total 39; compression: 99.9%
  149. Mar 31 01:09:24 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 39(1), total 39; compression: 99.9%
  150. Mar 31 01:09:24 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Outdated -> Inconsistent ) [receive-bitmap]
  151. Mar 31 01:09:24 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: resync-susp( no -> connection dependency ) [receive-bitmap]
  152. Mar 31 01:09:24 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( WFBitMapT -> SyncTarget ) [receive-bitmap]
  153. Mar 31 01:09:24 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: resync-susp( no -> connection dependency ) [receive-bitmap]
  154. Mar 31 01:09:24 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: Began resync as SyncTarget (will sync 1372 KB [343 bits set]).
  155. Mar 31 01:09:24 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: Resync done (total 1 sec; paused 0 sec; 1372 K/sec)
  156. Mar 31 01:09:24 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: Peer was unstable during resync
  157. Mar 31 01:09:24 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: resync-susp( connection dependency -> no ) [resync-finished]
  158. Mar 31 01:09:24 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( SyncTarget -> Established ) [resync-finished]
  159. Mar 31 01:09:24 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: resync-susp( connection dependency -> no ) [resync-finished]
  160. Mar 31 01:09:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Handshake to peer 1 successful: Agreed network protocol version 122
  161. Mar 31 01:09:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  162. Mar 31 01:09:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Peer authenticated using 20 bytes HMAC
  163. Mar 31 01:09:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3079894093: 1->2 role( Secondary ) conn( Connected )
  164. Mar 31 01:09:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: drbd_sync_handshake:
  165. Mar 31 01:09:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: self 3B36CC1B2C2CCBAC:3B36CC1B2C2CCBAC:52F5EAB2DCF34652:28D31133FEBC993A bits:0 flags:124
  166. Mar 31 01:09:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: peer ADB250ED33B11982:3B36CC1B2C2CCBAC:52F5EAB2DCF34652:28D31133FEBC993A bits:0 flags:1100
  167. Mar 31 01:09:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: uuid_compare()=target-use-bitmap by rule=bitmap-peer
  168. Mar 31 01:09:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 3079894093 (primary_nodes=8)
  169. Mar 31 01:09:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( quorum -> no ) [remote]
  170. Mar 31 01:09:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  171. Mar 31 01:09:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( no -> yes ) [remote]
  172. Mar 31 01:09:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( DUnknown -> UpToDate ) repl( Off -> WFBitMapT ) [remote]
  173. Mar 31 01:09:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 21(1), total 21; compression: 100.0%
  174. Mar 31 01:09:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 21(1), total 21; compression: 100.0%
  175. Mar 31 01:09:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: resync-susp( no -> connection dependency ) [receive-bitmap]
  176. Mar 31 01:09:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: resync-susp( no -> connection dependency ) [receive-bitmap]
  177. Mar 31 01:09:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( WFBitMapT -> SyncTarget ) [receive-bitmap]
  178. Mar 31 01:09:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: Began resync as SyncTarget (will sync 0 KB [0 bits set]).
  179. Mar 31 01:09:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: Resync done (total 1 sec; paused 0 sec; 0 K/sec)
  180. Mar 31 01:09:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: Peer was unstable during resync
  181. Mar 31 01:09:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: resync-susp( connection dependency -> no ) [resync-finished]
  182. Mar 31 01:09:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: resync-susp( connection dependency -> no ) [resync-finished]
  183. Mar 31 01:09:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( SyncTarget -> Established ) [resync-finished]
  184. Mar 31 01:09:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Handshake to peer 3 successful: Agreed network protocol version 122
  185. Mar 31 01:09:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  186. Mar 31 01:09:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Peer authenticated using 20 bytes HMAC
  187. Mar 31 01:09:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 97363440: 2->3 role( Secondary ) conn( Connected )
  188. Mar 31 01:09:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: self 3B36CC1B2C2CCBAC:28D31133FEBC993A:52F5EAB2DCF34652:28D31133FEBC993A bits:0 flags:0
  189. Mar 31 01:09:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: peer's exposed UUID: ADB250ED33B11983
  190. Mar 31 01:09:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 97363440: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF0
  191. Mar 31 01:09:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 97363440 (60ms)
  192. Mar 31 01:09:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connecting -> Connected ) peer( Unknown -> Primary ) [connected]
  193. Mar 31 01:09:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( DUnknown -> Diskless ) repl( Off -> Established ) [connected]
  194. Mar 31 01:09:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: strategy = target-use-bitmap
  195. Mar 31 01:09:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: strategy = target-use-bitmap
  196. Mar 31 01:09:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: drbd_sync_handshake:
  197. Mar 31 01:09:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: self 3B36CC1B2C2CCBAC:0000000000000000:52F5EAB2DCF34652:28D31133FEBC993A bits:0 flags:104
  198. Mar 31 01:09:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: peer ADB250ED33B11982:3B36CC1B2C2CCBAC:52F5EAB2DCF34652:0000000000000000 bits:343 flags:1100
  199. Mar 31 01:09:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: uuid_compare()=target-use-bitmap by rule=bitmap-peer
  200. Mar 31 01:09:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: Becoming WFBitMapT because primary is diskless
  201. Mar 31 01:09:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( Established -> WFBitMapT ) [diskless-primary]
  202. Mar 31 01:09:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 30(1), total 30; compression: 100.0%
  203. Mar 31 01:09:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 30(1), total 30; compression: 100.0%
  204. Mar 31 01:09:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: resync-susp( no -> connection dependency ) [receive-bitmap]
  205. Mar 31 01:09:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( WFBitMapT -> SyncTarget ) [receive-bitmap]
  206. Mar 31 01:09:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: resync-susp( no -> connection dependency ) [receive-bitmap]
  207. Mar 31 01:09:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: Began resync as SyncTarget (will sync 152 KB [38 bits set]).
  208. Mar 31 01:09:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: Resync done (total 1 sec; paused 0 sec; 152 K/sec)
  209. Mar 31 01:09:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: updated UUIDs ADB250ED33B11982:0000000000000000:52F5EAB2DCF34652:28D31133FEBC993A
  210. Mar 31 01:09:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Inconsistent -> UpToDate ) [resync-finished]
  211. Mar 31 01:09:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: resync-susp( connection dependency -> no ) [resync-finished]
  212. Mar 31 01:09:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( SyncTarget -> Established ) [resync-finished]
  213. Mar 31 01:09:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: resync-susp( connection dependency -> no ) [resync-finished]
  214. Mar 31 01:09:46 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1665238630: 1->all empty
  215. Mar 31 01:09:46 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: sock was shut down by peer
  216. Mar 31 01:09:46 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connected -> BrokenPipe ) peer( Secondary -> Unknown )
  217. Mar 31 01:09:46 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
  218. Mar 31 01:09:46 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Terminating sender thread
  219. Mar 31 01:09:46 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Starting sender thread (peer-node-id 1)
  220. Mar 31 01:09:46 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Rejecting concurrent remote state change 1088512057 because of state change 1665238630
  221. Mar 31 01:09:46 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Aborting remote state change 1665238630 commit not possible
  222. Mar 31 01:09:46 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3183198224: 2->all empty
  223. Mar 31 01:09:46 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 3183198224: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF2
  224. Mar 31 01:09:46 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 3183198224 (0ms)
  225. Mar 31 01:09:46 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2895404419: 0->all empty
  226. Mar 31 01:09:46 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 2895404419 (primary_nodes=8)
  227. Mar 31 01:09:46 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Connection closed
  228. Mar 31 01:09:46 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( BrokenPipe -> Unconnected ) [disconnected]
  229. Mar 31 01:09:46 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Restarting receiver thread
  230. Mar 31 01:09:46 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Unconnected -> Connecting ) [connecting]
  231. Mar 31 01:09:46 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: received new current UUID: DD95BBD64DA6BD63 weak_nodes=FFFFFFFFFFFFFFFA
  232. Mar 31 01:09:46 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2955162861: 3->all empty
  233. Mar 31 01:09:46 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 2955162861 (primary_nodes=8)
  234. Mar 31 01:10:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Handshake to peer 1 successful: Agreed network protocol version 122
  235. Mar 31 01:10:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  236. Mar 31 01:10:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Peer authenticated using 20 bytes HMAC
  237. Mar 31 01:10:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4229506866: 1->2 role( Secondary ) conn( Connected )
  238. Mar 31 01:10:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: drbd_sync_handshake:
  239. Mar 31 01:10:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: self DD95BBD64DA6BD62:ADB250ED33B11982:52F5EAB2DCF34652:28D31133FEBC993A bits:230 flags:100
  240. Mar 31 01:10:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: peer ADB250ED33B11982:0000000000000000:3B36CC1B2C2CCBAC:52F5EAB2DCF34652 bits:75 flags:1120
  241. Mar 31 01:10:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: uuid_compare()=source-use-bitmap by rule=bitmap-self
  242. Mar 31 01:10:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 4229506866 (primary_nodes=8)
  243. Mar 31 01:10:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  244. Mar 31 01:10:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( DUnknown -> Consistent ) repl( Off -> WFBitMapS ) [remote]
  245. Mar 31 01:10:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( Consistent -> Outdated ) [peer-state]
  246. Mar 31 01:10:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 37(1), total 37; compression: 99.9%
  247. Mar 31 01:10:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 37(1), total 37; compression: 99.9%
  248. Mar 31 01:10:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( Outdated -> Inconsistent ) repl( WFBitMapS -> SyncSource ) [receive-bitmap]
  249. Mar 31 01:10:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: Began resync as SyncSource (will sync 1068 KB [267 bits set]).
  250. Mar 31 01:10:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: Resync done (total 1 sec; paused 0 sec; 1068 K/sec)
  251. Mar 31 01:10:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( SyncSource -> Established ) [resync-finished]
  252. Mar 31 01:10:15 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 461079150: 0->1 role( Secondary ) conn( Connected )
  253. Mar 31 01:10:15 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 461079150 (primary_nodes=8)
  254. Mar 31 01:10:15 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: resync-susp( no -> peer ) [peer-state]
  255. Mar 31 01:10:15 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: resync-susp( peer -> no ) [peer-state]
  256. Mar 31 01:10:16 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4185765531: 1->3 role( Secondary ) conn( Connected )
  257. Mar 31 01:10:16 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 4185765531 (primary_nodes=8)
  258. Mar 31 01:10:16 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: drbd_sync_handshake:
  259. Mar 31 01:10:16 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: self DD95BBD64DA6BD62:ADB250ED33B11982:52F5EAB2DCF34652:28D31133FEBC993A bits:230 flags:100
  260. Mar 31 01:10:16 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: peer ADB250ED33B11982:0000000000000000:3B36CC1B2C2CCBAC:52F5EAB2DCF34652 bits:267 flags:1384
  261. Mar 31 01:10:16 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: uuid_compare()=source-use-bitmap by rule=bitmap-self
  262. Mar 31 01:10:16 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: Becoming WFBitMapS because primary is diskless
  263. Mar 31 01:10:16 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( Established -> WFBitMapS ) [diskless-primary]
  264. Mar 31 01:10:16 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 21(1), total 21; compression: 100.0%
  265. Mar 31 01:10:16 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 37(1), total 37; compression: 99.9%
  266. Mar 31 01:10:16 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( WFBitMapS -> SyncSource ) [receive-bitmap]
  267. Mar 31 01:10:16 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: Began resync as SyncSource (will sync 1068 KB [267 bits set]).
  268. Mar 31 01:10:16 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: updated UUIDs DD95BBD64DA6BD62:0000000000000000:ADB250ED33B11982:52F5EAB2DCF34652
  269. Mar 31 01:10:16 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: Resync done (total 1 sec; paused 0 sec; 1068 K/sec)
  270. Mar 31 01:10:16 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( Inconsistent -> UpToDate ) repl( SyncSource -> Established ) [resync-finished]
  271. Mar 31 01:10:42 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4191436868: 0->all empty
  272. Mar 31 01:10:42 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( UpToDate -> Consistent ) [peer-state]
  273. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( no -> quorum ) [peer-state]
  274. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( yes -> no ) [peer-state]
  275. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( UpToDate -> Consistent ) [peer-state]
  276. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: PingAck did not arrive in time.
  277. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )
  278. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( UpToDate -> Consistent )
  279. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
  280. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d tcp:storage-load-test-0: Not enough data received; missing 2272 bytes
  281. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: error receiving P_DATA, e: -61 l: 4096!
  282. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: My uuid_flags changed from 0x100 to 0x120 during handshake.
  283. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Terminating sender thread
  284. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Starting sender thread (peer-node-id 3)
  285. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Connection closed
  286. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( NetworkFailure -> Unconnected ) [disconnected]
  287. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Restarting receiver thread
  288. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Unconnected -> Connecting ) [connecting]
  289. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Consistent -> Outdated ) [receive-peer-dagtag]
  290. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( Established -> WFBitMapT ) [receive-peer-dagtag]
  291. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Reconciliation resync because 'storage-load-test-0' disappeared. (o=-16)
  292. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Aborting remote state change 4191436868
  293. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1771109539: 2->all empty
  294. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( Established -> WFBitMapT ) [receive-peer-dagtag]
  295. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Reconciliation resync because 'storage-load-test-0' disappeared. (o=-16)
  296. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Aborting local state change 1771109539 to yield to remote state change 595626168.
  297. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 496ms
  298. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 595626168: 1->all empty
  299. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 30(1), total 30; compression: 100.0%
  300. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 30(1), total 30; compression: 100.0%
  301. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Outdated -> Inconsistent ) [receive-bitmap]
  302. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: resync-susp( no -> connection dependency ) [receive-bitmap]
  303. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: resync-susp( no -> connection dependency ) [receive-bitmap]
  304. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( WFBitMapT -> SyncTarget ) [receive-bitmap]
  305. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 595626168 (primary_nodes=0)
  306. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 871012577: 1->all empty
  307. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( Consistent -> UpToDate ) [peer-state]
  308. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: Began resync as SyncTarget (will sync 148 KB [37 bits set]).
  309. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 37(1), total 37; compression: 99.9%
  310. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 37(1), total 37; compression: 99.9%
  311. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( WFBitMapT -> PausedSyncT ) [receive-bitmap]
  312. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( quorum -> no ) [peer-state]
  313. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( no -> yes ) [peer-state]
  314. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( Consistent -> UpToDate ) [peer-state]
  315. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: Began resync as PausedSyncT (will sync 300 KB [75 bits set]).
  316. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 871012577 (primary_nodes=0)
  317. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: Resync done (total 1 sec; paused 0 sec; 148 K/sec)
  318. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: updated UUIDs DD95BBD64DA6BD62:0000000000000000:3B36CC1B2C2CCBAC:ADB250ED33B11982
  319. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Inconsistent -> UpToDate ) [resync-finished]
  320. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: resync-susp( connection dependency -> no ) [resync-finished]
  321. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( PausedSyncT -> Established ) resync-susp( connection dependency -> no ) [resync-finished]
  322. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( SyncTarget -> Established ) [resync-finished]
  323. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3701694474: 0->all empty
  324. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 3701694474 (primary_nodes=0)
  325. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1033669755: 0->all empty
  326. Mar 31 01:10:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 1033669755 (primary_nodes=0)
  327. Mar 31 01:10:44 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1413252624: 2->all empty
  328. Mar 31 01:10:44 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 1413252624: primary_nodes=0, weak_nodes=0
  329. Mar 31 01:10:44 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 1413252624 (0ms)
  330. Mar 31 01:10:44 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2834901056: 2->all empty
  331. Mar 31 01:10:44 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 2834901056: primary_nodes=0, weak_nodes=0
  332. Mar 31 01:10:44 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 2834901056 (0ms)
  333. Mar 31 01:11:12 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1914803476: 1->3 role( Secondary ) conn( Connected )
  334. Mar 31 01:11:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 1914803476 (primary_nodes=8)
  335. Mar 31 01:11:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3587873295: 3->all empty
  336. Mar 31 01:11:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 3587873295 (primary_nodes=8)
  337. Mar 31 01:11:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( UpToDate -> Outdated ) [far-away]
  338. Mar 31 01:11:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( no -> quorum ) [peer-state]
  339. Mar 31 01:11:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( yes -> no ) [peer-state]
  340. Mar 31 01:11:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( UpToDate -> Outdated ) [peer-state]
  341. Mar 31 01:11:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3330819532: 1->all empty
  342. Mar 31 01:11:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( UpToDate -> Consistent ) [peer-state]
  343. Mar 31 01:11:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Outdated -> Consistent ) [peer-state]
  344. Mar 31 01:11:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( Outdated -> Consistent ) [peer-state]
  345. Mar 31 01:11:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 3330819532 (primary_nodes=0)
  346. Mar 31 01:11:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1043972389: 1->all empty
  347. Mar 31 01:11:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( quorum -> no ) [peer-state]
  348. Mar 31 01:11:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Consistent -> UpToDate ) quorum( no -> yes ) [peer-state]
  349. Mar 31 01:11:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( Consistent -> UpToDate ) [peer-state]
  350. Mar 31 01:11:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( Consistent -> UpToDate ) [peer-state]
  351. Mar 31 01:11:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( UpToDate -> Outdated ) [peer-state]
  352. Mar 31 01:11:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( no -> quorum ) [receive-peer-dagtag]
  353. Mar 31 01:11:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( UpToDate -> Outdated ) quorum( yes -> no ) [receive-peer-dagtag]
  354. Mar 31 01:11:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( Established -> WFBitMapT ) [receive-peer-dagtag]
  355. Mar 31 01:11:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Reconciliation resync because 'storage-load-test-0' disappeared. (o=-40)
  356. Mar 31 01:11:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 1043972389 (primary_nodes=0)
  357. Mar 31 01:11:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1142912854: 1->3 role( Secondary ) conn( Connected )
  358. Mar 31 01:11:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 1142912854 (primary_nodes=8)
  359. Mar 31 01:11:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1970256621: 3->all empty
  360. Mar 31 01:11:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 1970256621 (primary_nodes=8)
  361. Mar 31 01:11:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Handshake to peer 3 successful: Agreed network protocol version 122
  362. Mar 31 01:11:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  363. Mar 31 01:11:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Peer authenticated using 20 bytes HMAC
  364. Mar 31 01:11:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1547400645: 2->3 role( Secondary ) conn( Connected )
  365. Mar 31 01:11:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: self DD95BBD64DA6BD62:28D31133FEBC993A:3B36CC1B2C2CCBAC:ADB250ED33B11982 bits:0 flags:0
  366. Mar 31 01:11:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: peer's exposed UUID: DD95BBD64DA6BD63
  367. Mar 31 01:11:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 1547400645: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF1
  368. Mar 31 01:11:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 1547400645 (36ms)
  369. Mar 31 01:11:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: Upgrading local disk to D_UP_TO_DATE since current UUID matches.
  370. Mar 31 01:11:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connecting -> Connected ) peer( Unknown -> Primary ) [connected]
  371. Mar 31 01:11:13 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( DUnknown -> Diskless ) repl( Off -> Established ) [connected]
  372. Mar 31 01:11:16 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1713475212: 0->3 role( Secondary ) conn( Connected )
  373. Mar 31 01:11:16 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 1713475212 (primary_nodes=8)
  374. Mar 31 01:11:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1652681402: 0->all empty
  375. Mar 31 01:11:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: PingAck did not arrive in time.
  376. Mar 31 01:11:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  377. Mar 31 01:11:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( Established -> Off )
  378. Mar 31 01:11:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Terminating sender thread
  379. Mar 31 01:11:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Starting sender thread (peer-node-id 0)
  380. Mar 31 01:11:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Aborting remote state change 1652681402 commit not possible
  381. Mar 31 01:11:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 4152775671: 2->all empty
  382. Mar 31 01:11:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Connection closed
  383. Mar 31 01:11:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( NetworkFailure -> Unconnected ) [disconnected]
  384. Mar 31 01:11:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Restarting receiver thread
  385. Mar 31 01:11:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Unconnected -> Connecting ) [connecting]
  386. Mar 31 01:11:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 4152775671: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF1
  387. Mar 31 01:11:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 4152775671 (8ms)
  388. Mar 31 01:11:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 797125860: 1->all empty
  389. Mar 31 01:11:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Rejecting concurrent remote state change 1902223197 because of state change 797125860
  390. Mar 31 01:11:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 797125860 (primary_nodes=8)
  391. Mar 31 01:11:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3924326760: 3->all empty
  392. Mar 31 01:11:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 3924326760 (primary_nodes=8)
  393. Mar 31 01:12:03 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Handshake to peer 0 successful: Agreed network protocol version 122
  394. Mar 31 01:12:03 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  395. Mar 31 01:12:03 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Peer authenticated using 20 bytes HMAC
  396. Mar 31 01:12:03 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2217267134: 0->2 role( Secondary ) conn( Connected )
  397. Mar 31 01:12:03 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: drbd_sync_handshake:
  398. Mar 31 01:12:03 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: self DD95BBD64DA6BD62:0000000000000000:3B36CC1B2C2CCBAC:ADB250ED33B11982 bits:0 flags:100
  399. Mar 31 01:12:03 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: peer DD95BBD64DA6BD62:0000000000000000:ADB250ED33B11982:3B36CC1B2C2CCBAC bits:0 flags:1120
  400. Mar 31 01:12:03 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: uuid_compare()=no-sync by rule=reconnected
  401. Mar 31 01:12:03 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 2217267134 (primary_nodes=8)
  402. Mar 31 01:12:03 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  403. Mar 31 01:12:03 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( Off -> Established ) [remote]
  404. Mar 31 01:12:03 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: cleared bm UUID and bitmap DD95BBD64DA6BD62:0000000000000000:3B36CC1B2C2CCBAC:ADB250ED33B11982
  405. Mar 31 01:12:08 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2130346017: 0->3 role( Secondary ) conn( Connected )
  406. Mar 31 01:12:08 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 2130346017 (primary_nodes=8)
  407. Mar 31 01:12:08 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( quorum -> no ) [peer-state]
  408. Mar 31 01:12:08 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( no -> yes ) [peer-state]
  409. Mar 31 01:12:08 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( Outdated -> UpToDate ) [peer-state]
  410. Mar 31 01:12:11 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1527465761: 0->1 role( Secondary ) conn( Connected )
  411. Mar 31 01:12:11 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 1527465761 (primary_nodes=8)
  412. Mar 31 01:12:24 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: PingAck did not arrive in time.
  413. Mar 31 01:12:24 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )
  414. Mar 31 01:12:24 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
  415. Mar 31 01:12:24 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3438499854: 2->all empty
  416. Mar 31 01:12:24 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Terminating sender thread
  417. Mar 31 01:12:24 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Starting sender thread (peer-node-id 3)
  418. Mar 31 01:12:24 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Rejecting concurrent remote state change 1389696837 because of state change 3438499854
  419. Mar 31 01:12:24 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Rejecting concurrent remote state change 1389696837 because of state change 3438499854
  420. Mar 31 01:12:24 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Connection closed
  421. Mar 31 01:12:24 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( NetworkFailure -> Unconnected ) [disconnected]
  422. Mar 31 01:12:24 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Restarting receiver thread
  423. Mar 31 01:12:24 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Unconnected -> Connecting ) [connecting]
  424. Mar 31 01:12:24 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Rejecting concurrent remote state change 2149326564 because of state change 3438499854
  425. Mar 31 01:12:24 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Rejecting concurrent remote state change 2149326564 because of state change 3438499854
  426. Mar 31 01:12:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: PingAck did not arrive in time.
  427. Mar 31 01:12:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( no -> quorum )
  428. Mar 31 01:12:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  429. Mar 31 01:12:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( yes -> no )
  430. Mar 31 01:12:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
  431. Mar 31 01:12:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: PingAck did not arrive in time.
  432. Mar 31 01:12:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  433. Mar 31 01:12:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( UpToDate -> DUnknown ) repl( WFBitMapT -> Off )
  434. Mar 31 01:12:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 56ms
  435. Mar 31 01:12:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Terminating sender thread
  436. Mar 31 01:12:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Terminating sender thread
  437. Mar 31 01:12:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Starting sender thread (peer-node-id 1)
  438. Mar 31 01:12:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Starting sender thread (peer-node-id 0)
  439. Mar 31 01:12:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Connection closed
  440. Mar 31 01:12:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( NetworkFailure -> Unconnected ) [disconnected]
  441. Mar 31 01:12:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Restarting receiver thread
  442. Mar 31 01:12:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Unconnected -> Connecting ) [connecting]
  443. Mar 31 01:12:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3915969191: 2->all empty
  444. Mar 31 01:12:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 3915969191 (0ms)
  445. Mar 31 01:12:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Connection closed
  446. Mar 31 01:12:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( NetworkFailure -> Unconnected ) [disconnected]
  447. Mar 31 01:12:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Restarting receiver thread
  448. Mar 31 01:12:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Unconnected -> Connecting ) [connecting]
  449. Mar 31 01:12:47 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Handshake to peer 3 successful: Agreed network protocol version 122
  450. Mar 31 01:12:47 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  451. Mar 31 01:12:47 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Peer authenticated using 20 bytes HMAC
  452. Mar 31 01:12:47 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 4282827581: 2->3 role( Secondary ) conn( Connected )
  453. Mar 31 01:12:47 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: self DD95BBD64DA6BD62:28D31133FEBC993A:3B36CC1B2C2CCBAC:ADB250ED33B11982 bits:0 flags:0
  454. Mar 31 01:12:47 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: peer's exposed UUID: DD95BBD64DA6BD63
  455. Mar 31 01:12:47 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 4282827581: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF0
  456. Mar 31 01:12:47 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 4282827581 (32ms)
  457. Mar 31 01:12:47 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: Upgrading local disk to D_UP_TO_DATE since current UUID matches.
  458. Mar 31 01:12:47 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connecting -> Connected ) peer( Unknown -> Primary ) [connected]
  459. Mar 31 01:12:47 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Outdated -> UpToDate ) [connected]
  460. Mar 31 01:12:47 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( DUnknown -> Diskless ) repl( Off -> Established ) [connected]
  461. Mar 31 01:12:47 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: received new current UUID: DD95BBD64DA6BD63 weak_nodes=FFFFFFFFFFFFFFFC
  462. Mar 31 01:12:47 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Handshake to peer 0 successful: Agreed network protocol version 122
  463. Mar 31 01:12:47 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  464. Mar 31 01:12:47 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Peer authenticated using 20 bytes HMAC
  465. Mar 31 01:12:47 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3429861694: 0->2 role( Secondary ) conn( Connected )
  466. Mar 31 01:12:47 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Handshake to peer 1 successful: Agreed network protocol version 122
  467. Mar 31 01:12:47 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  468. Mar 31 01:12:47 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Peer authenticated using 20 bytes HMAC
  469. Mar 31 01:12:47 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: drbd_sync_handshake:
  470. Mar 31 01:12:47 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: self DD95BBD64DA6BD62:DD95BBD64DA6BD63:3B36CC1B2C2CCBAC:ADB250ED33B11982 bits:0 flags:100
  471. Mar 31 01:12:47 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: peer DD95BBD64DA6BD62:DD95BBD64DA6BD62:ADB250ED33B11982:3B36CC1B2C2CCBAC bits:310 flags:1100
  472. Mar 31 01:12:47 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: uuid_compare()=no-sync by rule=reconnected
  473. Mar 31 01:12:47 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 3429861694 (primary_nodes=8)
  474. Mar 31 01:12:47 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( quorum -> no ) [remote]
  475. Mar 31 01:12:47 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  476. Mar 31 01:12:47 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( no -> yes ) [remote]
  477. Mar 31 01:12:47 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( DUnknown -> UpToDate ) repl( Off -> Established ) [remote]
  478. Mar 31 01:12:47 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: cleared bm UUID and bitmap DD95BBD64DA6BD62:0000000000000000:3B36CC1B2C2CCBAC:ADB250ED33B11982
  479. Mar 31 01:12:47 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1152771459: 1->2 role( Secondary ) conn( Connected )
  480. Mar 31 01:12:47 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: drbd_sync_handshake:
  481. Mar 31 01:12:47 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: self DD95BBD64DA6BD62:DD95BBD64DA6BD63:3B36CC1B2C2CCBAC:ADB250ED33B11982 bits:0 flags:100
  482. Mar 31 01:12:47 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: peer DD95BBD64DA6BD62:0000000000000000:3B36CC1B2C2CCBAC:52F5EAB2DCF34652 bits:315 flags:1100
  483. Mar 31 01:12:47 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: uuid_compare()=no-sync by rule=reconnected
  484. Mar 31 01:12:47 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 1152771459 (primary_nodes=8)
  485. Mar 31 01:12:47 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  486. Mar 31 01:12:47 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( DUnknown -> UpToDate ) repl( Off -> Established ) [remote]
  487. Mar 31 01:12:47 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: cleared bm UUID and bitmap DD95BBD64DA6BD62:0000000000000000:3B36CC1B2C2CCBAC:ADB250ED33B11982
  488. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3292241375: 0->all empty
  489. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( UpToDate -> Consistent ) [peer-state]
  490. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( no -> quorum ) [peer-state]
  491. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( yes -> no ) [peer-state]
  492. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( UpToDate -> Consistent ) [peer-state]
  493. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: PingAck did not arrive in time.
  494. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )
  495. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( UpToDate -> Consistent )
  496. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
  497. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: error receiving P_DATA, e: -512 l: 106496!
  498. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: My bitmap UUID changed during handshake. 0xDD95BBD64DA6BD63 to 0x0
  499. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: My uuid_flags changed from 0x100 to 0x120 during handshake.
  500. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Terminating sender thread
  501. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Starting sender thread (peer-node-id 3)
  502. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Connection closed
  503. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( NetworkFailure -> Unconnected ) [disconnected]
  504. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Restarting receiver thread
  505. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Unconnected -> Connecting ) [connecting]
  506. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: No reconciliation resync even though 'storage-load-test-0' disappeared. (o=0)
  507. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Consistent -> Outdated ) [receive-peer-dagtag]
  508. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( Established -> WFBitMapT ) [receive-peer-dagtag]
  509. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Reconciliation resync because 'storage-load-test-0' disappeared. (o=-208)
  510. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Aborting remote state change 3292241375
  511. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( Consistent -> Outdated ) [peer-state]
  512. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 4075443310: 2->all empty
  513. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Aborting local state change 4075443310 to yield to remote state change 4119964504.
  514. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 280ms
  515. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4119964504: 1->all empty
  516. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 30(1), total 30; compression: 100.0%
  517. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 30(1), total 30; compression: 100.0%
  518. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Outdated -> Inconsistent ) [receive-bitmap]
  519. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: resync-susp( no -> connection dependency ) [receive-bitmap]
  520. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: resync-susp( no -> connection dependency ) [receive-bitmap]
  521. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( WFBitMapT -> SyncTarget ) [receive-bitmap]
  522. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 4119964504 (primary_nodes=0)
  523. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( Consistent -> UpToDate ) [peer-state]
  524. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: Began resync as SyncTarget (will sync 132 KB [33 bits set]).
  525. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: updated UUIDs DD95BBD64DA6BD62:0000000000000000:3B36CC1B2C2CCBAC:ADB250ED33B11982
  526. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: My uuid_flags changed from 0x120 to 0x924 during handshake.
  527. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( Outdated -> Inconsistent ) [peer-state]
  528. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( Inconsistent -> Consistent ) [peer-state]
  529. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( quorum -> no ) [peer-state]
  530. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( no -> yes ) [peer-state]
  531. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( Consistent -> UpToDate ) [peer-state]
  532. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: Resync done (total 1 sec; paused 0 sec; 132 K/sec)
  533. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: updated UUIDs DD95BBD64DA6BD62:0000000000000000:3B36CC1B2C2CCBAC:ADB250ED33B11982
  534. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Inconsistent -> UpToDate ) [resync-finished]
  535. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: resync-susp( connection dependency -> no ) [resync-finished]
  536. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: resync-susp( connection dependency -> no ) [resync-finished]
  537. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( SyncTarget -> Established ) [resync-finished]
  538. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3079372999: 2->all empty
  539. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 3079372999: primary_nodes=0, weak_nodes=0
  540. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 3079372999 (0ms)
  541. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2335269304: 2->all empty
  542. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 2335269304: primary_nodes=0, weak_nodes=0
  543. Mar 31 01:13:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 2335269304 (0ms)
  544. Mar 31 01:13:06 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1444156437: 0->all empty
  545. Mar 31 01:13:06 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 1444156437 (primary_nodes=0)
  546. Mar 31 01:13:06 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3261655757: 0->all empty
  547. Mar 31 01:13:06 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 3261655757 (primary_nodes=0)
  548. Mar 31 01:13:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3125836916: 1->3 role( Secondary ) conn( Connected )
  549. Mar 31 01:13:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 3125836916 (primary_nodes=8)
  550. Mar 31 01:13:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2402789986: 3->all empty
  551. Mar 31 01:13:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 2402789986 (primary_nodes=8)
  552. Mar 31 01:13:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( UpToDate -> Outdated ) [far-away]
  553. Mar 31 01:13:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( no -> quorum ) [peer-state]
  554. Mar 31 01:13:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( yes -> no ) [peer-state]
  555. Mar 31 01:13:43 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( UpToDate -> Outdated ) [peer-state]
  556. Mar 31 01:13:46 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1570761256: 0->3 role( Secondary ) conn( Connected )
  557. Mar 31 01:13:46 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Aborting remote state change 1570761256
  558. Mar 31 01:13:47 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3138477124: 0->3 role( Secondary ) conn( Connected )
  559. Mar 31 01:13:47 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Aborting remote state change 3138477124
  560. Mar 31 01:13:48 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3535667735: 0->3 role( Secondary ) conn( Connected )
  561. Mar 31 01:13:48 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Aborting remote state change 3535667735
  562. Mar 31 01:13:48 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Handshake to peer 3 successful: Agreed network protocol version 122
  563. Mar 31 01:13:48 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  564. Mar 31 01:13:48 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Peer authenticated using 20 bytes HMAC
  565. Mar 31 01:13:48 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2072727487: 2->3 role( Secondary ) conn( Connected )
  566. Mar 31 01:13:48 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: self DD95BBD64DA6BD62:28D31133FEBC993A:3B36CC1B2C2CCBAC:ADB250ED33B11982 bits:0 flags:0
  567. Mar 31 01:13:48 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: peer's exposed UUID: DD95BBD64DA6BD63
  568. Mar 31 01:13:48 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Declined by peer storage-load-test-0 (id: 3), see the kernel log there
  569. Mar 31 01:13:48 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Aborting cluster-wide state change 2072727487 (24ms) rv = -10
  570. Mar 31 01:13:49 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3417612: 0->3 role( Secondary ) conn( Connected )
  571. Mar 31 01:13:49 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Aborting remote state change 3417612
  572. Mar 31 01:13:49 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 204289545: 2->3 role( Secondary ) conn( Connected )
  573. Mar 31 01:13:49 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: self DD95BBD64DA6BD62:28D31133FEBC993A:3B36CC1B2C2CCBAC:ADB250ED33B11982 bits:0 flags:0
  574. Mar 31 01:13:49 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: peer's exposed UUID: DD95BBD64DA6BD63
  575. Mar 31 01:13:49 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Declined by peer storage-load-test-0 (id: 3), see the kernel log there
  576. Mar 31 01:13:49 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Aborting cluster-wide state change 204289545 (40ms) rv = -10
  577. Mar 31 01:13:50 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2615676757: 0->3 role( Secondary ) conn( Connected )
  578. Mar 31 01:13:50 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Aborting remote state change 2615676757
  579. Mar 31 01:13:50 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2561739627: 2->3 role( Secondary ) conn( Connected )
  580. Mar 31 01:13:50 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: self DD95BBD64DA6BD62:28D31133FEBC993A:3B36CC1B2C2CCBAC:ADB250ED33B11982 bits:0 flags:0
  581. Mar 31 01:13:50 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: peer's exposed UUID: DD95BBD64DA6BD63
  582. Mar 31 01:13:50 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Declined by peer storage-load-test-0 (id: 3), see the kernel log there
  583. Mar 31 01:13:50 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Aborting cluster-wide state change 2561739627 (32ms) rv = -10
  584. Mar 31 01:13:51 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2363701962: 0->3 role( Secondary ) conn( Connected )
  585. Mar 31 01:13:51 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Aborting remote state change 2363701962
  586. Mar 31 01:13:51 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1348648508: 2->3 role( Secondary ) conn( Connected )
  587. Mar 31 01:13:51 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: self DD95BBD64DA6BD62:28D31133FEBC993A:3B36CC1B2C2CCBAC:ADB250ED33B11982 bits:0 flags:0
  588. Mar 31 01:13:51 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: peer's exposed UUID: DD95BBD64DA6BD63
  589. Mar 31 01:13:51 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Declined by peer storage-load-test-0 (id: 3), see the kernel log there
  590. Mar 31 01:13:51 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Aborting cluster-wide state change 1348648508 (36ms) rv = -10
  591. Mar 31 01:13:52 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3862016044: 0->all empty
  592. Mar 31 01:13:52 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Rejecting concurrent remote state change 3246052411 because of state change 3862016044
  593. Mar 31 01:13:52 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Rejecting concurrent remote state change 3276519458 because of state change 3862016044
  594. Mar 31 01:13:52 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: PingAck did not arrive in time.
  595. Mar 31 01:13:52 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  596. Mar 31 01:13:52 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
  597. Mar 31 01:13:52 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Terminating sender thread
  598. Mar 31 01:13:52 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Starting sender thread (peer-node-id 1)
  599. Mar 31 01:13:52 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Aborting remote state change 3862016044
  600. Mar 31 01:13:52 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2331013833: 2->all empty
  601. Mar 31 01:13:52 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 2331013833: primary_nodes=0, weak_nodes=0
  602. Mar 31 01:13:52 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 2331013833 (0ms)
  603. Mar 31 01:13:52 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3115004085: 0->3 role( Secondary ) conn( Connected )
  604. Mar 31 01:13:52 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Connection closed
  605. Mar 31 01:13:52 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( NetworkFailure -> Unconnected ) [disconnected]
  606. Mar 31 01:13:52 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Restarting receiver thread
  607. Mar 31 01:13:52 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Unconnected -> Connecting ) [connecting]
  608. Mar 31 01:13:52 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Aborting remote state change 3115004085
  609. Mar 31 01:13:52 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 4263931465: 2->3 role( Secondary ) conn( Connected )
  610. Mar 31 01:13:52 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: self DD95BBD64DA6BD62:28D31133FEBC993A:3B36CC1B2C2CCBAC:ADB250ED33B11982 bits:0 flags:0
  611. Mar 31 01:13:52 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: peer's exposed UUID: DD95BBD64DA6BD63
  612. Mar 31 01:13:52 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 4263931465: primary_nodes=8, weak_nodes=0
  613. Mar 31 01:13:52 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 4263931465 (36ms)
  614. Mar 31 01:13:52 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: Upgrading local disk to D_UP_TO_DATE since current UUID matches.
  615. Mar 31 01:13:52 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connecting -> Connected ) peer( Unknown -> Primary ) [connected]
  616. Mar 31 01:13:52 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Outdated -> UpToDate ) [connected]
  617. Mar 31 01:13:52 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( DUnknown -> Diskless ) repl( Off -> Established ) [connected]
  618. Mar 31 01:13:52 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: received new current UUID: DD95BBD64DA6BD63 weak_nodes=FFFFFFFFFFFFFFFF
  619. Mar 31 01:13:52 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4025883806: 0->all empty
  620. Mar 31 01:13:52 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 4025883806 (primary_nodes=8)
  621. Mar 31 01:13:52 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3928376880: 3->all empty
  622. Mar 31 01:13:52 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Aborting remote state change 3928376880
  623. Mar 31 01:13:53 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3773917208: 3->all empty
  624. Mar 31 01:13:53 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 3773917208 (primary_nodes=8)
  625. Mar 31 01:13:53 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 417657209: 0->3 role( Secondary ) conn( Connected )
  626. Mar 31 01:13:53 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 417657209 (primary_nodes=8)
  627. Mar 31 01:13:53 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: updated UUIDs DD95BBD64DA6BD62:0000000000000000:3B36CC1B2C2CCBAC:ADB250ED33B11982
  628. Mar 31 01:13:53 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( quorum -> no ) [peer-state]
  629. Mar 31 01:13:53 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( no -> yes ) [peer-state]
  630. Mar 31 01:13:53 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( Outdated -> UpToDate ) [peer-state]
  631. Mar 31 01:14:35 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3446771404: 1->3 role( Secondary ) conn( Connected )
  632. Mar 31 01:14:35 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 3446771404 (primary_nodes=8)
  633. Mar 31 01:14:36 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 864936807: 0->1 role( Secondary ) conn( Connected )
  634. Mar 31 01:14:36 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 864936807 (primary_nodes=8)
  635. Mar 31 01:14:38 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Handshake to peer 1 successful: Agreed network protocol version 122
  636. Mar 31 01:14:38 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  637. Mar 31 01:14:38 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Peer authenticated using 20 bytes HMAC
  638. Mar 31 01:14:38 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1444235534: 1->2 role( Secondary ) conn( Connected )
  639. Mar 31 01:14:38 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: drbd_sync_handshake:
  640. Mar 31 01:14:38 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: self DD95BBD64DA6BD62:0000000000000000:3B36CC1B2C2CCBAC:ADB250ED33B11982 bits:342 flags:100
  641. Mar 31 01:14:38 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: peer DD95BBD64DA6BD62:DD95BBD64DA6BD63:3B36CC1B2C2CCBAC:52F5EAB2DCF34652 bits:50 flags:1100
  642. Mar 31 01:14:38 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: uuid_compare()=no-sync by rule=reconnected
  643. Mar 31 01:14:38 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 1444235534 (primary_nodes=8)
  644. Mar 31 01:14:38 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  645. Mar 31 01:14:38 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( DUnknown -> UpToDate ) repl( Off -> Established ) [remote]
  646. Mar 31 01:14:38 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: cleared bm UUID and bitmap DD95BBD64DA6BD62:0000000000000000:3B36CC1B2C2CCBAC:ADB250ED33B11982
  647. Mar 31 01:14:38 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: bitmap WRITE of 2 pages took 8 ms
  648. Mar 31 01:14:56 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: PingAck did not arrive in time.
  649. Mar 31 01:14:56 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  650. Mar 31 01:14:56 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
  651. Mar 31 01:14:56 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3111857493: 2->all empty
  652. Mar 31 01:14:56 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Terminating sender thread
  653. Mar 31 01:14:56 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Starting sender thread (peer-node-id 0)
  654. Mar 31 01:14:56 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 3111857493: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF0
  655. Mar 31 01:14:56 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 3111857493 (4ms)
  656. Mar 31 01:14:56 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Connection closed
  657. Mar 31 01:14:56 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( NetworkFailure -> Unconnected ) [disconnected]
  658. Mar 31 01:14:56 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Restarting receiver thread
  659. Mar 31 01:14:56 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Unconnected -> Connecting ) [connecting]
  660. Mar 31 01:14:57 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1421634337: 3->all empty
  661. Mar 31 01:14:57 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Aborting remote state change 1421634337
  662. Mar 31 01:14:57 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3701488114: 1->all empty
  663. Mar 31 01:14:57 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 3701488114 (primary_nodes=8)
  664. Mar 31 01:14:57 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 216853111: 3->all empty
  665. Mar 31 01:14:57 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 216853111 (primary_nodes=8)
  666. Mar 31 01:14:59 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: received new current UUID: D8796B7E2A90EF4D weak_nodes=FFFFFFFFFFFFFFF9
  667. Mar 31 01:15:16 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1653285823: 0->3 role( Secondary ) conn( Connected )
  668. Mar 31 01:15:16 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 1653285823 (primary_nodes=8)
  669. Mar 31 01:15:16 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3579449599: 0->1 role( Secondary ) conn( Connected )
  670. Mar 31 01:15:16 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Duplicate prepare for remote state change 3579449599
  671. Mar 31 01:15:17 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 3579449599 (primary_nodes=8)
  672. Mar 31 01:15:17 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: clearing bitmap UUID and content (48 bits) for node 0 (storage-load-test-2)(slot 0)
  673. Mar 31 01:15:17 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: bitmap WRITE of 2 pages took 8 ms
  674. Mar 31 01:15:19 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Handshake to peer 0 successful: Agreed network protocol version 122
  675. Mar 31 01:15:19 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  676. Mar 31 01:15:19 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Peer authenticated using 20 bytes HMAC
  677. Mar 31 01:15:19 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2130924816: 0->2 role( Secondary ) conn( Connected )
  678. Mar 31 01:15:19 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: drbd_sync_handshake:
  679. Mar 31 01:15:19 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: self D8796B7E2A90EF4C:0000000000000000:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC bits:0 flags:100
  680. Mar 31 01:15:19 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: peer D8796B7E2A90EF4C:0000000000000000:ADB250ED33B11982:3B36CC1B2C2CCBAC bits:0 flags:1100
  681. Mar 31 01:15:19 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: uuid_compare()=no-sync by rule=reconnected
  682. Mar 31 01:15:19 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 2130924816 (primary_nodes=8)
  683. Mar 31 01:15:19 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  684. Mar 31 01:15:19 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( DUnknown -> UpToDate ) repl( Off -> Established ) [remote]
  685. Mar 31 01:15:19 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: cleared bm UUID and bitmap D8796B7E2A90EF4C:0000000000000000:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC
  686. Mar 31 01:15:29 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: PingAck did not arrive in time.
  687. Mar 31 01:15:29 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  688. Mar 31 01:15:29 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
  689. Mar 31 01:15:29 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 821105369: 2->all empty
  690. Mar 31 01:15:29 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Terminating sender thread
  691. Mar 31 01:15:29 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Starting sender thread (peer-node-id 0)
  692. Mar 31 01:15:29 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Aborting local state change 821105369 to yield to remote state change 4158750261.
  693. Mar 31 01:15:29 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 228ms
  694. Mar 31 01:15:29 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4158750261: 0->all empty
  695. Mar 31 01:15:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Connection closed
  696. Mar 31 01:15:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( NetworkFailure -> Unconnected ) [disconnected]
  697. Mar 31 01:15:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Restarting receiver thread
  698. Mar 31 01:15:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Unconnected -> Connecting ) [connecting]
  699. Mar 31 01:15:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: PingAck did not arrive in time.
  700. Mar 31 01:15:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )
  701. Mar 31 01:15:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( UpToDate -> Consistent )
  702. Mar 31 01:15:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
  703. Mar 31 01:15:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: PingAck did not arrive in time.
  704. Mar 31 01:15:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( no -> quorum )
  705. Mar 31 01:15:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  706. Mar 31 01:15:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( yes -> no )
  707. Mar 31 01:15:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
  708. Mar 31 01:15:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: error receiving P_DATA, e: -512 l: 106496!
  709. Mar 31 01:15:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Terminating sender thread
  710. Mar 31 01:15:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Starting sender thread (peer-node-id 1)
  711. Mar 31 01:15:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Terminating sender thread
  712. Mar 31 01:15:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Starting sender thread (peer-node-id 3)
  713. Mar 31 01:15:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Aborting remote state change 4158750261 commit not possible
  714. Mar 31 01:15:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2175579844: 2->all empty
  715. Mar 31 01:15:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 2175579844 (0ms)
  716. Mar 31 01:15:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Consistent -> UpToDate ) [lost-peer]
  717. Mar 31 01:15:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Connection closed
  718. Mar 31 01:15:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( NetworkFailure -> Unconnected ) [disconnected]
  719. Mar 31 01:15:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Restarting receiver thread
  720. Mar 31 01:15:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Unconnected -> Connecting ) [connecting]
  721. Mar 31 01:15:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Connection closed
  722. Mar 31 01:15:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( NetworkFailure -> Unconnected ) [disconnected]
  723. Mar 31 01:15:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Restarting receiver thread
  724. Mar 31 01:15:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Unconnected -> Connecting ) [connecting]
  725. Mar 31 01:16:01 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Handshake to peer 3 successful: Agreed network protocol version 122
  726. Mar 31 01:16:01 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  727. Mar 31 01:16:01 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Peer authenticated using 20 bytes HMAC
  728. Mar 31 01:16:01 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1631962529: 2->3 role( Secondary ) conn( Connected )
  729. Mar 31 01:16:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: self D8796B7E2A90EF4C:28D31133FEBC993A:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC bits:0 flags:0
  730. Mar 31 01:16:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: peer's exposed UUID: 752996FB4D06CB51
  731. Mar 31 01:16:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 1631962529: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF0
  732. Mar 31 01:16:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 1631962529 (36ms)
  733. Mar 31 01:16:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connecting -> Connected ) peer( Unknown -> Primary ) [connected]
  734. Mar 31 01:16:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( UpToDate -> Outdated ) [connected]
  735. Mar 31 01:16:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( DUnknown -> Diskless ) repl( Off -> Established ) [connected]
  736. Mar 31 01:16:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Handshake to peer 0 successful: Agreed network protocol version 122
  737. Mar 31 01:16:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  738. Mar 31 01:16:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Peer authenticated using 20 bytes HMAC
  739. Mar 31 01:16:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3369737057: 0->2 role( Secondary ) conn( Connected )
  740. Mar 31 01:16:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: drbd_sync_handshake:
  741. Mar 31 01:16:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: self D8796B7E2A90EF4C:0000000000000000:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC bits:7 flags:100
  742. Mar 31 01:16:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: peer 752996FB4D06CB50:D8796B7E2A90EF4C:ADB250ED33B11982:3B36CC1B2C2CCBAC bits:265 flags:1100
  743. Mar 31 01:16:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: uuid_compare()=target-use-bitmap by rule=bitmap-peer
  744. Mar 31 01:16:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 3369737057 (primary_nodes=8)
  745. Mar 31 01:16:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  746. Mar 31 01:16:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( DUnknown -> UpToDate ) repl( Off -> WFBitMapT ) [remote]
  747. Mar 31 01:16:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: Setting exposed data uuid: 752996FB4D06CB50
  748. Mar 31 01:16:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 40(1), total 40; compression: 99.9%
  749. Mar 31 01:16:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 40(1), total 40; compression: 99.9%
  750. Mar 31 01:16:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Outdated -> Inconsistent ) [receive-bitmap]
  751. Mar 31 01:16:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: resync-susp( no -> connection dependency ) [receive-bitmap]
  752. Mar 31 01:16:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( WFBitMapT -> SyncTarget ) [receive-bitmap]
  753. Mar 31 01:16:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: resync-susp( no -> connection dependency ) [receive-bitmap]
  754. Mar 31 01:16:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: Began resync as SyncTarget (will sync 1088 KB [272 bits set]).
  755. Mar 31 01:16:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: Resync done (total 1 sec; paused 0 sec; 1088 K/sec)
  756. Mar 31 01:16:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: updated UUIDs 752996FB4D06CB50:0000000000000000:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC
  757. Mar 31 01:16:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( quorum -> no ) [resync-finished]
  758. Mar 31 01:16:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Inconsistent -> UpToDate ) quorum( no -> yes ) [resync-finished]
  759. Mar 31 01:16:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: resync-susp( connection dependency -> no ) [resync-finished]
  760. Mar 31 01:16:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( SyncTarget -> Established ) [resync-finished]
  761. Mar 31 01:16:02 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: resync-susp( connection dependency -> no ) [resync-finished]
  762. Mar 31 01:16:04 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Handshake to peer 1 successful: Agreed network protocol version 122
  763. Mar 31 01:16:04 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  764. Mar 31 01:16:04 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Peer authenticated using 20 bytes HMAC
  765. Mar 31 01:16:04 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2284281406: 1->2 role( Secondary ) conn( Connected )
  766. Mar 31 01:16:04 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: drbd_sync_handshake:
  767. Mar 31 01:16:04 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: self 752996FB4D06CB50:0000000000000000:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC bits:0 flags:100
  768. Mar 31 01:16:04 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: peer 752996FB4D06CB50:0000000000000000:D8796B7E2A90EF4C:DD95BBD64DA6BD62 bits:0 flags:1100
  769. Mar 31 01:16:04 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: uuid_compare()=no-sync by rule=reconnected
  770. Mar 31 01:16:04 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 2284281406 (primary_nodes=8)
  771. Mar 31 01:16:04 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  772. Mar 31 01:16:04 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( DUnknown -> UpToDate ) repl( Off -> Established ) [remote]
  773. Mar 31 01:16:04 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: cleared bm UUID and bitmap 752996FB4D06CB50:0000000000000000:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC
  774. Mar 31 01:16:26 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1388789927: 0->all empty
  775. Mar 31 01:16:26 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( UpToDate -> Consistent ) [peer-state]
  776. Mar 31 01:16:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( no -> quorum ) [peer-state]
  777. Mar 31 01:16:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( yes -> no ) [peer-state]
  778. Mar 31 01:16:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( UpToDate -> Consistent ) [peer-state]
  779. Mar 31 01:16:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: PingAck did not arrive in time.
  780. Mar 31 01:16:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )
  781. Mar 31 01:16:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( UpToDate -> Consistent )
  782. Mar 31 01:16:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
  783. Mar 31 01:16:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: My uuid_flags changed from 0x100 to 0x120 during handshake.
  784. Mar 31 01:16:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Terminating sender thread
  785. Mar 31 01:16:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Starting sender thread (peer-node-id 3)
  786. Mar 31 01:16:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Connection closed
  787. Mar 31 01:16:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( NetworkFailure -> Unconnected ) [disconnected]
  788. Mar 31 01:16:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Restarting receiver thread
  789. Mar 31 01:16:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Unconnected -> Connecting ) [connecting]
  790. Mar 31 01:16:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: No reconciliation resync even though 'storage-load-test-0' disappeared. (o=0)
  791. Mar 31 01:16:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Aborting remote state change 1388789927
  792. Mar 31 01:16:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( Established -> WFBitMapS ) [receive-peer-dagtag]
  793. Mar 31 01:16:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Reconciliation resync because 'storage-load-test-0' disappeared. (o=208)
  794. Mar 31 01:16:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3626300921: 1->all empty
  795. Mar 31 01:16:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( Consistent -> Outdated ) [peer-state]
  796. Mar 31 01:16:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 30(1), total 30; compression: 100.0%
  797. Mar 31 01:16:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 3626300921 (primary_nodes=0)
  798. Mar 31 01:16:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( Outdated -> Inconsistent ) resync-susp( no -> peer ) [peer-state]
  799. Mar 31 01:16:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2248437662: 2->all empty
  800. Mar 31 01:16:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 300ms
  801. Mar 31 01:16:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( Inconsistent -> Consistent ) resync-susp( peer -> no ) [peer-state]
  802. Mar 31 01:16:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 111167260: 2->all empty
  803. Mar 31 01:16:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 111167260: primary_nodes=0, weak_nodes=0
  804. Mar 31 01:16:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 111167260 (0ms)
  805. Mar 31 01:16:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Consistent -> UpToDate ) [lost-peer]
  806. Mar 31 01:16:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 194935479: 2->all empty
  807. Mar 31 01:16:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 194935479: primary_nodes=0, weak_nodes=0
  808. Mar 31 01:16:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 194935479 (0ms)
  809. Mar 31 01:16:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( quorum -> no ) [peer-state]
  810. Mar 31 01:16:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( no -> yes ) [peer-state]
  811. Mar 31 01:16:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( Consistent -> UpToDate ) [peer-state]
  812. Mar 31 01:16:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2038869718: 0->all empty
  813. Mar 31 01:16:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 2038869718 (primary_nodes=0)
  814. Mar 31 01:16:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2081815640: 0->all empty
  815. Mar 31 01:16:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 2081815640 (primary_nodes=0)
  816. Mar 31 01:16:37 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3768514093: 0->3 role( Secondary ) conn( Connected )
  817. Mar 31 01:16:37 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 3768514093 (primary_nodes=8)
  818. Mar 31 01:16:37 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 88211161: 0->all empty
  819. Mar 31 01:16:37 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( no -> quorum ) [peer-state]
  820. Mar 31 01:16:37 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( yes -> no ) [peer-state]
  821. Mar 31 01:16:37 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( UpToDate -> Consistent ) [peer-state]
  822. Mar 31 01:16:37 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 88211161 (primary_nodes=0)
  823. Mar 31 01:16:37 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( quorum -> no ) [peer-state]
  824. Mar 31 01:16:37 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( no -> yes ) [peer-state]
  825. Mar 31 01:16:37 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( Consistent -> UpToDate ) [peer-state]
  826. Mar 31 01:16:37 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 718664504: 0->all empty
  827. Mar 31 01:16:37 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 718664504 (primary_nodes=0)
  828. Mar 31 01:16:37 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( no -> quorum ) [receive-peer-dagtag]
  829. Mar 31 01:16:37 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( UpToDate -> Outdated ) quorum( yes -> no ) [receive-peer-dagtag]
  830. Mar 31 01:16:37 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( Established -> WFBitMapT ) [receive-peer-dagtag]
  831. Mar 31 01:16:37 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Reconciliation resync because 'storage-load-test-0' disappeared. (o=-32)
  832. Mar 31 01:16:37 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( Consistent -> Outdated ) [peer-state]
  833. Mar 31 01:16:37 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 778947052: 1->3 role( Secondary ) conn( Connected )
  834. Mar 31 01:16:37 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 778947052 (primary_nodes=8)
  835. Mar 31 01:16:38 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Handshake to peer 3 successful: Agreed network protocol version 122
  836. Mar 31 01:16:38 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  837. Mar 31 01:16:38 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Peer authenticated using 20 bytes HMAC
  838. Mar 31 01:16:38 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2467925668: 2->3 role( Secondary ) conn( Connected )
  839. Mar 31 01:16:38 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: self 752996FB4D06CB50:28D31133FEBC993A:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC bits:0 flags:0
  840. Mar 31 01:16:38 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: peer's exposed UUID: 28983A7814F88E65
  841. Mar 31 01:16:38 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 2467925668: primary_nodes=8, weak_nodes=0
  842. Mar 31 01:16:38 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 2467925668 (32ms)
  843. Mar 31 01:16:38 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connecting -> Connected ) peer( Unknown -> Primary ) [connected]
  844. Mar 31 01:16:38 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( DUnknown -> Diskless ) repl( Off -> Established ) [connected]
  845. Mar 31 01:16:38 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: strategy = no-sync
  846. Mar 31 01:16:38 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 433044289: 3->all empty
  847. Mar 31 01:16:38 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 433044289 (primary_nodes=8)
  848. Mar 31 01:16:38 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 722555851: 3->all empty
  849. Mar 31 01:16:38 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 722555851 (primary_nodes=8)
  850. Mar 31 01:16:38 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2386366568: 0->3 role( Secondary ) conn( Connected )
  851. Mar 31 01:16:38 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 2386366568 (primary_nodes=8)
  852. Mar 31 01:16:38 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( UpToDate -> Outdated ) [peer-state]
  853. Mar 31 01:16:38 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1209087830: 3->all empty
  854. Mar 31 01:16:38 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Aborting remote state change 1209087830
  855. Mar 31 01:16:38 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3472239397: 0->all empty
  856. Mar 31 01:16:38 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 3472239397 (primary_nodes=8)
  857. Mar 31 01:16:38 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2761915777: 3->all empty
  858. Mar 31 01:16:38 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 2761915777 (primary_nodes=8)
  859. Mar 31 01:16:38 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1779301481: 3->all empty
  860. Mar 31 01:16:38 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 1779301481 (primary_nodes=8)
  861. Mar 31 01:16:58 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: PingAck did not arrive in time.
  862. Mar 31 01:16:58 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  863. Mar 31 01:16:58 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( WFBitMapS -> Off )
  864. Mar 31 01:16:58 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1400865519: 2->all empty
  865. Mar 31 01:16:58 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Terminating sender thread
  866. Mar 31 01:16:58 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Starting sender thread (peer-node-id 1)
  867. Mar 31 01:16:58 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Connection closed
  868. Mar 31 01:16:58 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( NetworkFailure -> Unconnected ) [disconnected]
  869. Mar 31 01:16:58 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Restarting receiver thread
  870. Mar 31 01:16:58 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Unconnected -> Connecting ) [connecting]
  871. Mar 31 01:16:59 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 12ms
  872. Mar 31 01:16:59 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2703312900: 0->all empty
  873. Mar 31 01:16:59 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Aborting remote state change 2703312900
  874. Mar 31 01:16:59 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2293185259: 2->all empty
  875. Mar 31 01:16:59 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Rejecting concurrent remote state change 525779288 because of state change 2293185259
  876. Mar 31 01:16:59 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 2293185259: primary_nodes=8, weak_nodes=0
  877. Mar 31 01:16:59 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 2293185259 (0ms)
  878. Mar 31 01:16:59 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4138258921: 3->all empty
  879. Mar 31 01:16:59 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 4138258921 (primary_nodes=8)
  880. Mar 31 01:16:59 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 204760639: 0->all empty
  881. Mar 31 01:16:59 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 204760639 (primary_nodes=8)
  882. Mar 31 01:17:28 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Handshake to peer 1 successful: Agreed network protocol version 122
  883. Mar 31 01:17:28 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  884. Mar 31 01:17:28 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Peer authenticated using 20 bytes HMAC
  885. Mar 31 01:17:28 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 332174034: 1->2 role( Secondary ) conn( Connected )
  886. Mar 31 01:17:28 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: drbd_sync_handshake:
  887. Mar 31 01:17:28 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: self 752996FB4D06CB50:0000000000000000:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC bits:0 flags:100
  888. Mar 31 01:17:28 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: peer 752996FB4D06CB50:0000000000000000:ADB250ED33B11982:D8796B7E2A90EF4C bits:0 flags:1120
  889. Mar 31 01:17:28 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: uuid_compare()=no-sync by rule=reconnected
  890. Mar 31 01:17:28 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 332174034 (primary_nodes=8)
  891. Mar 31 01:17:28 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  892. Mar 31 01:17:28 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( Off -> Established ) [remote]
  893. Mar 31 01:17:28 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: cleared bm UUID and bitmap 752996FB4D06CB50:0000000000000000:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC
  894. Mar 31 01:17:28 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3438731863: 0->1 role( Secondary ) conn( Connected )
  895. Mar 31 01:17:28 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 3438731863 (primary_nodes=8)
  896. Mar 31 01:17:28 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 369871165: 1->3 role( Secondary ) conn( Connected )
  897. Mar 31 01:17:28 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 369871165 (primary_nodes=8)
  898. Mar 31 01:17:49 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 581687972: 1->all empty
  899. Mar 31 01:17:49 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Rejecting concurrent remote state change 3193316345 because of state change 581687972
  900. Mar 31 01:17:50 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: PingAck did not arrive in time.
  901. Mar 31 01:17:50 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  902. Mar 31 01:17:50 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( WFBitMapT -> Off )
  903. Mar 31 01:17:50 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Terminating sender thread
  904. Mar 31 01:17:50 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Starting sender thread (peer-node-id 0)
  905. Mar 31 01:17:50 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Aborting remote state change 581687972
  906. Mar 31 01:17:50 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3634414869: 2->all empty
  907. Mar 31 01:17:50 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 3634414869: primary_nodes=8, weak_nodes=0
  908. Mar 31 01:17:50 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 3634414869 (0ms)
  909. Mar 31 01:17:50 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Connection closed
  910. Mar 31 01:17:50 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( NetworkFailure -> Unconnected ) [disconnected]
  911. Mar 31 01:17:50 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Restarting receiver thread
  912. Mar 31 01:17:50 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Unconnected -> Connecting ) [connecting]
  913. Mar 31 01:17:50 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1347114405: 1->all empty
  914. Mar 31 01:17:50 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 1347114405 (primary_nodes=8)
  915. Mar 31 01:18:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Handshake to peer 0 successful: Agreed network protocol version 122
  916. Mar 31 01:18:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  917. Mar 31 01:18:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Peer authenticated using 20 bytes HMAC
  918. Mar 31 01:18:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1603385043: 0->2 role( Secondary ) conn( Connected )
  919. Mar 31 01:18:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: drbd_sync_handshake:
  920. Mar 31 01:18:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: self 752996FB4D06CB50:0000000000000000:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC bits:0 flags:100
  921. Mar 31 01:18:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: peer 752996FB4D06CB50:0000000000000000:D8796B7E2A90EF4C:ADB250ED33B11982 bits:1 flags:1120
  922. Mar 31 01:18:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: uuid_compare()=no-sync by rule=reconnected
  923. Mar 31 01:18:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 1603385043 (primary_nodes=8)
  924. Mar 31 01:18:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  925. Mar 31 01:18:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( Off -> Established ) [remote]
  926. Mar 31 01:18:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: cleared bm UUID and bitmap 752996FB4D06CB50:0000000000000000:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC
  927. Mar 31 01:18:29 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2458447738: 0->1 role( Secondary ) conn( Connected )
  928. Mar 31 01:18:29 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 2458447738 (primary_nodes=8)
  929. Mar 31 01:18:39 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: PingAck did not arrive in time.
  930. Mar 31 01:18:39 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  931. Mar 31 01:18:39 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( Established -> Off )
  932. Mar 31 01:18:39 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: PingAck did not arrive in time.
  933. Mar 31 01:18:39 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )
  934. Mar 31 01:18:39 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
  935. Mar 31 01:18:39 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3650973266: 2->all empty
  936. Mar 31 01:18:39 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Terminating sender thread
  937. Mar 31 01:18:39 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Terminating sender thread
  938. Mar 31 01:18:39 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Starting sender thread (peer-node-id 3)
  939. Mar 31 01:18:39 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Starting sender thread (peer-node-id 0)
  940. Mar 31 01:18:39 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Aborting local state change 3650973266 to yield to remote state change 2710131752.
  941. Mar 31 01:18:39 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 32ms
  942. Mar 31 01:18:39 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2710131752: 0->all empty
  943. Mar 31 01:18:39 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 2710131752 (primary_nodes=8)
  944. Mar 31 01:18:39 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1561974972: 3->all empty
  945. Mar 31 01:18:39 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 1561974972 (primary_nodes=8)
  946. Mar 31 01:18:39 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Connection closed
  947. Mar 31 01:18:39 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( NetworkFailure -> Unconnected ) [disconnected]
  948. Mar 31 01:18:39 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Restarting receiver thread
  949. Mar 31 01:18:39 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Unconnected -> Connecting ) [connecting]
  950. Mar 31 01:18:39 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3789656244: 2->all empty
  951. Mar 31 01:18:39 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Connection closed
  952. Mar 31 01:18:39 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( NetworkFailure -> Unconnected ) [disconnected]
  953. Mar 31 01:18:39 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Restarting receiver thread
  954. Mar 31 01:18:39 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Unconnected -> Connecting ) [connecting]
  955. Mar 31 01:18:40 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: sock was shut down by peer
  956. Mar 31 01:18:40 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connected -> BrokenPipe ) peer( Secondary -> Unknown )
  957. Mar 31 01:18:40 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( Established -> Off )
  958. Mar 31 01:18:40 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 768ms
  959. Mar 31 01:18:40 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Terminating sender thread
  960. Mar 31 01:18:40 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Starting sender thread (peer-node-id 1)
  961. Mar 31 01:18:40 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Connection closed
  962. Mar 31 01:18:40 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( BrokenPipe -> Unconnected ) [disconnected]
  963. Mar 31 01:18:40 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Restarting receiver thread
  964. Mar 31 01:18:40 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Unconnected -> Connecting ) [connecting]
  965. Mar 31 01:18:41 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3656044282: 2->all empty
  966. Mar 31 01:18:41 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 3656044282 (0ms)
  967. Mar 31 01:19:04 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Handshake to peer 0 successful: Agreed network protocol version 122
  968. Mar 31 01:19:04 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  969. Mar 31 01:19:04 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Peer authenticated using 20 bytes HMAC
  970. Mar 31 01:19:04 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4181492047: 0->2 role( Secondary ) conn( Connected )
  971. Mar 31 01:19:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: drbd_sync_handshake:
  972. Mar 31 01:19:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: self 752996FB4D06CB50:0000000000000000:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC bits:0 flags:120
  973. Mar 31 01:19:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: peer 752996FB4D06CB50:0000000000000000:D8796B7E2A90EF4C:ADB250ED33B11982 bits:0 flags:1120
  974. Mar 31 01:19:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: uuid_compare()=no-sync by rule=reconnected
  975. Mar 31 01:19:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 4181492047 (primary_nodes=8)
  976. Mar 31 01:19:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  977. Mar 31 01:19:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( Off -> Established ) [remote]
  978. Mar 31 01:19:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: cleared bm UUID and bitmap 752996FB4D06CB50:0000000000000000:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC
  979. Mar 31 01:19:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Handshake to peer 3 successful: Agreed network protocol version 122
  980. Mar 31 01:19:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  981. Mar 31 01:19:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Peer authenticated using 20 bytes HMAC
  982. Mar 31 01:19:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2159242530: 2->3 role( Secondary ) conn( Connected )
  983. Mar 31 01:19:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: self 752996FB4D06CB50:28D31133FEBC993A:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC bits:0 flags:0
  984. Mar 31 01:19:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: peer's exposed UUID: 28983A7814F88E65
  985. Mar 31 01:19:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 2159242530: primary_nodes=8, weak_nodes=0
  986. Mar 31 01:19:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 2159242530 (40ms)
  987. Mar 31 01:19:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connecting -> Connected ) peer( Unknown -> Primary ) [connected]
  988. Mar 31 01:19:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( DUnknown -> Diskless ) repl( Off -> Established ) [connected]
  989. Mar 31 01:19:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Handshake to peer 1 successful: Agreed network protocol version 122
  990. Mar 31 01:19:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  991. Mar 31 01:19:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Peer authenticated using 20 bytes HMAC
  992. Mar 31 01:19:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3871126125: 1->2 role( Secondary ) conn( Connected )
  993. Mar 31 01:19:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: drbd_sync_handshake:
  994. Mar 31 01:19:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: self 752996FB4D06CB50:0000000000000000:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC bits:0 flags:100
  995. Mar 31 01:19:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: peer 752996FB4D06CB50:0000000000000000:ADB250ED33B11982:D8796B7E2A90EF4C bits:0 flags:1100
  996. Mar 31 01:19:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: uuid_compare()=no-sync by rule=reconnected
  997. Mar 31 01:19:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 3871126125 (primary_nodes=8)
  998. Mar 31 01:19:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  999. Mar 31 01:19:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( Off -> Established ) [remote]
  1000. Mar 31 01:19:05 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: cleared bm UUID and bitmap 752996FB4D06CB50:0000000000000000:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC
  1001. Mar 31 01:19:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: PingAck did not arrive in time.
  1002. Mar 31 01:19:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )
  1003. Mar 31 01:19:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
  1004. Mar 31 01:19:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2693247151: 2->all empty
  1005. Mar 31 01:19:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Terminating sender thread
  1006. Mar 31 01:19:25 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Starting sender thread (peer-node-id 3)
  1007. Mar 31 01:19:26 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Connection closed
  1008. Mar 31 01:19:26 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( NetworkFailure -> Unconnected ) [disconnected]
  1009. Mar 31 01:19:26 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Restarting receiver thread
  1010. Mar 31 01:19:26 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Unconnected -> Connecting ) [connecting]
  1011. Mar 31 01:19:26 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 156ms
  1012. Mar 31 01:19:26 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 595089147: 1->all empty
  1013. Mar 31 01:19:26 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 595089147 (primary_nodes=0)
  1014. Mar 31 01:19:26 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 73580424: 2->all empty
  1015. Mar 31 01:19:26 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 73580424: primary_nodes=0, weak_nodes=0
  1016. Mar 31 01:19:26 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 73580424 (0ms)
  1017. Mar 31 01:20:00 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Handshake to peer 3 successful: Agreed network protocol version 122
  1018. Mar 31 01:20:00 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  1019. Mar 31 01:20:00 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Peer authenticated using 20 bytes HMAC
  1020. Mar 31 01:20:00 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1789966072: 2->3 role( Secondary ) conn( Connected )
  1021. Mar 31 01:20:00 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: self 752996FB4D06CB50:28D31133FEBC993A:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC bits:0 flags:0
  1022. Mar 31 01:20:00 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: peer's exposed UUID: 28983A7814F88E65
  1023. Mar 31 01:20:00 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 1789966072: primary_nodes=8, weak_nodes=0
  1024. Mar 31 01:20:00 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 1789966072 (40ms)
  1025. Mar 31 01:20:00 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connecting -> Connected ) peer( Unknown -> Primary ) [connected]
  1026. Mar 31 01:20:00 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( DUnknown -> Diskless ) repl( Off -> Established ) [connected]
  1027. Mar 31 01:20:07 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 178417790: 1->3 role( Secondary ) conn( Connected )
  1028. Mar 31 01:20:07 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 178417790 (primary_nodes=8)
  1029. Mar 31 01:20:18 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1409325880: 0->all empty
  1030. Mar 31 01:20:18 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Aborting remote state change 1409325880
  1031. Mar 31 01:20:18 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3320924655: 1->all empty
  1032. Mar 31 01:20:18 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: PingAck did not arrive in time.
  1033. Mar 31 01:20:18 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  1034. Mar 31 01:20:18 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( Established -> Off )
  1035. Mar 31 01:20:18 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Terminating sender thread
  1036. Mar 31 01:20:18 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Starting sender thread (peer-node-id 0)
  1037. Mar 31 01:20:18 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Aborting remote state change 3320924655
  1038. Mar 31 01:20:18 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3305118617: 2->all empty
  1039. Mar 31 01:20:18 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 3305118617: primary_nodes=8, weak_nodes=0
  1040. Mar 31 01:20:18 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 3305118617 (0ms)
  1041. Mar 31 01:20:19 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Connection closed
  1042. Mar 31 01:20:19 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( NetworkFailure -> Unconnected ) [disconnected]
  1043. Mar 31 01:20:19 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Restarting receiver thread
  1044. Mar 31 01:20:19 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Unconnected -> Connecting ) [connecting]
  1045. Mar 31 01:20:19 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3950734662: 1->all empty
  1046. Mar 31 01:20:19 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 3950734662 (primary_nodes=8)
  1047. Mar 31 01:20:56 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 26180170: 0->1 role( Secondary ) conn( Connected )
  1048. Mar 31 01:20:56 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 26180170 (primary_nodes=8)
  1049. Mar 31 01:21:00 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Handshake to peer 0 successful: Agreed network protocol version 122
  1050. Mar 31 01:21:00 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  1051. Mar 31 01:21:00 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Peer authenticated using 20 bytes HMAC
  1052. Mar 31 01:21:00 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 113751511: 0->2 role( Secondary ) conn( Connected )
  1053. Mar 31 01:21:00 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: drbd_sync_handshake:
  1054. Mar 31 01:21:00 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: self 752996FB4D06CB50:0000000000000000:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC bits:0 flags:100
  1055. Mar 31 01:21:00 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: peer 752996FB4D06CB50:0000000000000000:D8796B7E2A90EF4C:ADB250ED33B11982 bits:0 flags:1120
  1056. Mar 31 01:21:00 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: uuid_compare()=no-sync by rule=reconnected
  1057. Mar 31 01:21:00 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 113751511 (primary_nodes=8)
  1058. Mar 31 01:21:00 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  1059. Mar 31 01:21:00 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( Off -> Established ) [remote]
  1060. Mar 31 01:21:00 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: cleared bm UUID and bitmap 752996FB4D06CB50:0000000000000000:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC
  1061. Mar 31 01:21:07 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: PingAck did not arrive in time.
  1062. Mar 31 01:21:07 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  1063. Mar 31 01:21:07 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( Established -> Off )
  1064. Mar 31 01:21:07 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3394754177: 2->all empty
  1065. Mar 31 01:21:07 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Terminating sender thread
  1066. Mar 31 01:21:07 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Starting sender thread (peer-node-id 1)
  1067. Mar 31 01:21:07 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Connection closed
  1068. Mar 31 01:21:07 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( NetworkFailure -> Unconnected ) [disconnected]
  1069. Mar 31 01:21:07 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Restarting receiver thread
  1070. Mar 31 01:21:07 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Unconnected -> Connecting ) [connecting]
  1071. Mar 31 01:21:07 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 420ms
  1072. Mar 31 01:21:07 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2936172748: 3->all empty
  1073. Mar 31 01:21:07 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Rejecting concurrent remote state change 74975511 because of state change 2936172748
  1074. Mar 31 01:21:07 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 2936172748 (primary_nodes=8)
  1075. Mar 31 01:21:08 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 191871833: 0->all empty
  1076. Mar 31 01:21:08 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 191871833 (primary_nodes=8)
  1077. Mar 31 01:21:08 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1983572376: 2->all empty
  1078. Mar 31 01:21:08 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 1983572376: primary_nodes=8, weak_nodes=0
  1079. Mar 31 01:21:08 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 1983572376 (0ms)
  1080. Mar 31 01:21:32 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Handshake to peer 1 successful: Agreed network protocol version 122
  1081. Mar 31 01:21:32 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  1082. Mar 31 01:21:32 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Peer authenticated using 20 bytes HMAC
  1083. Mar 31 01:21:32 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1146113168: 1->2 role( Secondary ) conn( Connected )
  1084. Mar 31 01:21:32 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: drbd_sync_handshake:
  1085. Mar 31 01:21:32 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: self 752996FB4D06CB50:0000000000000000:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC bits:0 flags:100
  1086. Mar 31 01:21:32 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: peer 752996FB4D06CB50:0000000000000000:ADB250ED33B11982:D8796B7E2A90EF4C bits:0 flags:1120
  1087. Mar 31 01:21:32 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: uuid_compare()=no-sync by rule=reconnected
  1088. Mar 31 01:21:32 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 1146113168 (primary_nodes=8)
  1089. Mar 31 01:21:32 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  1090. Mar 31 01:21:32 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( Off -> Established ) [remote]
  1091. Mar 31 01:21:32 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: cleared bm UUID and bitmap 752996FB4D06CB50:0000000000000000:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC
  1092. Mar 31 01:21:34 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4164491876: 1->3 role( Secondary ) conn( Connected )
  1093. Mar 31 01:21:34 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 4164491876 (primary_nodes=8)
  1094. Mar 31 01:21:34 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1675388973: 0->1 role( Secondary ) conn( Connected )
  1095. Mar 31 01:21:34 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 1675388973 (primary_nodes=8)
  1096. Mar 31 01:21:55 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: PingAck did not arrive in time.
  1097. Mar 31 01:21:55 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )
  1098. Mar 31 01:21:55 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
  1099. Mar 31 01:21:55 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2583071922: 2->all empty
  1100. Mar 31 01:21:55 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Terminating sender thread
  1101. Mar 31 01:21:55 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Starting sender thread (peer-node-id 3)
  1102. Mar 31 01:21:55 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Connection closed
  1103. Mar 31 01:21:55 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( NetworkFailure -> Unconnected ) [disconnected]
  1104. Mar 31 01:21:55 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Restarting receiver thread
  1105. Mar 31 01:21:55 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Unconnected -> Connecting ) [connecting]
  1106. Mar 31 01:21:55 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 592ms
  1107. Mar 31 01:21:55 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 937972097: 1->all empty
  1108. Mar 31 01:21:55 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 937972097 (primary_nodes=0)
  1109. Mar 31 01:21:55 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3407140815: 2->all empty
  1110. Mar 31 01:21:55 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 3407140815: primary_nodes=0, weak_nodes=0
  1111. Mar 31 01:21:55 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 3407140815 (0ms)
  1112. Mar 31 01:22:33 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Handshake to peer 3 successful: Agreed network protocol version 122
  1113. Mar 31 01:22:33 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  1114. Mar 31 01:22:33 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Peer authenticated using 20 bytes HMAC
  1115. Mar 31 01:22:33 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3835743607: 2->3 role( Secondary ) conn( Connected )
  1116. Mar 31 01:22:33 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: self 752996FB4D06CB50:28D31133FEBC993A:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC bits:0 flags:0
  1117. Mar 31 01:22:33 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: peer's exposed UUID: 28983A7814F88E65
  1118. Mar 31 01:22:33 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 3835743607: primary_nodes=8, weak_nodes=0
  1119. Mar 31 01:22:33 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 3835743607 (48ms)
  1120. Mar 31 01:22:33 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connecting -> Connected ) peer( Unknown -> Primary ) [connected]
  1121. Mar 31 01:22:33 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( DUnknown -> Diskless ) repl( Off -> Established ) [connected]
  1122. Mar 31 01:22:33 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1248691193: 1->3 role( Secondary ) conn( Connected )
  1123. Mar 31 01:22:33 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 1248691193 (primary_nodes=8)
  1124. Mar 31 01:22:54 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 449856471: 1->all empty
  1125. Mar 31 01:22:54 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Rejecting concurrent remote state change 2313837381 because of state change 449856471
  1126. Mar 31 01:22:55 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: PingAck did not arrive in time.
  1127. Mar 31 01:22:55 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  1128. Mar 31 01:22:55 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( Established -> Off )
  1129. Mar 31 01:22:55 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Terminating sender thread
  1130. Mar 31 01:22:55 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Starting sender thread (peer-node-id 0)
  1131. Mar 31 01:22:55 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Aborting remote state change 449856471
  1132. Mar 31 01:22:55 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 4094485826: 2->all empty
  1133. Mar 31 01:22:55 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 4094485826: primary_nodes=8, weak_nodes=0
  1134. Mar 31 01:22:55 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 4094485826 (0ms)
  1135. Mar 31 01:22:55 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Connection closed
  1136. Mar 31 01:22:55 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( NetworkFailure -> Unconnected ) [disconnected]
  1137. Mar 31 01:22:55 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Restarting receiver thread
  1138. Mar 31 01:22:55 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Unconnected -> Connecting ) [connecting]
  1139. Mar 31 01:22:55 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1321179860: 1->all empty
  1140. Mar 31 01:22:55 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 1321179860 (primary_nodes=8)
  1141. Mar 31 01:23:21 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4104775491: 0->1 role( Secondary ) conn( Connected )
  1142. Mar 31 01:23:21 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 4104775491 (primary_nodes=8)
  1143. Mar 31 01:23:21 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Handshake to peer 0 successful: Agreed network protocol version 122
  1144. Mar 31 01:23:21 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  1145. Mar 31 01:23:21 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Peer authenticated using 20 bytes HMAC
  1146. Mar 31 01:23:21 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 8244816: 0->2 role( Secondary ) conn( Connected )
  1147. Mar 31 01:23:21 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: drbd_sync_handshake:
  1148. Mar 31 01:23:21 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: self 752996FB4D06CB50:0000000000000000:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC bits:0 flags:100
  1149. Mar 31 01:23:21 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: peer 752996FB4D06CB50:0000000000000000:D8796B7E2A90EF4C:ADB250ED33B11982 bits:0 flags:1120
  1150. Mar 31 01:23:21 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: uuid_compare()=no-sync by rule=reconnected
  1151. Mar 31 01:23:21 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 8244816 (primary_nodes=8)
  1152. Mar 31 01:23:21 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  1153. Mar 31 01:23:21 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( Off -> Established ) [remote]
  1154. Mar 31 01:23:21 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: cleared bm UUID and bitmap 752996FB4D06CB50:0000000000000000:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC
  1155. Mar 31 01:23:31 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: PingAck did not arrive in time.
  1156. Mar 31 01:23:31 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  1157. Mar 31 01:23:31 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( Established -> Off )
  1158. Mar 31 01:23:31 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2259298468: 2->all empty
  1159. Mar 31 01:23:31 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Terminating sender thread
  1160. Mar 31 01:23:31 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Starting sender thread (peer-node-id 1)
  1161. Mar 31 01:23:31 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Aborting local state change 2259298468 to yield to remote state change 2945318694.
  1162. Mar 31 01:23:31 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 320ms
  1163. Mar 31 01:23:31 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2945318694: 1->all empty
  1164. Mar 31 01:23:31 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Duplicate prepare for remote state change 2945318694
  1165. Mar 31 01:23:31 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 2945318694 (primary_nodes=8)
  1166. Mar 31 01:23:31 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Connection closed
  1167. Mar 31 01:23:31 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( NetworkFailure -> Unconnected ) [disconnected]
  1168. Mar 31 01:23:31 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Restarting receiver thread
  1169. Mar 31 01:23:31 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Unconnected -> Connecting ) [connecting]
  1170. Mar 31 01:23:32 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1772856635: 2->all empty
  1171. Mar 31 01:23:32 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: sock was shut down by peer
  1172. Mar 31 01:23:32 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connected -> BrokenPipe ) peer( Primary -> Unknown )
  1173. Mar 31 01:23:32 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
  1174. Mar 31 01:23:32 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 524ms
  1175. Mar 31 01:23:32 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Terminating sender thread
  1176. Mar 31 01:23:32 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Starting sender thread (peer-node-id 3)
  1177. Mar 31 01:23:32 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Connection closed
  1178. Mar 31 01:23:32 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( BrokenPipe -> Unconnected ) [disconnected]
  1179. Mar 31 01:23:32 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Restarting receiver thread
  1180. Mar 31 01:23:32 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Unconnected -> Connecting ) [connecting]
  1181. Mar 31 01:23:32 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: PingAck did not arrive in time.
  1182. Mar 31 01:23:32 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  1183. Mar 31 01:23:32 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( Established -> Off )
  1184. Mar 31 01:23:32 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Terminating sender thread
  1185. Mar 31 01:23:32 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Starting sender thread (peer-node-id 0)
  1186. Mar 31 01:23:32 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Connection closed
  1187. Mar 31 01:23:32 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( NetworkFailure -> Unconnected ) [disconnected]
  1188. Mar 31 01:23:32 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Restarting receiver thread
  1189. Mar 31 01:23:32 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Unconnected -> Connecting ) [connecting]
  1190. Mar 31 01:23:32 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3666307175: 2->all empty
  1191. Mar 31 01:23:32 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 3666307175 (0ms)
  1192. Mar 31 01:24:17 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Handshake to peer 3 successful: Agreed network protocol version 122
  1193. Mar 31 01:24:17 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  1194. Mar 31 01:24:17 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Peer authenticated using 20 bytes HMAC
  1195. Mar 31 01:24:17 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 664435784: 2->3 role( Secondary ) conn( Connected )
  1196. Mar 31 01:24:17 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: self 752996FB4D06CB50:28D31133FEBC993A:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC bits:0 flags:0
  1197. Mar 31 01:24:17 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: peer's exposed UUID: 28983A7814F88E65
  1198. Mar 31 01:24:17 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 664435784: primary_nodes=8, weak_nodes=0
  1199. Mar 31 01:24:17 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 664435784 (40ms)
  1200. Mar 31 01:24:17 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connecting -> Connected ) peer( Unknown -> Primary ) [connected]
  1201. Mar 31 01:24:17 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( DUnknown -> Diskless ) repl( Off -> Established ) [connected]
  1202. Mar 31 01:24:18 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Handshake to peer 0 successful: Agreed network protocol version 122
  1203. Mar 31 01:24:18 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  1204. Mar 31 01:24:18 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Peer authenticated using 20 bytes HMAC
  1205. Mar 31 01:24:18 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 426807218: 0->2 role( Secondary ) conn( Connected )
  1206. Mar 31 01:24:18 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: drbd_sync_handshake:
  1207. Mar 31 01:24:18 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: self 752996FB4D06CB50:0000000000000000:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC bits:0 flags:100
  1208. Mar 31 01:24:18 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: peer 752996FB4D06CB50:0000000000000000:D8796B7E2A90EF4C:ADB250ED33B11982 bits:0 flags:1120
  1209. Mar 31 01:24:18 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: uuid_compare()=no-sync by rule=reconnected
  1210. Mar 31 01:24:18 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 426807218 (primary_nodes=8)
  1211. Mar 31 01:24:18 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  1212. Mar 31 01:24:18 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( Off -> Established ) [remote]
  1213. Mar 31 01:24:18 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: cleared bm UUID and bitmap 752996FB4D06CB50:0000000000000000:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC
  1214. Mar 31 01:24:20 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Handshake to peer 1 successful: Agreed network protocol version 122
  1215. Mar 31 01:24:20 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  1216. Mar 31 01:24:20 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Peer authenticated using 20 bytes HMAC
  1217. Mar 31 01:24:20 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2001482756: 1->2 role( Secondary ) conn( Connected )
  1218. Mar 31 01:24:20 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: drbd_sync_handshake:
  1219. Mar 31 01:24:20 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: self 752996FB4D06CB50:0000000000000000:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC bits:0 flags:100
  1220. Mar 31 01:24:20 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: peer 752996FB4D06CB50:0000000000000000:ADB250ED33B11982:D8796B7E2A90EF4C bits:0 flags:1100
  1221. Mar 31 01:24:20 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: uuid_compare()=no-sync by rule=reconnected
  1222. Mar 31 01:24:20 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 2001482756 (primary_nodes=8)
  1223. Mar 31 01:24:20 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  1224. Mar 31 01:24:20 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( Off -> Established ) [remote]
  1225. Mar 31 01:24:20 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: cleared bm UUID and bitmap 752996FB4D06CB50:0000000000000000:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC
  1226. Mar 31 01:24:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1784832376: 0->all empty
  1227. Mar 31 01:24:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Rejecting concurrent remote state change 4088068711 because of state change 1784832376
  1228. Mar 31 01:24:31 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: PingAck did not arrive in time.
  1229. Mar 31 01:24:31 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  1230. Mar 31 01:24:31 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( Established -> Off )
  1231. Mar 31 01:24:31 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Terminating sender thread
  1232. Mar 31 01:24:31 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Starting sender thread (peer-node-id 1)
  1233. Mar 31 01:24:31 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Aborting remote state change 1784832376
  1234. Mar 31 01:24:31 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 4218099027: 2->all empty
  1235. Mar 31 01:24:31 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Rejecting concurrent remote state change 4144099619 because of state change 4218099027
  1236. Mar 31 01:24:31 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 4218099027: primary_nodes=8, weak_nodes=0
  1237. Mar 31 01:24:31 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 4218099027 (0ms)
  1238. Mar 31 01:24:31 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Connection closed
  1239. Mar 31 01:24:31 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( NetworkFailure -> Unconnected ) [disconnected]
  1240. Mar 31 01:24:31 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Restarting receiver thread
  1241. Mar 31 01:24:31 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Unconnected -> Connecting ) [connecting]
  1242. Mar 31 01:24:31 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1696622909: 0->all empty
  1243. Mar 31 01:24:31 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 1696622909 (primary_nodes=8)
  1244. Mar 31 01:24:31 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3061440329: 3->all empty
  1245. Mar 31 01:24:31 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 3061440329 (primary_nodes=8)
  1246. Mar 31 01:25:22 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Handshake to peer 1 successful: Agreed network protocol version 122
  1247. Mar 31 01:25:22 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  1248. Mar 31 01:25:22 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Peer authenticated using 20 bytes HMAC
  1249. Mar 31 01:25:22 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4181019255: 1->2 role( Secondary ) conn( Connected )
  1250. Mar 31 01:25:22 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: drbd_sync_handshake:
  1251. Mar 31 01:25:22 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: self 752996FB4D06CB50:0000000000000000:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC bits:0 flags:100
  1252. Mar 31 01:25:22 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: peer 752996FB4D06CB50:0000000000000000:ADB250ED33B11982:D8796B7E2A90EF4C bits:0 flags:1120
  1253. Mar 31 01:25:22 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: uuid_compare()=no-sync by rule=reconnected
  1254. Mar 31 01:25:22 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 4181019255 (primary_nodes=8)
  1255. Mar 31 01:25:22 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  1256. Mar 31 01:25:22 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( Off -> Established ) [remote]
  1257. Mar 31 01:25:22 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: cleared bm UUID and bitmap 752996FB4D06CB50:0000000000000000:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC
  1258. Mar 31 01:25:22 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 283636919: 1->3 role( Secondary ) conn( Connected )
  1259. Mar 31 01:25:22 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 283636919 (primary_nodes=8)
  1260. Mar 31 01:25:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1804534236: 0->1 role( Secondary ) conn( Connected )
  1261. Mar 31 01:25:27 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 1804534236 (primary_nodes=8)
  1262. Mar 31 01:45:33 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3130963472: 3->1 conn( Disconnecting ) pdsk( Outdated )
  1263. Mar 31 01:45:33 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 3130963472 (primary_nodes=8)
  1264. Mar 31 01:45:33 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3905303763: 3->2 conn( Disconnecting ) pdsk( Outdated )
  1265. Mar 31 01:45:33 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 3905303763 (primary_nodes=8)
  1266. Mar 31 01:45:33 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connected -> TearDown ) peer( Primary -> Unknown ) [remote]
  1267. Mar 31 01:45:33 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( Diskless -> DUnknown ) repl( Established -> Off ) [remote]
  1268. Mar 31 01:45:33 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Terminating sender thread
  1269. Mar 31 01:45:33 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Starting sender thread (peer-node-id 3)
  1270. Mar 31 01:45:34 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Connection closed
  1271. Mar 31 01:45:34 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( TearDown -> Unconnected ) [disconnected]
  1272. Mar 31 01:45:34 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Restarting receiver thread
  1273. Mar 31 01:45:34 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Unconnected -> Connecting ) [connecting]
  1274. Mar 31 01:45:40 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3269594929: 1->3 role( Secondary ) conn( Connected )
  1275. Mar 31 01:45:40 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Handshake to peer 3 successful: Agreed network protocol version 122
  1276. Mar 31 01:45:40 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  1277. Mar 31 01:45:40 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Peer authenticated using 20 bytes HMAC
  1278. Mar 31 01:45:40 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 3269594929 (primary_nodes=8)
  1279. Mar 31 01:45:40 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3578569625: 2->3 role( Secondary ) conn( Connected )
  1280. Mar 31 01:45:40 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: self 752996FB4D06CB50:28D31133FEBC993A:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC bits:0 flags:0
  1281. Mar 31 01:45:40 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: peer's exposed UUID: 28983A7814F88E65
  1282. Mar 31 01:45:40 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 3578569625: primary_nodes=8, weak_nodes=0
  1283. Mar 31 01:45:40 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 3578569625 (40ms)
  1284. Mar 31 01:45:40 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connecting -> Connected ) peer( Unknown -> Primary ) [connected]
  1285. Mar 31 01:45:40 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( DUnknown -> Diskless ) repl( Off -> Established ) [connected]
  1286. Mar 31 01:45:40 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 775278204: 0->3 role( Secondary ) conn( Connected )
  1287. Mar 31 01:45:40 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 775278204 (primary_nodes=8)
  1288. Mar 31 02:29:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: PingAck did not arrive in time.
  1289. Mar 31 02:29:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )
  1290. Mar 31 02:29:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
  1291. Mar 31 02:29:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 4051346483: 2->all empty
  1292. Mar 31 02:29:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Terminating sender thread
  1293. Mar 31 02:29:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Starting sender thread (peer-node-id 3)
  1294. Mar 31 02:29:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Connection closed
  1295. Mar 31 02:29:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( NetworkFailure -> Unconnected ) [disconnected]
  1296. Mar 31 02:29:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Restarting receiver thread
  1297. Mar 31 02:29:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Unconnected -> Connecting ) [connecting]
  1298. Mar 31 02:29:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 376ms
  1299. Mar 31 02:29:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3984025428: 1->all empty
  1300. Mar 31 02:29:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Aborting remote state change 3984025428
  1301. Mar 31 02:29:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3048988671: 1->all empty
  1302. Mar 31 02:29:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 3048988671 (primary_nodes=0)
  1303. Mar 31 02:29:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3703767319: 0->all empty
  1304. Mar 31 02:29:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 3703767319 (primary_nodes=0)
  1305. Mar 31 02:29:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 4067614425: 2->all empty
  1306. Mar 31 02:29:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 4067614425: primary_nodes=0, weak_nodes=0
  1307. Mar 31 02:29:30 storage-load-test-3 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 4067614425 (4ms)
  1308.  
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!