API tools faq
paste
Advertisement
AleksZi

dmesg_storage-load-test-2.log

AleksZi
Mar 30th, 2025
15
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 197.22 KB | None | 0 0
raw download clone embed print report
  1. Mar 31 01:04:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Starting worker thread (node-id 0)
  2. Mar 31 01:04:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: meta-data IO uses: blk-bio
  3. Mar 31 01:04:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Diskless -> Attaching ) [attach]
  4. Mar 31 01:04:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: Maximum number of peer devices = 7
  5. Mar 31 01:04:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Method to ensure write ordering: flush
  6. Mar 31 01:04:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: drbd_bm_resize called with capacity == 2104816
  7. Mar 31 01:04:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: resync bitmap: bits=263102 words=28777 pages=57
  8. Mar 31 01:04:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: size = 1028 MB (1052408 KB)
  9. Mar 31 01:04:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Attaching -> Inconsistent ) [attach]
  10. Mar 31 01:04:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: attached to current UUID: 0000000000000004
  11. Mar 31 01:04:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2513066767: 0->all role( Primary ) disk( UpToDate )
  12. Mar 31 01:04:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 2513066767 (0ms)
  13. Mar 31 01:04:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: role( Secondary -> Primary ) [primary]
  14. Mar 31 01:04:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Inconsistent -> UpToDate ) [primary]
  15. Mar 31 01:04:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: persisting effective size = 1028 MB (1052408 KB)
  16. Mar 31 01:04:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Forced to consider local data as UpToDate!
  17. Mar 31 01:04:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: new current UUID: 52F5EAB2DCF34653 weak: FFFFFFFFFFFFFFFE
  18. Mar 31 01:05:00 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 658463043: 0->all role( Secondary )
  19. Mar 31 01:05:00 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 658463043 (0ms)
  20. Mar 31 01:05:00 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: role( Primary -> Secondary ) [secondary]
  21. Mar 31 01:05:00 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3200819717: 0->all role( Primary )
  22. Mar 31 01:05:00 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 3200819717 (0ms)
  23. Mar 31 01:05:00 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: role( Secondary -> Primary ) [primary]
  24. Mar 31 01:05:00 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2036742683: 0->all role( Secondary )
  25. Mar 31 01:05:00 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 2036742683 (0ms)
  26. Mar 31 01:05:00 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: role( Primary -> Secondary ) [secondary]
  27. Mar 31 01:05:02 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( no -> quorum ) [resource-options]
  28. Mar 31 01:05:02 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( yes -> no ) [resource-options]
  29. Mar 31 01:05:02 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Starting sender thread (peer-node-id 1)
  30. Mar 31 01:05:02 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Starting sender thread (peer-node-id 2)
  31. Mar 31 01:05:02 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( StandAlone -> Unconnected ) [connect]
  32. Mar 31 01:05:02 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( StandAlone -> Unconnected ) [connect]
  33. Mar 31 01:05:02 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Starting receiver thread (peer-node-id 1)
  34. Mar 31 01:05:02 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Unconnected -> Connecting ) [connecting]
  35. Mar 31 01:05:02 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Starting receiver thread (peer-node-id 2)
  36. Mar 31 01:05:02 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Unconnected -> Connecting ) [connecting]
  37. Mar 31 01:05:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Handshake to peer 2 successful: Agreed network protocol version 122
  38. Mar 31 01:05:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  39. Mar 31 01:05:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Peer authenticated using 20 bytes HMAC
  40. Mar 31 01:05:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3072324464: 0->2 role( Secondary ) conn( Connected )
  41. Mar 31 01:05:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: drbd_sync_handshake:
  42. Mar 31 01:05:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: self 52F5EAB2DCF34652:28D31133FEBC993B:0000000000000000:0000000000000000 bits:263102 flags:20
  43. Mar 31 01:05:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: peer 0000000000000004:0000000000000000:0000000000000000:0000000000000000 bits:0 flags:24
  44. Mar 31 01:05:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: uuid_compare()=source-set-bitmap by rule=just-created-peer
  45. Mar 31 01:05:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: Setting and writing one bitmap slot, after drbd_sync_handshake
  46. Mar 31 01:05:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Handshake to peer 1 successful: Agreed network protocol version 122
  47. Mar 31 01:05:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  48. Mar 31 01:05:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Peer authenticated using 20 bytes HMAC
  49. Mar 31 01:05:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 3072324464: primary_nodes=0, weak_nodes=0
  50. Mar 31 01:05:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 3072324464 (32ms)
  51. Mar 31 01:05:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [connected]
  52. Mar 31 01:05:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( DUnknown -> Inconsistent ) repl( Off -> WFBitMapS ) [connected]
  53. Mar 31 01:05:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 21(1), total 21; compression: 100.0%
  54. Mar 31 01:05:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 21(1), total 21; compression: 100.0%
  55. Mar 31 01:05:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( WFBitMapS -> SyncSource ) [receive-bitmap]
  56. Mar 31 01:05:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: Began resync as SyncSource (will sync 1052408 KB [263102 bits set]).
  57. Mar 31 01:05:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2377433559: 0->1 role( Secondary ) conn( Connected )
  58. Mar 31 01:05:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: drbd_sync_handshake:
  59. Mar 31 01:05:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: self 52F5EAB2DCF34652:28D31133FEBC993B:0000000000000000:0000000000000000 bits:263102 flags:20
  60. Mar 31 01:05:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: peer 0000000000000004:0000000000000000:0000000000000000:0000000000000000 bits:0 flags:24
  61. Mar 31 01:05:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: uuid_compare()=source-set-bitmap by rule=just-created-peer
  62. Mar 31 01:05:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: Setting and writing one bitmap slot, after drbd_sync_handshake
  63. Mar 31 01:05:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 2377433559: primary_nodes=0, weak_nodes=0
  64. Mar 31 01:05:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 2377433559 (28ms)
  65. Mar 31 01:05:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [connected]
  66. Mar 31 01:05:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( DUnknown -> Inconsistent ) repl( Off -> WFBitMapS ) [connected]
  67. Mar 31 01:05:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 21(1), total 21; compression: 100.0%
  68. Mar 31 01:05:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 21(1), total 21; compression: 100.0%
  69. Mar 31 01:05:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( WFBitMapS -> SyncSource ) [receive-bitmap]
  70. Mar 31 01:05:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: Began resync as SyncSource (will sync 1052408 KB [263102 bits set]).
  71. Mar 31 01:05:14 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: updated UUIDs 52F5EAB2DCF34652:0000000000000000:0000000000000000:0000000000000000
  72. Mar 31 01:05:14 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: Resync done (total 10 sec; paused 0 sec; 105240 K/sec)
  73. Mar 31 01:05:14 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( quorum -> no ) [resync-finished]
  74. Mar 31 01:05:14 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( no -> yes ) [resync-finished]
  75. Mar 31 01:05:14 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( Inconsistent -> UpToDate ) repl( SyncSource -> Established ) [resync-finished]
  76. Mar 31 01:05:14 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: updated UUIDs 52F5EAB2DCF34652:0000000000000000:0000000000000000:0000000000000000
  77. Mar 31 01:05:14 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: Resync done (total 11 sec; paused 0 sec; 95672 K/sec)
  78. Mar 31 01:05:14 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( Inconsistent -> UpToDate ) repl( SyncSource -> Established ) [resync-finished]
  79. Mar 31 01:05:15 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3028870165: 0->all role( Primary )
  80. Mar 31 01:05:15 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 3028870165: primary_nodes=1, weak_nodes=FFFFFFFFFFFFFFF8
  81. Mar 31 01:05:15 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 3028870165 (0ms)
  82. Mar 31 01:05:15 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: role( Secondary -> Primary ) [mount:111298 auto-promote]
  83. Mar 31 01:07:54 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2893496555: 0->all role( Secondary )
  84. Mar 31 01:07:54 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 2893496555: primary_nodes=0, weak_nodes=0
  85. Mar 31 01:07:54 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 2893496555 (0ms)
  86. Mar 31 01:07:54 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: role( Primary -> Secondary ) [umount:126262 auto-demote]
  87. Mar 31 01:07:58 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Starting sender thread (peer-node-id 3)
  88. Mar 31 01:07:58 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( StandAlone -> Unconnected ) [connect]
  89. Mar 31 01:07:58 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Starting receiver thread (peer-node-id 3)
  90. Mar 31 01:07:58 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Unconnected -> Connecting ) [connecting]
  91. Mar 31 01:07:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4112650981: 2->3 role( Secondary ) conn( Connected )
  92. Mar 31 01:07:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 4112650981 (primary_nodes=0)
  93. Mar 31 01:07:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1056996341: 1->3 role( Secondary ) conn( Connected )
  94. Mar 31 01:07:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Handshake to peer 3 successful: Agreed network protocol version 122
  95. Mar 31 01:07:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  96. Mar 31 01:07:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Peer authenticated using 20 bytes HMAC
  97. Mar 31 01:07:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 1056996341 (primary_nodes=0)
  98. Mar 31 01:07:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1852736253: 0->3 role( Secondary ) conn( Connected )
  99. Mar 31 01:07:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: self 52F5EAB2DCF34652:28D31133FEBC993B:0000000000000000:0000000000000000 bits:0 flags:0
  100. Mar 31 01:07:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: peer's exposed UUID: 52F5EAB2DCF34652
  101. Mar 31 01:07:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 1852736253: primary_nodes=0, weak_nodes=0
  102. Mar 31 01:07:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 1852736253 (28ms)
  103. Mar 31 01:07:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [connected]
  104. Mar 31 01:07:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( DUnknown -> Diskless ) repl( Off -> Established ) [connected]
  105. Mar 31 01:08:04 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 229711114: 3->all role( Primary )
  106. Mar 31 01:08:04 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 229711114 (primary_nodes=8)
  107. Mar 31 01:08:04 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: peer( Secondary -> Primary ) [remote]
  108. Mar 31 01:08:04 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: received new current UUID: 3B36CC1B2C2CCBAD weak_nodes=FFFFFFFFFFFFFFF8
  109. Mar 31 01:08:40 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: PingAck did not arrive in time.
  110. Mar 31 01:08:40 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  111. Mar 31 01:08:40 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
  112. Mar 31 01:08:40 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3427757018: 0->all empty
  113. Mar 31 01:08:40 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Terminating sender thread
  114. Mar 31 01:08:40 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Starting sender thread (peer-node-id 2)
  115. Mar 31 01:08:40 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Connection closed
  116. Mar 31 01:08:40 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( NetworkFailure -> Unconnected ) [disconnected]
  117. Mar 31 01:08:40 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Restarting receiver thread
  118. Mar 31 01:08:40 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Unconnected -> Connecting ) [connecting]
  119. Mar 31 01:08:41 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 48ms
  120. Mar 31 01:08:41 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 780112742: 1->all empty
  121. Mar 31 01:08:41 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Aborting remote state change 780112742
  122. Mar 31 01:08:41 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 457764969: 0->all empty
  123. Mar 31 01:08:41 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Rejecting concurrent remote state change 1468276786 because of state change 457764969
  124. Mar 31 01:08:41 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 1184ms
  125. Mar 31 01:08:41 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 999804101: 3->all empty
  126. Mar 31 01:08:41 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 999804101 (primary_nodes=8)
  127. Mar 31 01:08:41 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: received new current UUID: ADB250ED33B11983 weak_nodes=FFFFFFFFFFFFFFFC
  128. Mar 31 01:08:41 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1825593975: 1->all empty
  129. Mar 31 01:08:41 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 1825593975 (primary_nodes=8)
  130. Mar 31 01:08:42 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3548852179: 0->all empty
  131. Mar 31 01:08:42 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 3548852179: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF4
  132. Mar 31 01:08:42 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 3548852179 (0ms)
  133. Mar 31 01:09:24 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Handshake to peer 2 successful: Agreed network protocol version 122
  134. Mar 31 01:09:24 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  135. Mar 31 01:09:24 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Peer authenticated using 20 bytes HMAC
  136. Mar 31 01:09:24 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 841577807: 0->2 role( Secondary ) conn( Connected )
  137. Mar 31 01:09:24 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: drbd_sync_handshake:
  138. Mar 31 01:09:24 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: self ADB250ED33B11982:3B36CC1B2C2CCBAC:52F5EAB2DCF34652:0000000000000000 bits:343 flags:100
  139. Mar 31 01:09:24 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: peer 3B36CC1B2C2CCBAC:0000000000000000:52F5EAB2DCF34652:28D31133FEBC993A bits:0 flags:1120
  140. Mar 31 01:09:24 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: uuid_compare()=source-use-bitmap by rule=bitmap-self
  141. Mar 31 01:09:24 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 841577807: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF4
  142. Mar 31 01:09:24 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 841577807 (40ms)
  143. Mar 31 01:09:24 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [connected]
  144. Mar 31 01:09:24 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( DUnknown -> Consistent ) repl( Off -> WFBitMapS ) [connected]
  145. Mar 31 01:09:24 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( Consistent -> Outdated ) [peer-state]
  146. Mar 31 01:09:24 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 39(1), total 39; compression: 99.9%
  147. Mar 31 01:09:24 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 39(1), total 39; compression: 99.9%
  148. Mar 31 01:09:24 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( Outdated -> Inconsistent ) repl( WFBitMapS -> SyncSource ) [receive-bitmap]
  149. Mar 31 01:09:24 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: Began resync as SyncSource (will sync 1372 KB [343 bits set]).
  150. Mar 31 01:09:24 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: Resync done (total 1 sec; paused 0 sec; 1372 K/sec)
  151. Mar 31 01:09:24 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( SyncSource -> Established ) [resync-finished]
  152. Mar 31 01:09:25 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3079894093: 1->2 role( Secondary ) conn( Connected )
  153. Mar 31 01:09:25 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 3079894093 (primary_nodes=8)
  154. Mar 31 01:09:25 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: resync-susp( no -> peer ) [peer-state]
  155. Mar 31 01:09:25 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: resync-susp( peer -> no ) [peer-state]
  156. Mar 31 01:09:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 97363440: 2->3 role( Secondary ) conn( Connected )
  157. Mar 31 01:09:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 97363440 (primary_nodes=8)
  158. Mar 31 01:09:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: drbd_sync_handshake:
  159. Mar 31 01:09:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: self ADB250ED33B11982:3B36CC1B2C2CCBAC:52F5EAB2DCF34652:0000000000000000 bits:343 flags:100
  160. Mar 31 01:09:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: peer 3B36CC1B2C2CCBAC:0000000000000000:52F5EAB2DCF34652:28D31133FEBC993A bits:0 flags:1384
  161. Mar 31 01:09:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: uuid_compare()=source-use-bitmap by rule=bitmap-self
  162. Mar 31 01:09:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: Becoming WFBitMapS because primary is diskless
  163. Mar 31 01:09:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( Established -> WFBitMapS ) [diskless-primary]
  164. Mar 31 01:09:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 30(1), total 30; compression: 100.0%
  165. Mar 31 01:09:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 30(1), total 30; compression: 100.0%
  166. Mar 31 01:09:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( WFBitMapS -> SyncSource ) [receive-bitmap]
  167. Mar 31 01:09:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: Began resync as SyncSource (will sync 152 KB [38 bits set]).
  168. Mar 31 01:09:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: updated UUIDs ADB250ED33B11982:0000000000000000:3B36CC1B2C2CCBAC:52F5EAB2DCF34652
  169. Mar 31 01:09:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: Resync done (total 1 sec; paused 0 sec; 304 K/sec)
  170. Mar 31 01:09:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: expected n_oos:38 to be equal to rs_failed:0
  171. Mar 31 01:09:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( Inconsistent -> UpToDate ) repl( SyncSource -> Established ) [resync-finished]
  172. Mar 31 01:09:46 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: PingAck did not arrive in time.
  173. Mar 31 01:09:46 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  174. Mar 31 01:09:46 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
  175. Mar 31 01:09:46 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1665238630: 1->all empty
  176. Mar 31 01:09:46 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Terminating sender thread
  177. Mar 31 01:09:46 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Starting sender thread (peer-node-id 1)
  178. Mar 31 01:09:46 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Connection closed
  179. Mar 31 01:09:46 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( NetworkFailure -> Unconnected ) [disconnected]
  180. Mar 31 01:09:46 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Restarting receiver thread
  181. Mar 31 01:09:46 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Unconnected -> Connecting ) [connecting]
  182. Mar 31 01:09:46 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Rejecting concurrent remote state change 1088512057 because of state change 1665238630
  183. Mar 31 01:09:46 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Aborting remote state change 1665238630
  184. Mar 31 01:09:46 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3183198224: 2->all empty
  185. Mar 31 01:09:46 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 3183198224 (primary_nodes=8)
  186. Mar 31 01:09:46 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2895404419: 0->all empty
  187. Mar 31 01:09:46 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 2895404419: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF2
  188. Mar 31 01:09:46 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 2895404419 (0ms)
  189. Mar 31 01:09:46 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: received new current UUID: DD95BBD64DA6BD63 weak_nodes=FFFFFFFFFFFFFFFA
  190. Mar 31 01:09:46 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2955162861: 3->all empty
  191. Mar 31 01:09:46 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 2955162861 (primary_nodes=8)
  192. Mar 31 01:10:13 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4229506866: 1->2 role( Secondary ) conn( Connected )
  193. Mar 31 01:10:13 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 4229506866 (primary_nodes=8)
  194. Mar 31 01:10:15 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Handshake to peer 1 successful: Agreed network protocol version 122
  195. Mar 31 01:10:15 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  196. Mar 31 01:10:15 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Peer authenticated using 20 bytes HMAC
  197. Mar 31 01:10:15 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 461079150: 0->1 role( Secondary ) conn( Connected )
  198. Mar 31 01:10:15 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: drbd_sync_handshake:
  199. Mar 31 01:10:15 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: self DD95BBD64DA6BD62:ADB250ED33B11982:3B36CC1B2C2CCBAC:52F5EAB2DCF34652 bits:192 flags:100
  200. Mar 31 01:10:15 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: peer ADB250ED33B11982:ADB250ED33B11982:3B36CC1B2C2CCBAC:52F5EAB2DCF34652 bits:267 flags:1124
  201. Mar 31 01:10:15 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: uuid_compare()=source-use-bitmap by rule=bitmap-self
  202. Mar 31 01:10:15 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 461079150: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF2
  203. Mar 31 01:10:15 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 461079150 (44ms)
  204. Mar 31 01:10:15 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [connected]
  205. Mar 31 01:10:15 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( DUnknown -> Inconsistent ) repl( Off -> WFBitMapS ) [connected]
  206. Mar 31 01:10:15 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 32(1), total 32; compression: 100.0%
  207. Mar 31 01:10:15 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 37(1), total 37; compression: 99.9%
  208. Mar 31 01:10:15 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( WFBitMapS -> SyncSource ) [receive-bitmap]
  209. Mar 31 01:10:15 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: Began resync as SyncSource (will sync 1068 KB [267 bits set]).
  210. Mar 31 01:10:15 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: Resync done (total 1 sec; paused 0 sec; 1068 K/sec)
  211. Mar 31 01:10:15 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( SyncSource -> Established ) [resync-finished]
  212. Mar 31 01:10:16 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4185765531: 1->3 role( Secondary ) conn( Connected )
  213. Mar 31 01:10:16 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 4185765531 (primary_nodes=8)
  214. Mar 31 01:10:16 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: resync-susp( no -> peer ) [peer-state]
  215. Mar 31 01:10:16 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: updated UUIDs DD95BBD64DA6BD62:0000000000000000:ADB250ED33B11982:3B36CC1B2C2CCBAC
  216. Mar 31 01:10:16 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( Inconsistent -> UpToDate ) resync-susp( peer -> no ) [peer-state]
  217. Mar 31 01:10:42 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: PingAck did not arrive in time.
  218. Mar 31 01:10:42 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )
  219. Mar 31 01:10:42 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( UpToDate -> Consistent )
  220. Mar 31 01:10:42 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
  221. Mar 31 01:10:42 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 4191436868: 0->all empty
  222. Mar 31 01:10:42 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Terminating sender thread
  223. Mar 31 01:10:42 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Starting sender thread (peer-node-id 3)
  224. Mar 31 01:10:43 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Connection closed
  225. Mar 31 01:10:43 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( NetworkFailure -> Unconnected ) [disconnected]
  226. Mar 31 01:10:43 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Restarting receiver thread
  227. Mar 31 01:10:43 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Unconnected -> Connecting ) [connecting]
  228. Mar 31 01:10:43 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 468ms
  229. Mar 31 01:10:43 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( no -> quorum ) [peer-state]
  230. Mar 31 01:10:43 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( yes -> no ) [peer-state]
  231. Mar 31 01:10:43 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( UpToDate -> Consistent ) [peer-state]
  232. Mar 31 01:10:43 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: No reconciliation resync even though 'storage-load-test-0' disappeared. (o=0)
  233. Mar 31 01:10:43 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 595626168: 1->all empty
  234. Mar 31 01:10:43 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( UpToDate -> Consistent ) [peer-state]
  235. Mar 31 01:10:43 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( Established -> WFBitMapS ) [receive-peer-dagtag]
  236. Mar 31 01:10:43 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Reconciliation resync because 'storage-load-test-0' disappeared. (o=16)
  237. Mar 31 01:10:43 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 37(1), total 37; compression: 99.9%
  238. Mar 31 01:10:43 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Rejecting concurrent remote state change 1771109539 because of state change 595626168
  239. Mar 31 01:10:43 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 595626168 (primary_nodes=0)
  240. Mar 31 01:10:43 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 871012577: 1->all empty
  241. Mar 31 01:10:43 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( quorum -> no ) [peer-state]
  242. Mar 31 01:10:43 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Consistent -> UpToDate ) quorum( no -> yes ) [peer-state]
  243. Mar 31 01:10:43 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( Consistent -> UpToDate ) [peer-state]
  244. Mar 31 01:10:43 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( Consistent -> Outdated ) [peer-state]
  245. Mar 31 01:10:43 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 871012577 (primary_nodes=0)
  246. Mar 31 01:10:43 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 37(1), total 37; compression: 99.9%
  247. Mar 31 01:10:43 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( Outdated -> Inconsistent ) repl( WFBitMapS -> SyncSource ) [receive-bitmap]
  248. Mar 31 01:10:43 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( SyncSource -> PausedSyncS ) resync-susp( no -> peer ) [peer-state]
  249. Mar 31 01:10:43 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: Resync suspended
  250. Mar 31 01:10:43 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: Began resync as SyncSource (will sync 300 KB [75 bits set]).
  251. Mar 31 01:10:43 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: Resync done (total 1 sec; paused 0 sec; 300 K/sec)
  252. Mar 31 01:10:43 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( Inconsistent -> UpToDate ) repl( PausedSyncS -> Established ) [resync-finished]
  253. Mar 31 01:10:43 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: resync-susp( peer -> no ) [peer-state]
  254. Mar 31 01:10:43 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3701694474: 0->all empty
  255. Mar 31 01:10:43 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 3701694474: primary_nodes=0, weak_nodes=0
  256. Mar 31 01:10:43 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 3701694474 (0ms)
  257. Mar 31 01:10:43 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1033669755: 0->all empty
  258. Mar 31 01:10:43 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 1033669755: primary_nodes=0, weak_nodes=0
  259. Mar 31 01:10:43 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 1033669755 (0ms)
  260. Mar 31 01:10:44 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1413252624: 2->all empty
  261. Mar 31 01:10:44 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 1413252624 (primary_nodes=0)
  262. Mar 31 01:10:44 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2834901056: 2->all empty
  263. Mar 31 01:10:44 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 2834901056 (primary_nodes=0)
  264. Mar 31 01:11:12 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1914803476: 1->3 role( Secondary ) conn( Connected )
  265. Mar 31 01:11:13 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 1914803476 (primary_nodes=8)
  266. Mar 31 01:11:13 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3587873295: 3->all empty
  267. Mar 31 01:11:13 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 3587873295 (primary_nodes=8)
  268. Mar 31 01:11:13 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( UpToDate -> Outdated ) [far-away]
  269. Mar 31 01:11:13 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3330819532: 1->all empty
  270. Mar 31 01:11:13 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( no -> quorum ) [peer-state]
  271. Mar 31 01:11:13 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( yes -> no ) [peer-state]
  272. Mar 31 01:11:13 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( UpToDate -> Outdated ) [peer-state]
  273. Mar 31 01:11:13 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( UpToDate -> Consistent ) [peer-state]
  274. Mar 31 01:11:13 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: Upgrading local disk to Consistent after unstable/weak (and no resync).
  275. Mar 31 01:11:13 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Outdated -> Consistent ) [after-unstable]
  276. Mar 31 01:11:13 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 3330819532 (primary_nodes=0)
  277. Mar 31 01:11:13 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1043972389: 1->all empty
  278. Mar 31 01:11:13 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( Outdated -> Consistent ) [peer-state]
  279. Mar 31 01:11:13 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( quorum -> no ) [peer-state]
  280. Mar 31 01:11:13 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Consistent -> UpToDate ) quorum( no -> yes ) [peer-state]
  281. Mar 31 01:11:13 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( Consistent -> UpToDate ) [peer-state]
  282. Mar 31 01:11:13 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( Consistent -> UpToDate ) [peer-state]
  283. Mar 31 01:11:13 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( UpToDate -> Outdated ) [receive-peer-dagtag]
  284. Mar 31 01:11:13 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( Established -> WFBitMapT ) [receive-peer-dagtag]
  285. Mar 31 01:11:13 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Reconciliation resync because 'storage-load-test-0' disappeared. (o=-40)
  286. Mar 31 01:11:13 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 1043972389 (primary_nodes=0)
  287. Mar 31 01:11:13 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( no -> quorum ) [peer-state]
  288. Mar 31 01:11:13 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( yes -> no ) [peer-state]
  289. Mar 31 01:11:13 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( UpToDate -> Outdated ) [peer-state]
  290. Mar 31 01:11:13 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1142912854: 1->3 role( Secondary ) conn( Connected )
  291. Mar 31 01:11:13 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 1142912854 (primary_nodes=8)
  292. Mar 31 01:11:13 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1970256621: 3->all empty
  293. Mar 31 01:11:13 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 1970256621 (primary_nodes=8)
  294. Mar 31 01:11:13 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1547400645: 2->3 role( Secondary ) conn( Connected )
  295. Mar 31 01:11:13 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 1547400645 (primary_nodes=8)
  296. Mar 31 01:11:16 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Handshake to peer 3 successful: Agreed network protocol version 122
  297. Mar 31 01:11:16 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  298. Mar 31 01:11:16 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Peer authenticated using 20 bytes HMAC
  299. Mar 31 01:11:16 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1713475212: 0->3 role( Secondary ) conn( Connected )
  300. Mar 31 01:11:16 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: self DD95BBD64DA6BD62:28D31133FEBC993B:ADB250ED33B11982:3B36CC1B2C2CCBAC bits:0 flags:0
  301. Mar 31 01:11:16 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: peer's exposed UUID: DD95BBD64DA6BD63
  302. Mar 31 01:11:16 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 1713475212: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF0
  303. Mar 31 01:11:16 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 1713475212 (32ms)
  304. Mar 31 01:11:16 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: Upgrading local disk to D_UP_TO_DATE since current UUID matches.
  305. Mar 31 01:11:16 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connecting -> Connected ) peer( Unknown -> Primary ) [connected]
  306. Mar 31 01:11:16 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( DUnknown -> Diskless ) repl( Off -> Established ) [connected]
  307. Mar 31 01:11:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: PingAck did not arrive in time.
  308. Mar 31 01:11:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )
  309. Mar 31 01:11:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
  310. Mar 31 01:11:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: PingAck did not arrive in time.
  311. Mar 31 01:11:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  312. Mar 31 01:11:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( UpToDate -> DUnknown ) repl( WFBitMapT -> Off )
  313. Mar 31 01:11:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1652681402: 0->all empty
  314. Mar 31 01:11:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Terminating sender thread
  315. Mar 31 01:11:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Terminating sender thread
  316. Mar 31 01:11:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Starting sender thread (peer-node-id 1)
  317. Mar 31 01:11:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Starting sender thread (peer-node-id 3)
  318. Mar 31 01:11:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Connection closed
  319. Mar 31 01:11:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( NetworkFailure -> Unconnected ) [disconnected]
  320. Mar 31 01:11:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Restarting receiver thread
  321. Mar 31 01:11:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Unconnected -> Connecting ) [connecting]
  322. Mar 31 01:11:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: meta connection shut down by peer.
  323. Mar 31 01:11:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  324. Mar 31 01:11:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( Established -> Off )
  325. Mar 31 01:11:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 148ms
  326. Mar 31 01:11:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Terminating sender thread
  327. Mar 31 01:11:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Starting sender thread (peer-node-id 2)
  328. Mar 31 01:11:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Connection closed
  329. Mar 31 01:11:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( NetworkFailure -> Unconnected ) [disconnected]
  330. Mar 31 01:11:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Restarting receiver thread
  331. Mar 31 01:11:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Unconnected -> Connecting ) [connecting]
  332. Mar 31 01:11:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Connection closed
  333. Mar 31 01:11:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( NetworkFailure -> Unconnected ) [disconnected]
  334. Mar 31 01:11:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Restarting receiver thread
  335. Mar 31 01:11:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Unconnected -> Connecting ) [connecting]
  336. Mar 31 01:11:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 4015285437: 0->all empty
  337. Mar 31 01:11:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 4015285437 (0ms)
  338. Mar 31 01:12:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Handshake to peer 2 successful: Agreed network protocol version 122
  339. Mar 31 01:12:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  340. Mar 31 01:12:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Peer authenticated using 20 bytes HMAC
  341. Mar 31 01:12:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2217267134: 0->2 role( Secondary ) conn( Connected )
  342. Mar 31 01:12:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: drbd_sync_handshake:
  343. Mar 31 01:12:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: self DD95BBD64DA6BD62:0000000000000000:ADB250ED33B11982:3B36CC1B2C2CCBAC bits:0 flags:120
  344. Mar 31 01:12:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: peer DD95BBD64DA6BD62:0000000000000000:3B36CC1B2C2CCBAC:ADB250ED33B11982 bits:0 flags:1100
  345. Mar 31 01:12:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: uuid_compare()=no-sync by rule=reconnected
  346. Mar 31 01:12:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 2217267134: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF0
  347. Mar 31 01:12:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 2217267134 (52ms)
  348. Mar 31 01:12:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [connected]
  349. Mar 31 01:12:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( Off -> Established ) [connected]
  350. Mar 31 01:12:03 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: cleared bm UUID and bitmap DD95BBD64DA6BD62:0000000000000000:ADB250ED33B11982:3B36CC1B2C2CCBAC
  351. Mar 31 01:12:08 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Handshake to peer 3 successful: Agreed network protocol version 122
  352. Mar 31 01:12:08 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  353. Mar 31 01:12:08 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Peer authenticated using 20 bytes HMAC
  354. Mar 31 01:12:08 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2130346017: 0->3 role( Secondary ) conn( Connected )
  355. Mar 31 01:12:08 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: self DD95BBD64DA6BD62:28D31133FEBC993B:ADB250ED33B11982:3B36CC1B2C2CCBAC bits:0 flags:0
  356. Mar 31 01:12:08 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: peer's exposed UUID: DD95BBD64DA6BD63
  357. Mar 31 01:12:08 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 2130346017: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF0
  358. Mar 31 01:12:08 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 2130346017 (20ms)
  359. Mar 31 01:12:08 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: Upgrading local disk to D_UP_TO_DATE since current UUID matches.
  360. Mar 31 01:12:08 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connecting -> Connected ) peer( Unknown -> Primary ) [connected]
  361. Mar 31 01:12:08 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Outdated -> UpToDate ) [connected]
  362. Mar 31 01:12:08 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( DUnknown -> Diskless ) repl( Off -> Established ) [connected]
  363. Mar 31 01:12:08 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: received new current UUID: DD95BBD64DA6BD63 weak_nodes=FFFFFFFFFFFFFFFD
  364. Mar 31 01:12:11 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Handshake to peer 1 successful: Agreed network protocol version 122
  365. Mar 31 01:12:11 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  366. Mar 31 01:12:11 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Peer authenticated using 20 bytes HMAC
  367. Mar 31 01:12:11 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1527465761: 0->1 role( Secondary ) conn( Connected )
  368. Mar 31 01:12:11 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: drbd_sync_handshake:
  369. Mar 31 01:12:11 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: self DD95BBD64DA6BD62:DD95BBD64DA6BD63:ADB250ED33B11982:3B36CC1B2C2CCBAC bits:0 flags:100
  370. Mar 31 01:12:11 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: peer DD95BBD64DA6BD62:0000000000000000:3B36CC1B2C2CCBAC:52F5EAB2DCF34652 bits:0 flags:1100
  371. Mar 31 01:12:11 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: uuid_compare()=no-sync by rule=reconnected
  372. Mar 31 01:12:11 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 1527465761: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF0
  373. Mar 31 01:12:11 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 1527465761 (32ms)
  374. Mar 31 01:12:11 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( quorum -> no ) [connected]
  375. Mar 31 01:12:11 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [connected]
  376. Mar 31 01:12:11 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( no -> yes ) [connected]
  377. Mar 31 01:12:11 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( DUnknown -> UpToDate ) repl( Off -> Established ) [connected]
  378. Mar 31 01:12:11 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: cleared bm UUID and bitmap DD95BBD64DA6BD62:0000000000000000:ADB250ED33B11982:3B36CC1B2C2CCBAC
  379. Mar 31 01:12:11 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: bitmap WRITE of 2 pages took 8 ms
  380. Mar 31 01:12:24 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1389696837: 3->all empty
  381. Mar 31 01:12:24 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Aborting remote state change 1389696837
  382. Mar 31 01:12:24 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2149326564: 3->all empty
  383. Mar 31 01:12:24 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Aborting remote state change 2149326564
  384. Mar 31 01:12:25 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3714550546: 1->all empty
  385. Mar 31 01:12:25 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: meta connection shut down by peer.
  386. Mar 31 01:12:25 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  387. Mar 31 01:12:25 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( Established -> Off )
  388. Mar 31 01:12:25 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Aborting remote state change 3714550546
  389. Mar 31 01:12:25 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2301440963: 0->all empty
  390. Mar 31 01:12:25 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Terminating sender thread
  391. Mar 31 01:12:25 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Starting sender thread (peer-node-id 2)
  392. Mar 31 01:12:25 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 2301440963: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF4
  393. Mar 31 01:12:25 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 2301440963 (0ms)
  394. Mar 31 01:12:25 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Connection closed
  395. Mar 31 01:12:25 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( NetworkFailure -> Unconnected ) [disconnected]
  396. Mar 31 01:12:25 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Restarting receiver thread
  397. Mar 31 01:12:25 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Unconnected -> Connecting ) [connecting]
  398. Mar 31 01:12:25 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3306662247: 1->all empty
  399. Mar 31 01:12:25 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 3306662247 (primary_nodes=8)
  400. Mar 31 01:12:25 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4059018192: 3->all empty
  401. Mar 31 01:12:25 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 4059018192 (primary_nodes=8)
  402. Mar 31 01:12:47 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4282827581: 2->3 role( Secondary ) conn( Connected )
  403. Mar 31 01:12:47 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 4282827581 (primary_nodes=8)
  404. Mar 31 01:12:47 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Handshake to peer 2 successful: Agreed network protocol version 122
  405. Mar 31 01:12:47 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  406. Mar 31 01:12:47 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Peer authenticated using 20 bytes HMAC
  407. Mar 31 01:12:47 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3429861694: 0->2 role( Secondary ) conn( Connected )
  408. Mar 31 01:12:47 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: drbd_sync_handshake:
  409. Mar 31 01:12:47 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: self DD95BBD64DA6BD62:DD95BBD64DA6BD62:ADB250ED33B11982:3B36CC1B2C2CCBAC bits:310 flags:100
  410. Mar 31 01:12:47 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: peer DD95BBD64DA6BD62:DD95BBD64DA6BD63:3B36CC1B2C2CCBAC:ADB250ED33B11982 bits:0 flags:1100
  411. Mar 31 01:12:47 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: uuid_compare()=no-sync by rule=reconnected
  412. Mar 31 01:12:47 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 3429861694: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF0
  413. Mar 31 01:12:47 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 3429861694 (40ms)
  414. Mar 31 01:12:47 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [connected]
  415. Mar 31 01:12:47 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( Outdated -> UpToDate ) repl( Off -> Established ) [connected]
  416. Mar 31 01:12:47 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: cleared bm UUID and bitmap DD95BBD64DA6BD62:0000000000000000:ADB250ED33B11982:3B36CC1B2C2CCBAC
  417. Mar 31 01:12:47 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1152771459: 1->2 role( Secondary ) conn( Connected )
  418. Mar 31 01:12:47 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 1152771459 (primary_nodes=8)
  419. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: PingAck did not arrive in time.
  420. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )
  421. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( UpToDate -> Consistent )
  422. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
  423. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d tcp:storage-load-test-0: Not enough data received; missing 832 bytes
  424. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3292241375: 0->all empty
  425. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: error receiving P_DATA, e: -61 l: 106496!
  426. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Terminating sender thread
  427. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Starting sender thread (peer-node-id 3)
  428. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Connection closed
  429. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( NetworkFailure -> Unconnected ) [disconnected]
  430. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Restarting receiver thread
  431. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Unconnected -> Connecting ) [connecting]
  432. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( no -> quorum ) [peer-state]
  433. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( yes -> no ) [peer-state]
  434. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( UpToDate -> Consistent ) [peer-state]
  435. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 492ms
  436. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( UpToDate -> Consistent ) [peer-state]
  437. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Consistent -> Outdated ) [receive-peer-dagtag]
  438. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( Established -> WFBitMapT ) [receive-peer-dagtag]
  439. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Reconciliation resync because 'storage-load-test-0' disappeared. (o=-208)
  440. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: Upgrading local disk to Consistent after unstable/weak (and no resync).
  441. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 30(1), total 30; compression: 100.0%
  442. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 30(1), total 30; compression: 100.0%
  443. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Outdated -> Inconsistent ) [receive-bitmap]
  444. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: resync-susp( no -> connection dependency ) [receive-bitmap]
  445. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: resync-susp( no -> connection dependency ) [receive-bitmap]
  446. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( WFBitMapT -> SyncTarget ) [receive-bitmap]
  447. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: Began resync as SyncTarget (will sync 132 KB [33 bits set]).
  448. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4119964504: 1->all empty
  449. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( Consistent -> Outdated ) [peer-state]
  450. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: Resync done (total 1 sec; paused 0 sec; 132 K/sec)
  451. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: updated UUIDs DD95BBD64DA6BD62:0000000000000000:ADB250ED33B11982:3B36CC1B2C2CCBAC
  452. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Inconsistent -> Consistent ) [resync-finished]
  453. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: resync-susp( connection dependency -> no ) [resync-finished]
  454. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: resync-susp( connection dependency -> no ) [resync-finished]
  455. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( SyncTarget -> Established ) [resync-finished]
  456. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 4119964504 (primary_nodes=0)
  457. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( quorum -> no ) [peer-state]
  458. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Consistent -> UpToDate ) quorum( no -> yes ) [peer-state]
  459. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( Consistent -> UpToDate ) [peer-state]
  460. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4075443310: 2->all empty
  461. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Aborting remote state change 4075443310
  462. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( Outdated -> Inconsistent ) resync-susp( no -> peer ) [peer-state]
  463. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( Inconsistent -> UpToDate ) resync-susp( peer -> no ) [peer-state]
  464. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3079372999: 2->all empty
  465. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 3079372999 (primary_nodes=0)
  466. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2335269304: 2->all empty
  467. Mar 31 01:13:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 2335269304 (primary_nodes=0)
  468. Mar 31 01:13:06 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1444156437: 0->all empty
  469. Mar 31 01:13:06 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 1444156437: primary_nodes=0, weak_nodes=0
  470. Mar 31 01:13:06 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 1444156437 (0ms)
  471. Mar 31 01:13:06 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3261655757: 0->all empty
  472. Mar 31 01:13:06 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 3261655757: primary_nodes=0, weak_nodes=0
  473. Mar 31 01:13:06 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 3261655757 (0ms)
  474. Mar 31 01:13:43 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3125836916: 1->3 role( Secondary ) conn( Connected )
  475. Mar 31 01:13:43 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 3125836916 (primary_nodes=8)
  476. Mar 31 01:13:43 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2402789986: 3->all empty
  477. Mar 31 01:13:43 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 2402789986 (primary_nodes=8)
  478. Mar 31 01:13:43 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( UpToDate -> Outdated ) [far-away]
  479. Mar 31 01:13:43 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( no -> quorum ) [peer-state]
  480. Mar 31 01:13:43 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( yes -> no ) [peer-state]
  481. Mar 31 01:13:43 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( UpToDate -> Outdated ) [peer-state]
  482. Mar 31 01:13:46 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Handshake to peer 3 successful: Agreed network protocol version 122
  483. Mar 31 01:13:46 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  484. Mar 31 01:13:46 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Peer authenticated using 20 bytes HMAC
  485. Mar 31 01:13:46 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1570761256: 0->3 role( Secondary ) conn( Connected )
  486. Mar 31 01:13:46 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: self DD95BBD64DA6BD62:28D31133FEBC993B:ADB250ED33B11982:3B36CC1B2C2CCBAC bits:0 flags:0
  487. Mar 31 01:13:46 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: peer's exposed UUID: DD95BBD64DA6BD63
  488. Mar 31 01:13:46 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Declined by peer storage-load-test-0 (id: 3), see the kernel log there
  489. Mar 31 01:13:46 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Aborting cluster-wide state change 1570761256 (36ms) rv = -10
  490. Mar 31 01:13:47 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3138477124: 0->3 role( Secondary ) conn( Connected )
  491. Mar 31 01:13:47 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: self DD95BBD64DA6BD62:28D31133FEBC993B:ADB250ED33B11982:3B36CC1B2C2CCBAC bits:0 flags:0
  492. Mar 31 01:13:47 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: peer's exposed UUID: DD95BBD64DA6BD63
  493. Mar 31 01:13:47 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Declined by peer storage-load-test-0 (id: 3), see the kernel log there
  494. Mar 31 01:13:47 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Aborting cluster-wide state change 3138477124 (40ms) rv = -10
  495. Mar 31 01:13:48 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3535667735: 0->3 role( Secondary ) conn( Connected )
  496. Mar 31 01:13:48 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: self DD95BBD64DA6BD62:28D31133FEBC993B:ADB250ED33B11982:3B36CC1B2C2CCBAC bits:0 flags:0
  497. Mar 31 01:13:48 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: peer's exposed UUID: DD95BBD64DA6BD63
  498. Mar 31 01:13:48 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Declined by peer storage-load-test-0 (id: 3), see the kernel log there
  499. Mar 31 01:13:48 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Aborting cluster-wide state change 3535667735 (36ms) rv = -10
  500. Mar 31 01:13:48 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2072727487: 2->3 role( Secondary ) conn( Connected )
  501. Mar 31 01:13:48 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Aborting remote state change 2072727487
  502. Mar 31 01:13:49 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3417612: 0->3 role( Secondary ) conn( Connected )
  503. Mar 31 01:13:49 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: self DD95BBD64DA6BD62:28D31133FEBC993B:ADB250ED33B11982:3B36CC1B2C2CCBAC bits:0 flags:0
  504. Mar 31 01:13:49 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: peer's exposed UUID: DD95BBD64DA6BD63
  505. Mar 31 01:13:49 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Declined by peer storage-load-test-0 (id: 3), see the kernel log there
  506. Mar 31 01:13:49 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Aborting cluster-wide state change 3417612 (36ms) rv = -10
  507. Mar 31 01:13:49 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 204289545: 2->3 role( Secondary ) conn( Connected )
  508. Mar 31 01:13:49 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Aborting remote state change 204289545
  509. Mar 31 01:13:50 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2615676757: 0->3 role( Secondary ) conn( Connected )
  510. Mar 31 01:13:50 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: self DD95BBD64DA6BD62:28D31133FEBC993B:ADB250ED33B11982:3B36CC1B2C2CCBAC bits:0 flags:0
  511. Mar 31 01:13:50 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: peer's exposed UUID: DD95BBD64DA6BD63
  512. Mar 31 01:13:50 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Declined by peer storage-load-test-0 (id: 3), see the kernel log there
  513. Mar 31 01:13:50 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Aborting cluster-wide state change 2615676757 (48ms) rv = -10
  514. Mar 31 01:13:50 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2561739627: 2->3 role( Secondary ) conn( Connected )
  515. Mar 31 01:13:50 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Aborting remote state change 2561739627
  516. Mar 31 01:13:51 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2363701962: 0->3 role( Secondary ) conn( Connected )
  517. Mar 31 01:13:51 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: self DD95BBD64DA6BD62:28D31133FEBC993B:ADB250ED33B11982:3B36CC1B2C2CCBAC bits:0 flags:0
  518. Mar 31 01:13:51 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: peer's exposed UUID: DD95BBD64DA6BD63
  519. Mar 31 01:13:51 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Declined by peer storage-load-test-0 (id: 3), see the kernel log there
  520. Mar 31 01:13:51 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Aborting cluster-wide state change 2363701962 (40ms) rv = -10
  521. Mar 31 01:13:51 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1348648508: 2->3 role( Secondary ) conn( Connected )
  522. Mar 31 01:13:51 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Aborting remote state change 1348648508
  523. Mar 31 01:13:52 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: PingAck did not arrive in time.
  524. Mar 31 01:13:52 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  525. Mar 31 01:13:52 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
  526. Mar 31 01:13:52 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3862016044: 0->all empty
  527. Mar 31 01:13:52 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Terminating sender thread
  528. Mar 31 01:13:52 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Starting sender thread (peer-node-id 1)
  529. Mar 31 01:13:52 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Connection closed
  530. Mar 31 01:13:52 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( NetworkFailure -> Unconnected ) [disconnected]
  531. Mar 31 01:13:52 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Restarting receiver thread
  532. Mar 31 01:13:52 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Unconnected -> Connecting ) [connecting]
  533. Mar 31 01:13:52 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 336ms
  534. Mar 31 01:13:52 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2331013833: 2->all empty
  535. Mar 31 01:13:52 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 2331013833 (primary_nodes=0)
  536. Mar 31 01:13:52 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3115004085: 0->3 role( Secondary ) conn( Connected )
  537. Mar 31 01:13:52 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: self DD95BBD64DA6BD62:28D31133FEBC993B:ADB250ED33B11982:3B36CC1B2C2CCBAC bits:0 flags:0
  538. Mar 31 01:13:52 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: peer's exposed UUID: DD95BBD64DA6BD63
  539. Mar 31 01:13:52 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Declined by peer storage-load-test-0 (id: 3), see the kernel log there
  540. Mar 31 01:13:52 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Aborting cluster-wide state change 3115004085 (32ms) rv = -10
  541. Mar 31 01:13:52 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4263931465: 2->3 role( Secondary ) conn( Connected )
  542. Mar 31 01:13:52 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 4263931465 (primary_nodes=8)
  543. Mar 31 01:13:52 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 4025883806: 0->all empty
  544. Mar 31 01:13:52 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 4025883806: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF3
  545. Mar 31 01:13:52 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 4025883806 (0ms)
  546. Mar 31 01:13:52 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( Outdated -> UpToDate ) [peer-state]
  547. Mar 31 01:13:52 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3928376880: 3->all empty
  548. Mar 31 01:13:52 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Aborting remote state change 3928376880
  549. Mar 31 01:13:53 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3773917208: 3->all empty
  550. Mar 31 01:13:53 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 3773917208 (primary_nodes=8)
  551. Mar 31 01:13:53 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 417657209: 0->3 role( Secondary ) conn( Connected )
  552. Mar 31 01:13:53 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: self DD95BBD64DA6BD62:28D31133FEBC993B:ADB250ED33B11982:3B36CC1B2C2CCBAC bits:0 flags:0
  553. Mar 31 01:13:53 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: peer's exposed UUID: DD95BBD64DA6BD63
  554. Mar 31 01:13:53 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 417657209: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF2
  555. Mar 31 01:13:53 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 417657209 (36ms)
  556. Mar 31 01:13:53 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: Upgrading local disk to D_UP_TO_DATE since current UUID matches.
  557. Mar 31 01:13:53 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( quorum -> no ) [connected]
  558. Mar 31 01:13:53 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connecting -> Connected ) peer( Unknown -> Primary ) [connected]
  559. Mar 31 01:13:53 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Outdated -> UpToDate ) quorum( no -> yes ) [connected]
  560. Mar 31 01:13:53 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( DUnknown -> Diskless ) repl( Off -> Established ) [connected]
  561. Mar 31 01:13:53 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: received new current UUID: DD95BBD64DA6BD63 weak_nodes=FFFFFFFFFFFFFFFB
  562. Mar 31 01:14:35 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3446771404: 1->3 role( Secondary ) conn( Connected )
  563. Mar 31 01:14:35 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 3446771404 (primary_nodes=8)
  564. Mar 31 01:14:36 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Handshake to peer 1 successful: Agreed network protocol version 122
  565. Mar 31 01:14:36 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  566. Mar 31 01:14:36 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Peer authenticated using 20 bytes HMAC
  567. Mar 31 01:14:36 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 864936807: 0->1 role( Secondary ) conn( Connected )
  568. Mar 31 01:14:36 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: drbd_sync_handshake:
  569. Mar 31 01:14:36 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: self DD95BBD64DA6BD62:DD95BBD64DA6BD62:ADB250ED33B11982:3B36CC1B2C2CCBAC bits:342 flags:100
  570. Mar 31 01:14:36 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: peer DD95BBD64DA6BD62:DD95BBD64DA6BD63:3B36CC1B2C2CCBAC:52F5EAB2DCF34652 bits:50 flags:1100
  571. Mar 31 01:14:36 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: uuid_compare()=no-sync by rule=reconnected
  572. Mar 31 01:14:36 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 864936807: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF0
  573. Mar 31 01:14:36 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 864936807 (36ms)
  574. Mar 31 01:14:36 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [connected]
  575. Mar 31 01:14:36 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( DUnknown -> UpToDate ) repl( Off -> Established ) [connected]
  576. Mar 31 01:14:36 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: cleared bm UUID and bitmap DD95BBD64DA6BD62:0000000000000000:ADB250ED33B11982:3B36CC1B2C2CCBAC
  577. Mar 31 01:14:38 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1444235534: 1->2 role( Secondary ) conn( Connected )
  578. Mar 31 01:14:38 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 1444235534 (primary_nodes=8)
  579. Mar 31 01:14:56 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Duplicate prepare for remote state change 3111857493
  580. Mar 31 01:14:56 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: PingAck did not arrive in time.
  581. Mar 31 01:14:56 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  582. Mar 31 01:14:56 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
  583. Mar 31 01:14:56 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Terminating sender thread
  584. Mar 31 01:14:56 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Starting sender thread (peer-node-id 2)
  585. Mar 31 01:14:56 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 3111857493 (primary_nodes=8)
  586. Mar 31 01:14:56 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3111857493: 2->all empty
  587. Mar 31 01:14:56 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 509583704: 0->all empty
  588. Mar 31 01:14:56 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Rejecting concurrent remote state change 3111857493 because of state change 509583704
  589. Mar 31 01:14:57 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Connection closed
  590. Mar 31 01:14:57 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( NetworkFailure -> Unconnected ) [disconnected]
  591. Mar 31 01:14:57 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Restarting receiver thread
  592. Mar 31 01:14:57 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Unconnected -> Connecting ) [connecting]
  593. Mar 31 01:14:57 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: PingAck did not arrive in time.
  594. Mar 31 01:14:57 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )
  595. Mar 31 01:14:57 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( UpToDate -> Consistent )
  596. Mar 31 01:14:57 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
  597. Mar 31 01:14:57 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: PingAck did not arrive in time.
  598. Mar 31 01:14:57 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( no -> quorum )
  599. Mar 31 01:14:57 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  600. Mar 31 01:14:57 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( yes -> no )
  601. Mar 31 01:14:57 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
  602. Mar 31 01:14:57 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 24ms
  603. Mar 31 01:14:57 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Terminating sender thread
  604. Mar 31 01:14:57 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Terminating sender thread
  605. Mar 31 01:14:57 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Starting sender thread (peer-node-id 1)
  606. Mar 31 01:14:57 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Starting sender thread (peer-node-id 3)
  607. Mar 31 01:14:57 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 4117911462: 0->all empty
  608. Mar 31 01:14:57 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 4117911462 (0ms)
  609. Mar 31 01:14:57 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Consistent -> UpToDate ) [lost-peer]
  610. Mar 31 01:14:57 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Connection closed
  611. Mar 31 01:14:57 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( NetworkFailure -> Unconnected ) [disconnected]
  612. Mar 31 01:14:57 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Restarting receiver thread
  613. Mar 31 01:14:57 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Unconnected -> Connecting ) [connecting]
  614. Mar 31 01:14:57 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Connection closed
  615. Mar 31 01:14:57 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( NetworkFailure -> Unconnected ) [disconnected]
  616. Mar 31 01:14:57 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Restarting receiver thread
  617. Mar 31 01:14:57 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Unconnected -> Connecting ) [connecting]
  618. Mar 31 01:15:16 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Handshake to peer 3 successful: Agreed network protocol version 122
  619. Mar 31 01:15:16 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  620. Mar 31 01:15:16 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Peer authenticated using 20 bytes HMAC
  621. Mar 31 01:15:16 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1653285823: 0->3 role( Secondary ) conn( Connected )
  622. Mar 31 01:15:16 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: self DD95BBD64DA6BD62:28D31133FEBC993B:ADB250ED33B11982:3B36CC1B2C2CCBAC bits:0 flags:0
  623. Mar 31 01:15:16 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: peer's exposed UUID: D8796B7E2A90EF4D
  624. Mar 31 01:15:16 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 1653285823: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF0
  625. Mar 31 01:15:16 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 1653285823 (32ms)
  626. Mar 31 01:15:16 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connecting -> Connected ) peer( Unknown -> Primary ) [connected]
  627. Mar 31 01:15:16 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( UpToDate -> Outdated ) [connected]
  628. Mar 31 01:15:16 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( DUnknown -> Diskless ) repl( Off -> Established ) [connected]
  629. Mar 31 01:15:16 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Handshake to peer 1 successful: Agreed network protocol version 122
  630. Mar 31 01:15:16 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  631. Mar 31 01:15:16 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Peer authenticated using 20 bytes HMAC
  632. Mar 31 01:15:16 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3579449599: 0->1 role( Secondary ) conn( Connected )
  633. Mar 31 01:15:17 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: drbd_sync_handshake:
  634. Mar 31 01:15:17 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: self DD95BBD64DA6BD62:0000000000000000:ADB250ED33B11982:3B36CC1B2C2CCBAC bits:0 flags:100
  635. Mar 31 01:15:17 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: peer D8796B7E2A90EF4C:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC:52F5EAB2DCF34652 bits:152 flags:1100
  636. Mar 31 01:15:17 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: uuid_compare()=target-use-bitmap by rule=bitmap-peer
  637. Mar 31 01:15:17 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 3579449599: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF0
  638. Mar 31 01:15:17 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 3579449599 (48ms)
  639. Mar 31 01:15:17 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [connected]
  640. Mar 31 01:15:17 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( DUnknown -> UpToDate ) repl( Off -> WFBitMapT ) [connected]
  641. Mar 31 01:15:17 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: Setting exposed data uuid: D8796B7E2A90EF4C
  642. Mar 31 01:15:17 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 31(1), total 31; compression: 100.0%
  643. Mar 31 01:15:17 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 31(1), total 31; compression: 100.0%
  644. Mar 31 01:15:17 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Outdated -> Inconsistent ) [receive-bitmap]
  645. Mar 31 01:15:17 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: resync-susp( no -> connection dependency ) [receive-bitmap]
  646. Mar 31 01:15:17 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: resync-susp( no -> connection dependency ) [receive-bitmap]
  647. Mar 31 01:15:17 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( WFBitMapT -> SyncTarget ) [receive-bitmap]
  648. Mar 31 01:15:17 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: Began resync as SyncTarget (will sync 608 KB [152 bits set]).
  649. Mar 31 01:15:17 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: Resync done (total 1 sec; paused 0 sec; 608 K/sec)
  650. Mar 31 01:15:17 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: updated UUIDs D8796B7E2A90EF4C:0000000000000000:ADB250ED33B11982:3B36CC1B2C2CCBAC
  651. Mar 31 01:15:17 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( quorum -> no ) [resync-finished]
  652. Mar 31 01:15:17 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Inconsistent -> UpToDate ) quorum( no -> yes ) [resync-finished]
  653. Mar 31 01:15:17 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: resync-susp( connection dependency -> no ) [resync-finished]
  654. Mar 31 01:15:17 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: resync-susp( connection dependency -> no ) [resync-finished]
  655. Mar 31 01:15:17 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( SyncTarget -> Established ) [resync-finished]
  656. Mar 31 01:15:19 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Handshake to peer 2 successful: Agreed network protocol version 122
  657. Mar 31 01:15:19 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  658. Mar 31 01:15:19 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Peer authenticated using 20 bytes HMAC
  659. Mar 31 01:15:19 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2130924816: 0->2 role( Secondary ) conn( Connected )
  660. Mar 31 01:15:19 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: drbd_sync_handshake:
  661. Mar 31 01:15:19 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: self D8796B7E2A90EF4C:0000000000000000:ADB250ED33B11982:3B36CC1B2C2CCBAC bits:0 flags:100
  662. Mar 31 01:15:19 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: peer D8796B7E2A90EF4C:0000000000000000:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC bits:0 flags:1100
  663. Mar 31 01:15:19 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: uuid_compare()=no-sync by rule=reconnected
  664. Mar 31 01:15:19 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 2130924816: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF0
  665. Mar 31 01:15:19 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 2130924816 (40ms)
  666. Mar 31 01:15:19 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [connected]
  667. Mar 31 01:15:19 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( DUnknown -> UpToDate ) repl( Off -> Established ) [connected]
  668. Mar 31 01:15:19 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: cleared bm UUID and bitmap D8796B7E2A90EF4C:0000000000000000:ADB250ED33B11982:3B36CC1B2C2CCBAC
  669. Mar 31 01:15:28 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Two-phase commit 0 timeout
  670. Mar 31 01:15:29 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: PingAck did not arrive in time.
  671. Mar 31 01:15:29 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  672. Mar 31 01:15:29 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
  673. Mar 31 01:15:29 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 4158750261: 0->all empty
  674. Mar 31 01:15:29 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Terminating sender thread
  675. Mar 31 01:15:29 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Starting sender thread (peer-node-id 2)
  676. Mar 31 01:15:29 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Connection closed
  677. Mar 31 01:15:29 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( NetworkFailure -> Unconnected ) [disconnected]
  678. Mar 31 01:15:29 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Restarting receiver thread
  679. Mar 31 01:15:29 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Unconnected -> Connecting ) [connecting]
  680. Mar 31 01:15:30 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 288ms
  681. Mar 31 01:15:30 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2259277806: 1->all empty
  682. Mar 31 01:15:30 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 2259277806 (primary_nodes=8)
  683. Mar 31 01:15:30 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2173774204: 3->all empty
  684. Mar 31 01:15:30 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Aborting remote state change 2173774204
  685. Mar 31 01:15:30 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: received new current UUID: 752996FB4D06CB51 weak_nodes=FFFFFFFFFFFFFFFC
  686. Mar 31 01:15:30 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 567398661: 0->all empty
  687. Mar 31 01:15:30 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 567398661: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF4
  688. Mar 31 01:15:30 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 567398661 (0ms)
  689. Mar 31 01:15:30 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3559347635: 3->all empty
  690. Mar 31 01:15:30 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 3559347635 (primary_nodes=8)
  691. Mar 31 01:16:01 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1631962529: 2->3 role( Secondary ) conn( Connected )
  692. Mar 31 01:16:02 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Committing remote state change 1631962529 (primary_nodes=8)
  693. Mar 31 01:16:02 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Handshake to peer 2 successful: Agreed network protocol version 122
  694. Mar 31 01:16:02 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  695. Mar 31 01:16:02 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Peer authenticated using 20 bytes HMAC
  696. Mar 31 01:16:02 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3369737057: 0->2 role( Secondary ) conn( Connected )
  697. Mar 31 01:16:02 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: drbd_sync_handshake:
  698. Mar 31 01:16:02 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: self 752996FB4D06CB50:D8796B7E2A90EF4C:ADB250ED33B11982:3B36CC1B2C2CCBAC bits:265 flags:100
  699. Mar 31 01:16:02 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: peer D8796B7E2A90EF4C:0000000000000000:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC bits:7 flags:1100
  700. Mar 31 01:16:02 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: uuid_compare()=source-use-bitmap by rule=bitmap-self
  701. Mar 31 01:16:02 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 3369737057: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF0
  702. Mar 31 01:16:02 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 3369737057 (36ms)
  703. Mar 31 01:16:02 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [connected]
  704. Mar 31 01:16:02 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( DUnknown -> Outdated ) repl( Off -> WFBitMapS ) [connected]
  705. Mar 31 01:16:02 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 40(1), total 40; compression: 99.9%
  706. Mar 31 01:16:02 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 40(1), total 40; compression: 99.9%
  707. Mar 31 01:16:02 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( Outdated -> Inconsistent ) repl( WFBitMapS -> SyncSource ) [receive-bitmap]
  708. Mar 31 01:16:02 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: Began resync as SyncSource (will sync 1088 KB [272 bits set]).
  709. Mar 31 01:16:02 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: updated UUIDs 752996FB4D06CB50:0000000000000000:D8796B7E2A90EF4C:ADB250ED33B11982
  710. Mar 31 01:16:02 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: Resync done (total 1 sec; paused 0 sec; 1088 K/sec)
  711. Mar 31 01:16:02 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( Inconsistent -> UpToDate ) repl( SyncSource -> Established ) [resync-finished]
  712. Mar 31 01:16:04 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2284281406: 1->2 role( Secondary ) conn( Connected )
  713. Mar 31 01:16:04 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 2284281406 (primary_nodes=8)
  714. Mar 31 01:16:26 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: PingAck did not arrive in time.
  715. Mar 31 01:16:26 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )
  716. Mar 31 01:16:26 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( UpToDate -> Consistent )
  717. Mar 31 01:16:26 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
  718. Mar 31 01:16:26 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1388789927: 0->all empty
  719. Mar 31 01:16:26 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Terminating sender thread
  720. Mar 31 01:16:26 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Starting sender thread (peer-node-id 3)
  721. Mar 31 01:16:26 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Connection closed
  722. Mar 31 01:16:26 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( NetworkFailure -> Unconnected ) [disconnected]
  723. Mar 31 01:16:26 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Restarting receiver thread
  724. Mar 31 01:16:26 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Unconnected -> Connecting ) [connecting]
  725. Mar 31 01:16:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 488ms
  726. Mar 31 01:16:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( no -> quorum ) [peer-state]
  727. Mar 31 01:16:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( yes -> no ) [peer-state]
  728. Mar 31 01:16:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( UpToDate -> Consistent ) [peer-state]
  729. Mar 31 01:16:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( UpToDate -> Consistent ) [peer-state]
  730. Mar 31 01:16:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( Established -> WFBitMapS ) [receive-peer-dagtag]
  731. Mar 31 01:16:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Reconciliation resync because 'storage-load-test-0' disappeared. (o=208)
  732. Mar 31 01:16:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 30(1), total 30; compression: 100.0%
  733. Mar 31 01:16:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3626300921: 1->all empty
  734. Mar 31 01:16:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( Consistent -> Outdated ) [peer-state]
  735. Mar 31 01:16:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: No reconciliation resync even though 'storage-load-test-0' disappeared. (o=0)
  736. Mar 31 01:16:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 30(1), total 30; compression: 100.0%
  737. Mar 31 01:16:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( Outdated -> Inconsistent ) repl( WFBitMapS -> SyncSource ) [receive-bitmap]
  738. Mar 31 01:16:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: Began resync as SyncSource (will sync 132 KB [33 bits set]).
  739. Mar 31 01:16:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Rejecting concurrent remote state change 2248437662 because of state change 3626300921
  740. Mar 31 01:16:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 3626300921 (primary_nodes=0)
  741. Mar 31 01:16:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: Resync done (total 1 sec; paused 0 sec; 132 K/sec)
  742. Mar 31 01:16:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( Inconsistent -> Consistent ) repl( SyncSource -> Established ) [resync-finished]
  743. Mar 31 01:16:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 111167260: 2->all empty
  744. Mar 31 01:16:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 111167260 (primary_nodes=0)
  745. Mar 31 01:16:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 194935479: 2->all empty
  746. Mar 31 01:16:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( quorum -> no ) [peer-state]
  747. Mar 31 01:16:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Consistent -> UpToDate ) quorum( no -> yes ) [peer-state]
  748. Mar 31 01:16:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( Consistent -> UpToDate ) [peer-state]
  749. Mar 31 01:16:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( Consistent -> UpToDate ) [peer-state]
  750. Mar 31 01:16:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 194935479 (primary_nodes=0)
  751. Mar 31 01:16:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2038869718: 0->all empty
  752. Mar 31 01:16:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 2038869718: primary_nodes=0, weak_nodes=0
  753. Mar 31 01:16:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 2038869718 (0ms)
  754. Mar 31 01:16:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2081815640: 0->all empty
  755. Mar 31 01:16:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 2081815640: primary_nodes=0, weak_nodes=0
  756. Mar 31 01:16:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 2081815640 (0ms)
  757. Mar 31 01:16:37 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Handshake to peer 3 successful: Agreed network protocol version 122
  758. Mar 31 01:16:37 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  759. Mar 31 01:16:37 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Peer authenticated using 20 bytes HMAC
  760. Mar 31 01:16:37 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3768514093: 0->3 role( Secondary ) conn( Connected )
  761. Mar 31 01:16:37 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: self 752996FB4D06CB50:28D31133FEBC993B:D8796B7E2A90EF4C:ADB250ED33B11982 bits:0 flags:0
  762. Mar 31 01:16:37 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: peer's exposed UUID: 752996FB4D06CB51
  763. Mar 31 01:16:37 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 3768514093: primary_nodes=8, weak_nodes=0
  764. Mar 31 01:16:37 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 3768514093 (32ms)
  765. Mar 31 01:16:37 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connecting -> Connected ) peer( Unknown -> Primary ) [connected]
  766. Mar 31 01:16:37 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( DUnknown -> Diskless ) repl( Off -> Established ) [connected]
  767. Mar 31 01:16:37 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: received new current UUID: 752996FB4D06CB51 weak_nodes=FFFFFFFFFFFFFFFE
  768. Mar 31 01:16:37 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: peer request with dagtag 22288 not found
  769. Mar 31 01:16:37 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )
  770. Mar 31 01:16:37 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( UpToDate -> Consistent )
  771. Mar 31 01:16:37 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
  772. Mar 31 01:16:37 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 88211161: 0->all empty
  773. Mar 31 01:16:37 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 88211161: primary_nodes=0, weak_nodes=0
  774. Mar 31 01:16:37 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 88211161 (12ms)
  775. Mar 31 01:16:37 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( Consistent -> UpToDate ) [lost-peer]
  776. Mar 31 01:16:37 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 718664504: 0->all empty
  777. Mar 31 01:16:37 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: updated UUIDs 752996FB4D06CB50:0000000000000000:D8796B7E2A90EF4C:ADB250ED33B11982
  778. Mar 31 01:16:37 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: My bitmap UUID changed during handshake. 0x752996FB4D06CB50 to 0x0
  779. Mar 31 01:16:37 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 718664504: primary_nodes=0, weak_nodes=0
  780. Mar 31 01:16:37 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 718664504 (16ms)
  781. Mar 31 01:16:37 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Terminating sender thread
  782. Mar 31 01:16:37 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Starting sender thread (peer-node-id 3)
  783. Mar 31 01:16:37 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Connection closed
  784. Mar 31 01:16:37 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( NetworkFailure -> Unconnected ) [disconnected]
  785. Mar 31 01:16:37 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Restarting receiver thread
  786. Mar 31 01:16:37 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Unconnected -> Connecting ) [connecting]
  787. Mar 31 01:16:37 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( UpToDate -> Outdated ) [peer-state]
  788. Mar 31 01:16:37 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( no -> quorum ) [peer-state]
  789. Mar 31 01:16:37 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( yes -> no ) [peer-state]
  790. Mar 31 01:16:37 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( UpToDate -> Outdated ) [peer-state]
  791. Mar 31 01:16:37 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 778947052: 1->3 role( Secondary ) conn( Connected )
  792. Mar 31 01:16:37 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 778947052 (primary_nodes=8)
  793. Mar 31 01:16:38 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2467925668: 2->3 role( Secondary ) conn( Connected )
  794. Mar 31 01:16:38 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 2467925668 (primary_nodes=8)
  795. Mar 31 01:16:38 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 433044289: 3->all empty
  796. Mar 31 01:16:38 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Duplicate prepare for remote state change 433044289
  797. Mar 31 01:16:38 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 433044289 (primary_nodes=8)
  798. Mar 31 01:16:38 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 722555851: 3->all empty
  799. Mar 31 01:16:38 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Ignoring P_TWOPC_COMMIT packet 433044289 current processing state change 722555851
  800. Mar 31 01:16:38 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Duplicate prepare for remote state change 722555851
  801. Mar 31 01:16:38 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 722555851 (primary_nodes=8)
  802. Mar 31 01:16:38 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Handshake to peer 3 successful: Agreed network protocol version 122
  803. Mar 31 01:16:38 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  804. Mar 31 01:16:38 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Peer authenticated using 20 bytes HMAC
  805. Mar 31 01:16:38 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2386366568: 0->3 role( Secondary ) conn( Connected )
  806. Mar 31 01:16:38 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: self 752996FB4D06CB50:28D31133FEBC993B:D8796B7E2A90EF4C:ADB250ED33B11982 bits:0 flags:0
  807. Mar 31 01:16:38 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: peer's exposed UUID: 28983A7814F88E65
  808. Mar 31 01:16:38 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 2386366568: primary_nodes=8, weak_nodes=0
  809. Mar 31 01:16:38 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 2386366568 (36ms)
  810. Mar 31 01:16:38 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connecting -> Connected ) peer( Unknown -> Primary ) [connected]
  811. Mar 31 01:16:38 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: disk( UpToDate -> Outdated ) [connected]
  812. Mar 31 01:16:38 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( DUnknown -> Diskless ) repl( Off -> Established ) [connected]
  813. Mar 31 01:16:38 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1209087830: 3->all empty
  814. Mar 31 01:16:38 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: meta connection shut down by peer.
  815. Mar 31 01:16:38 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )
  816. Mar 31 01:16:38 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
  817. Mar 31 01:16:38 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: sock was shut down by peer
  818. Mar 31 01:16:38 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: meta connection shut down by peer.
  819. Mar 31 01:16:38 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Terminating sender thread
  820. Mar 31 01:16:38 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Starting sender thread (peer-node-id 3)
  821. Mar 31 01:16:38 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Aborting remote state change 1209087830 commit not possible
  822. Mar 31 01:16:38 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3472239397: 0->all empty
  823. Mar 31 01:16:38 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 3472239397: primary_nodes=8, weak_nodes=0
  824. Mar 31 01:16:38 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 3472239397 (0ms)
  825. Mar 31 01:16:38 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Connection closed
  826. Mar 31 01:16:38 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( NetworkFailure -> Unconnected ) [disconnected]
  827. Mar 31 01:16:38 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Restarting receiver thread
  828. Mar 31 01:16:38 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Unconnected -> Connecting ) [connecting]
  829. Mar 31 01:16:38 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2761915777: 3->all empty
  830. Mar 31 01:16:38 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Duplicate prepare for remote state change 2761915777
  831. Mar 31 01:16:38 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 2761915777 (primary_nodes=8)
  832. Mar 31 01:16:38 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1779301481: 3->all empty
  833. Mar 31 01:16:38 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Ignoring P_TWOPC_COMMIT packet 2761915777 current processing state change 1779301481
  834. Mar 31 01:16:38 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Duplicate prepare for remote state change 1779301481
  835. Mar 31 01:16:38 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 1779301481 (primary_nodes=8)
  836. Mar 31 01:16:58 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1400865519: 2->all empty
  837. Mar 31 01:16:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: PingAck did not arrive in time.
  838. Mar 31 01:16:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  839. Mar 31 01:16:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( Established -> Off )
  840. Mar 31 01:16:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Terminating sender thread
  841. Mar 31 01:16:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Starting sender thread (peer-node-id 1)
  842. Mar 31 01:16:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Aborting remote state change 1400865519
  843. Mar 31 01:16:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2703312900: 0->all empty
  844. Mar 31 01:16:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Connection closed
  845. Mar 31 01:16:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( NetworkFailure -> Unconnected ) [disconnected]
  846. Mar 31 01:16:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Restarting receiver thread
  847. Mar 31 01:16:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Unconnected -> Connecting ) [connecting]
  848. Mar 31 01:16:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 352ms
  849. Mar 31 01:16:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2293185259: 2->all empty
  850. Mar 31 01:16:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 2293185259 (primary_nodes=8)
  851. Mar 31 01:16:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4138258921: 3->all empty
  852. Mar 31 01:16:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 4138258921 (primary_nodes=8)
  853. Mar 31 01:16:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 204760639: 0->all empty
  854. Mar 31 01:16:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 204760639: primary_nodes=8, weak_nodes=0
  855. Mar 31 01:16:59 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 204760639 (0ms)
  856. Mar 31 01:17:28 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 332174034: 1->2 role( Secondary ) conn( Connected )
  857. Mar 31 01:17:28 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 332174034 (primary_nodes=8)
  858. Mar 31 01:17:28 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Handshake to peer 1 successful: Agreed network protocol version 122
  859. Mar 31 01:17:28 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  860. Mar 31 01:17:28 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Peer authenticated using 20 bytes HMAC
  861. Mar 31 01:17:28 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3438731863: 0->1 role( Secondary ) conn( Connected )
  862. Mar 31 01:17:28 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: drbd_sync_handshake:
  863. Mar 31 01:17:28 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: self 752996FB4D06CB50:0000000000000000:D8796B7E2A90EF4C:ADB250ED33B11982 bits:1 flags:120
  864. Mar 31 01:17:28 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: peer 752996FB4D06CB50:0000000000000000:ADB250ED33B11982:D8796B7E2A90EF4C bits:0 flags:1120
  865. Mar 31 01:17:28 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: uuid_compare()=no-sync by rule=reconnected
  866. Mar 31 01:17:28 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 3438731863: primary_nodes=8, weak_nodes=0
  867. Mar 31 01:17:28 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 3438731863 (32ms)
  868. Mar 31 01:17:28 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [connected]
  869. Mar 31 01:17:28 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( Off -> Established ) [connected]
  870. Mar 31 01:17:28 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: cleared bm UUID and bitmap 752996FB4D06CB50:0000000000000000:D8796B7E2A90EF4C:ADB250ED33B11982
  871. Mar 31 01:17:28 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 369871165: 1->3 role( Secondary ) conn( Connected )
  872. Mar 31 01:17:28 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 369871165 (primary_nodes=8)
  873. Mar 31 01:17:49 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: PingAck did not arrive in time.
  874. Mar 31 01:17:49 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  875. Mar 31 01:17:49 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( Established -> Off )
  876. Mar 31 01:17:49 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3193316345: 0->all empty
  877. Mar 31 01:17:49 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Terminating sender thread
  878. Mar 31 01:17:49 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Starting sender thread (peer-node-id 1)
  879. Mar 31 01:17:49 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Connection closed
  880. Mar 31 01:17:49 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( NetworkFailure -> Unconnected ) [disconnected]
  881. Mar 31 01:17:49 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Restarting receiver thread
  882. Mar 31 01:17:49 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Unconnected -> Connecting ) [connecting]
  883. Mar 31 01:17:50 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: PingAck did not arrive in time.
  884. Mar 31 01:17:50 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  885. Mar 31 01:17:50 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( Established -> Off )
  886. Mar 31 01:17:50 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 140ms
  887. Mar 31 01:17:50 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Terminating sender thread
  888. Mar 31 01:17:50 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Starting sender thread (peer-node-id 2)
  889. Mar 31 01:17:50 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Connection closed
  890. Mar 31 01:17:50 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( NetworkFailure -> Unconnected ) [disconnected]
  891. Mar 31 01:17:50 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Restarting receiver thread
  892. Mar 31 01:17:50 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Unconnected -> Connecting ) [connecting]
  893. Mar 31 01:17:50 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 907812024: 0->all empty
  894. Mar 31 01:17:50 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 907812024 (0ms)
  895. Mar 31 01:18:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Handshake to peer 2 successful: Agreed network protocol version 122
  896. Mar 31 01:18:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  897. Mar 31 01:18:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Peer authenticated using 20 bytes HMAC
  898. Mar 31 01:18:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1603385043: 0->2 role( Secondary ) conn( Connected )
  899. Mar 31 01:18:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: drbd_sync_handshake:
  900. Mar 31 01:18:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: self 752996FB4D06CB50:0000000000000000:D8796B7E2A90EF4C:ADB250ED33B11982 bits:1 flags:120
  901. Mar 31 01:18:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: peer 752996FB4D06CB50:0000000000000000:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC bits:0 flags:1100
  902. Mar 31 01:18:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: uuid_compare()=no-sync by rule=reconnected
  903. Mar 31 01:18:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 1603385043: primary_nodes=8, weak_nodes=0
  904. Mar 31 01:18:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 1603385043 (48ms)
  905. Mar 31 01:18:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [connected]
  906. Mar 31 01:18:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( Off -> Established ) [connected]
  907. Mar 31 01:18:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: cleared bm UUID and bitmap 752996FB4D06CB50:0000000000000000:D8796B7E2A90EF4C:ADB250ED33B11982
  908. Mar 31 01:18:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: bitmap WRITE of 1 pages took 8 ms
  909. Mar 31 01:18:29 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Handshake to peer 1 successful: Agreed network protocol version 122
  910. Mar 31 01:18:29 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  911. Mar 31 01:18:29 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Peer authenticated using 20 bytes HMAC
  912. Mar 31 01:18:29 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2458447738: 0->1 role( Secondary ) conn( Connected )
  913. Mar 31 01:18:29 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: drbd_sync_handshake:
  914. Mar 31 01:18:29 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: self 752996FB4D06CB50:0000000000000000:D8796B7E2A90EF4C:ADB250ED33B11982 bits:0 flags:120
  915. Mar 31 01:18:29 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: peer 752996FB4D06CB50:0000000000000000:ADB250ED33B11982:D8796B7E2A90EF4C bits:0 flags:1100
  916. Mar 31 01:18:29 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: uuid_compare()=no-sync by rule=reconnected
  917. Mar 31 01:18:29 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 2458447738: primary_nodes=8, weak_nodes=0
  918. Mar 31 01:18:29 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 2458447738 (40ms)
  919. Mar 31 01:18:29 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [connected]
  920. Mar 31 01:18:29 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( Off -> Established ) [connected]
  921. Mar 31 01:18:29 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: cleared bm UUID and bitmap 752996FB4D06CB50:0000000000000000:D8796B7E2A90EF4C:ADB250ED33B11982
  922. Mar 31 01:18:39 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: PingAck did not arrive in time.
  923. Mar 31 01:18:39 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  924. Mar 31 01:18:39 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( Established -> Off )
  925. Mar 31 01:18:39 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2710131752: 0->all empty
  926. Mar 31 01:18:39 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Terminating sender thread
  927. Mar 31 01:18:39 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Starting sender thread (peer-node-id 2)
  928. Mar 31 01:18:39 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 2710131752: primary_nodes=8, weak_nodes=0
  929. Mar 31 01:18:39 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 2710131752 (0ms)
  930. Mar 31 01:18:39 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1561974972: 3->all empty
  931. Mar 31 01:18:39 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 1561974972 (primary_nodes=8)
  932. Mar 31 01:18:39 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Connection closed
  933. Mar 31 01:18:39 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( NetworkFailure -> Unconnected ) [disconnected]
  934. Mar 31 01:18:39 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Restarting receiver thread
  935. Mar 31 01:18:39 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Unconnected -> Connecting ) [connecting]
  936. Mar 31 01:18:40 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 307708718: 1->all empty
  937. Mar 31 01:18:40 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 307708718 (primary_nodes=8)
  938. Mar 31 01:19:04 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Handshake to peer 2 successful: Agreed network protocol version 122
  939. Mar 31 01:19:04 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  940. Mar 31 01:19:04 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Peer authenticated using 20 bytes HMAC
  941. Mar 31 01:19:04 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 4181492047: 0->2 role( Secondary ) conn( Connected )
  942. Mar 31 01:19:04 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: drbd_sync_handshake:
  943. Mar 31 01:19:04 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: self 752996FB4D06CB50:0000000000000000:D8796B7E2A90EF4C:ADB250ED33B11982 bits:0 flags:120
  944. Mar 31 01:19:04 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: peer 752996FB4D06CB50:0000000000000000:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC bits:0 flags:1120
  945. Mar 31 01:19:04 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: uuid_compare()=no-sync by rule=reconnected
  946. Mar 31 01:19:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 4181492047: primary_nodes=8, weak_nodes=0
  947. Mar 31 01:19:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 4181492047 (32ms)
  948. Mar 31 01:19:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [connected]
  949. Mar 31 01:19:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( Off -> Established ) [connected]
  950. Mar 31 01:19:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: cleared bm UUID and bitmap 752996FB4D06CB50:0000000000000000:D8796B7E2A90EF4C:ADB250ED33B11982
  951. Mar 31 01:19:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2159242530: 2->3 role( Secondary ) conn( Connected )
  952. Mar 31 01:19:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 2159242530 (primary_nodes=8)
  953. Mar 31 01:19:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3871126125: 1->2 role( Secondary ) conn( Connected )
  954. Mar 31 01:19:05 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 3871126125 (primary_nodes=8)
  955. Mar 31 01:19:25 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2693247151: 2->all empty
  956. Mar 31 01:19:26 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Aborting remote state change 2693247151
  957. Mar 31 01:19:26 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 595089147: 1->all empty
  958. Mar 31 01:19:26 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 595089147 (primary_nodes=0)
  959. Mar 31 01:19:26 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 73580424: 2->all empty
  960. Mar 31 01:19:26 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 73580424 (primary_nodes=0)
  961. Mar 31 01:20:00 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1789966072: 2->3 role( Secondary ) conn( Connected )
  962. Mar 31 01:20:00 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 1789966072 (primary_nodes=8)
  963. Mar 31 01:20:07 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 178417790: 1->3 role( Secondary ) conn( Connected )
  964. Mar 31 01:20:07 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 178417790 (primary_nodes=8)
  965. Mar 31 01:20:18 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: PingAck did not arrive in time.
  966. Mar 31 01:20:18 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  967. Mar 31 01:20:18 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( Established -> Off )
  968. Mar 31 01:20:18 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1409325880: 0->all empty
  969. Mar 31 01:20:18 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Terminating sender thread
  970. Mar 31 01:20:18 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Starting sender thread (peer-node-id 1)
  971. Mar 31 01:20:18 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 280ms
  972. Mar 31 01:20:18 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Connection closed
  973. Mar 31 01:20:18 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( NetworkFailure -> Unconnected ) [disconnected]
  974. Mar 31 01:20:18 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Restarting receiver thread
  975. Mar 31 01:20:18 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Unconnected -> Connecting ) [connecting]
  976. Mar 31 01:20:18 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2015879537: 0->all empty
  977. Mar 31 01:20:19 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: meta connection shut down by peer.
  978. Mar 31 01:20:19 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  979. Mar 31 01:20:19 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( Established -> Off )
  980. Mar 31 01:20:19 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 528ms
  981. Mar 31 01:20:19 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Terminating sender thread
  982. Mar 31 01:20:19 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Starting sender thread (peer-node-id 2)
  983. Mar 31 01:20:19 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Connection closed
  984. Mar 31 01:20:19 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( NetworkFailure -> Unconnected ) [disconnected]
  985. Mar 31 01:20:19 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Restarting receiver thread
  986. Mar 31 01:20:19 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Unconnected -> Connecting ) [connecting]
  987. Mar 31 01:20:19 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1212726962: 0->all empty
  988. Mar 31 01:20:19 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 1212726962 (0ms)
  989. Mar 31 01:20:56 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Handshake to peer 1 successful: Agreed network protocol version 122
  990. Mar 31 01:20:56 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  991. Mar 31 01:20:56 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Peer authenticated using 20 bytes HMAC
  992. Mar 31 01:20:56 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 26180170: 0->1 role( Secondary ) conn( Connected )
  993. Mar 31 01:20:56 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: drbd_sync_handshake:
  994. Mar 31 01:20:56 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: self 752996FB4D06CB50:0000000000000000:D8796B7E2A90EF4C:ADB250ED33B11982 bits:0 flags:120
  995. Mar 31 01:20:56 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: peer 752996FB4D06CB50:0000000000000000:ADB250ED33B11982:D8796B7E2A90EF4C bits:0 flags:1100
  996. Mar 31 01:20:56 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: uuid_compare()=no-sync by rule=reconnected
  997. Mar 31 01:20:56 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 26180170: primary_nodes=8, weak_nodes=0
  998. Mar 31 01:20:56 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 26180170 (36ms)
  999. Mar 31 01:20:56 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [connected]
  1000. Mar 31 01:20:56 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( Off -> Established ) [connected]
  1001. Mar 31 01:20:56 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: cleared bm UUID and bitmap 752996FB4D06CB50:0000000000000000:D8796B7E2A90EF4C:ADB250ED33B11982
  1002. Mar 31 01:21:00 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Handshake to peer 2 successful: Agreed network protocol version 122
  1003. Mar 31 01:21:00 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  1004. Mar 31 01:21:00 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Peer authenticated using 20 bytes HMAC
  1005. Mar 31 01:21:00 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 113751511: 0->2 role( Secondary ) conn( Connected )
  1006. Mar 31 01:21:00 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: drbd_sync_handshake:
  1007. Mar 31 01:21:00 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: self 752996FB4D06CB50:0000000000000000:D8796B7E2A90EF4C:ADB250ED33B11982 bits:0 flags:120
  1008. Mar 31 01:21:00 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: peer 752996FB4D06CB50:0000000000000000:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC bits:0 flags:1100
  1009. Mar 31 01:21:00 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: uuid_compare()=no-sync by rule=reconnected
  1010. Mar 31 01:21:00 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 113751511: primary_nodes=8, weak_nodes=0
  1011. Mar 31 01:21:00 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 113751511 (36ms)
  1012. Mar 31 01:21:00 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [connected]
  1013. Mar 31 01:21:00 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( Off -> Established ) [connected]
  1014. Mar 31 01:21:00 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: cleared bm UUID and bitmap 752996FB4D06CB50:0000000000000000:D8796B7E2A90EF4C:ADB250ED33B11982
  1015. Mar 31 01:21:07 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3394754177: 2->all empty
  1016. Mar 31 01:21:07 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: PingAck did not arrive in time.
  1017. Mar 31 01:21:07 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  1018. Mar 31 01:21:07 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( Established -> Off )
  1019. Mar 31 01:21:07 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Terminating sender thread
  1020. Mar 31 01:21:07 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Starting sender thread (peer-node-id 1)
  1021. Mar 31 01:21:07 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Aborting remote state change 3394754177
  1022. Mar 31 01:21:07 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 74975511: 0->all empty
  1023. Mar 31 01:21:07 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 228ms
  1024. Mar 31 01:21:07 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2936172748: 3->all empty
  1025. Mar 31 01:21:07 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 2936172748 (primary_nodes=8)
  1026. Mar 31 01:21:07 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Connection closed
  1027. Mar 31 01:21:07 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( NetworkFailure -> Unconnected ) [disconnected]
  1028. Mar 31 01:21:07 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Restarting receiver thread
  1029. Mar 31 01:21:07 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Unconnected -> Connecting ) [connecting]
  1030. Mar 31 01:21:08 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 191871833: 0->all empty
  1031. Mar 31 01:21:08 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 191871833: primary_nodes=8, weak_nodes=0
  1032. Mar 31 01:21:08 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 191871833 (0ms)
  1033. Mar 31 01:21:08 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1983572376: 2->all empty
  1034. Mar 31 01:21:08 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 1983572376 (primary_nodes=8)
  1035. Mar 31 01:21:32 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1146113168: 1->2 role( Secondary ) conn( Connected )
  1036. Mar 31 01:21:32 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 1146113168 (primary_nodes=8)
  1037. Mar 31 01:21:34 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4164491876: 1->3 role( Secondary ) conn( Connected )
  1038. Mar 31 01:21:34 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 4164491876 (primary_nodes=8)
  1039. Mar 31 01:21:34 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Handshake to peer 1 successful: Agreed network protocol version 122
  1040. Mar 31 01:21:34 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  1041. Mar 31 01:21:34 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Peer authenticated using 20 bytes HMAC
  1042. Mar 31 01:21:34 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1675388973: 0->1 role( Secondary ) conn( Connected )
  1043. Mar 31 01:21:34 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: drbd_sync_handshake:
  1044. Mar 31 01:21:34 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: self 752996FB4D06CB50:0000000000000000:D8796B7E2A90EF4C:ADB250ED33B11982 bits:0 flags:120
  1045. Mar 31 01:21:34 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: peer 752996FB4D06CB50:0000000000000000:ADB250ED33B11982:D8796B7E2A90EF4C bits:0 flags:1100
  1046. Mar 31 01:21:34 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: uuid_compare()=no-sync by rule=reconnected
  1047. Mar 31 01:21:34 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 1675388973: primary_nodes=8, weak_nodes=0
  1048. Mar 31 01:21:34 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 1675388973 (28ms)
  1049. Mar 31 01:21:34 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [connected]
  1050. Mar 31 01:21:34 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( Off -> Established ) [connected]
  1051. Mar 31 01:21:34 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: cleared bm UUID and bitmap 752996FB4D06CB50:0000000000000000:D8796B7E2A90EF4C:ADB250ED33B11982
  1052. Mar 31 01:21:55 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2583071922: 2->all empty
  1053. Mar 31 01:21:55 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Aborting remote state change 2583071922
  1054. Mar 31 01:21:55 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 937972097: 1->all empty
  1055. Mar 31 01:21:55 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 937972097 (primary_nodes=0)
  1056. Mar 31 01:21:55 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3407140815: 2->all empty
  1057. Mar 31 01:21:55 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 3407140815 (primary_nodes=0)
  1058. Mar 31 01:22:33 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3835743607: 2->3 role( Secondary ) conn( Connected )
  1059. Mar 31 01:22:33 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 3835743607 (primary_nodes=8)
  1060. Mar 31 01:22:33 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1248691193: 1->3 role( Secondary ) conn( Connected )
  1061. Mar 31 01:22:33 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 1248691193 (primary_nodes=8)
  1062. Mar 31 01:22:54 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: PingAck did not arrive in time.
  1063. Mar 31 01:22:54 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  1064. Mar 31 01:22:54 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( Established -> Off )
  1065. Mar 31 01:22:54 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2313837381: 0->all empty
  1066. Mar 31 01:22:54 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Terminating sender thread
  1067. Mar 31 01:22:54 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Starting sender thread (peer-node-id 1)
  1068. Mar 31 01:22:54 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Connection closed
  1069. Mar 31 01:22:54 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( NetworkFailure -> Unconnected ) [disconnected]
  1070. Mar 31 01:22:54 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Restarting receiver thread
  1071. Mar 31 01:22:54 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Unconnected -> Connecting ) [connecting]
  1072. Mar 31 01:22:55 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: PingAck did not arrive in time.
  1073. Mar 31 01:22:55 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  1074. Mar 31 01:22:55 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( Established -> Off )
  1075. Mar 31 01:22:55 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 244ms
  1076. Mar 31 01:22:55 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Terminating sender thread
  1077. Mar 31 01:22:55 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Starting sender thread (peer-node-id 2)
  1078. Mar 31 01:22:55 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Connection closed
  1079. Mar 31 01:22:55 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( NetworkFailure -> Unconnected ) [disconnected]
  1080. Mar 31 01:22:55 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Restarting receiver thread
  1081. Mar 31 01:22:55 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Unconnected -> Connecting ) [connecting]
  1082. Mar 31 01:22:55 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3182299549: 0->all empty
  1083. Mar 31 01:22:55 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 3182299549 (0ms)
  1084. Mar 31 01:23:21 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Handshake to peer 1 successful: Agreed network protocol version 122
  1085. Mar 31 01:23:21 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  1086. Mar 31 01:23:21 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Peer authenticated using 20 bytes HMAC
  1087. Mar 31 01:23:21 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 4104775491: 0->1 role( Secondary ) conn( Connected )
  1088. Mar 31 01:23:21 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: drbd_sync_handshake:
  1089. Mar 31 01:23:21 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: self 752996FB4D06CB50:0000000000000000:D8796B7E2A90EF4C:ADB250ED33B11982 bits:0 flags:120
  1090. Mar 31 01:23:21 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: peer 752996FB4D06CB50:0000000000000000:ADB250ED33B11982:D8796B7E2A90EF4C bits:0 flags:1100
  1091. Mar 31 01:23:21 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: uuid_compare()=no-sync by rule=reconnected
  1092. Mar 31 01:23:21 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 4104775491: primary_nodes=8, weak_nodes=0
  1093. Mar 31 01:23:21 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 4104775491 (40ms)
  1094. Mar 31 01:23:21 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [connected]
  1095. Mar 31 01:23:21 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( Off -> Established ) [connected]
  1096. Mar 31 01:23:21 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: cleared bm UUID and bitmap 752996FB4D06CB50:0000000000000000:D8796B7E2A90EF4C:ADB250ED33B11982
  1097. Mar 31 01:23:21 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Handshake to peer 2 successful: Agreed network protocol version 122
  1098. Mar 31 01:23:21 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  1099. Mar 31 01:23:21 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Peer authenticated using 20 bytes HMAC
  1100. Mar 31 01:23:21 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 8244816: 0->2 role( Secondary ) conn( Connected )
  1101. Mar 31 01:23:21 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: drbd_sync_handshake:
  1102. Mar 31 01:23:21 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: self 752996FB4D06CB50:0000000000000000:D8796B7E2A90EF4C:ADB250ED33B11982 bits:0 flags:120
  1103. Mar 31 01:23:21 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: peer 752996FB4D06CB50:0000000000000000:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC bits:0 flags:1100
  1104. Mar 31 01:23:21 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: uuid_compare()=no-sync by rule=reconnected
  1105. Mar 31 01:23:21 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 8244816: primary_nodes=8, weak_nodes=0
  1106. Mar 31 01:23:21 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 8244816 (24ms)
  1107. Mar 31 01:23:21 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [connected]
  1108. Mar 31 01:23:21 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( Off -> Established ) [connected]
  1109. Mar 31 01:23:21 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: cleared bm UUID and bitmap 752996FB4D06CB50:0000000000000000:D8796B7E2A90EF4C:ADB250ED33B11982
  1110. Mar 31 01:23:31 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2945318694: 1->all empty
  1111. Mar 31 01:23:31 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 2945318694 (primary_nodes=8)
  1112. Mar 31 01:23:32 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2793566032: 3->all empty
  1113. Mar 31 01:23:32 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: meta connection shut down by peer.
  1114. Mar 31 01:23:32 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  1115. Mar 31 01:23:32 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( Established -> Off )
  1116. Mar 31 01:23:32 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Terminating sender thread
  1117. Mar 31 01:23:32 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Starting sender thread (peer-node-id 2)
  1118. Mar 31 01:23:32 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Aborting remote state change 2793566032
  1119. Mar 31 01:23:32 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 532498931: 0->all empty
  1120. Mar 31 01:23:32 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 532498931: primary_nodes=8, weak_nodes=0
  1121. Mar 31 01:23:32 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 532498931 (0ms)
  1122. Mar 31 01:23:32 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Connection closed
  1123. Mar 31 01:23:32 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( NetworkFailure -> Unconnected ) [disconnected]
  1124. Mar 31 01:23:32 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Restarting receiver thread
  1125. Mar 31 01:23:32 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Unconnected -> Connecting ) [connecting]
  1126. Mar 31 01:23:32 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2699842262: 3->all empty
  1127. Mar 31 01:23:32 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 2699842262 (primary_nodes=8)
  1128. Mar 31 01:24:17 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 664435784: 2->3 role( Secondary ) conn( Connected )
  1129. Mar 31 01:24:17 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 664435784 (primary_nodes=8)
  1130. Mar 31 01:24:18 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Handshake to peer 2 successful: Agreed network protocol version 122
  1131. Mar 31 01:24:18 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  1132. Mar 31 01:24:18 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Peer authenticated using 20 bytes HMAC
  1133. Mar 31 01:24:18 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 426807218: 0->2 role( Secondary ) conn( Connected )
  1134. Mar 31 01:24:18 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: drbd_sync_handshake:
  1135. Mar 31 01:24:18 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: self 752996FB4D06CB50:0000000000000000:D8796B7E2A90EF4C:ADB250ED33B11982 bits:0 flags:120
  1136. Mar 31 01:24:18 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: peer 752996FB4D06CB50:0000000000000000:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC bits:0 flags:1100
  1137. Mar 31 01:24:18 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: uuid_compare()=no-sync by rule=reconnected
  1138. Mar 31 01:24:18 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 426807218: primary_nodes=8, weak_nodes=0
  1139. Mar 31 01:24:18 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 426807218 (32ms)
  1140. Mar 31 01:24:18 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [connected]
  1141. Mar 31 01:24:18 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( Off -> Established ) [connected]
  1142. Mar 31 01:24:18 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: cleared bm UUID and bitmap 752996FB4D06CB50:0000000000000000:D8796B7E2A90EF4C:ADB250ED33B11982
  1143. Mar 31 01:24:20 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2001482756: 1->2 role( Secondary ) conn( Connected )
  1144. Mar 31 01:24:20 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 2001482756 (primary_nodes=8)
  1145. Mar 31 01:24:30 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: PingAck did not arrive in time.
  1146. Mar 31 01:24:30 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  1147. Mar 31 01:24:30 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( Established -> Off )
  1148. Mar 31 01:24:30 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1784832376: 0->all empty
  1149. Mar 31 01:24:30 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Terminating sender thread
  1150. Mar 31 01:24:30 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Starting sender thread (peer-node-id 1)
  1151. Mar 31 01:24:30 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Connection closed
  1152. Mar 31 01:24:30 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( NetworkFailure -> Unconnected ) [disconnected]
  1153. Mar 31 01:24:30 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Restarting receiver thread
  1154. Mar 31 01:24:30 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Unconnected -> Connecting ) [connecting]
  1155. Mar 31 01:24:31 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 212ms
  1156. Mar 31 01:24:31 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4218099027: 2->all empty
  1157. Mar 31 01:24:31 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 4218099027 (primary_nodes=8)
  1158. Mar 31 01:24:31 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1696622909: 0->all empty
  1159. Mar 31 01:24:31 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 1696622909: primary_nodes=8, weak_nodes=0
  1160. Mar 31 01:24:31 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 1696622909 (0ms)
  1161. Mar 31 01:24:31 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3061440329: 3->all empty
  1162. Mar 31 01:24:31 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 3061440329 (primary_nodes=8)
  1163. Mar 31 01:25:22 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4181019255: 1->2 role( Secondary ) conn( Connected )
  1164. Mar 31 01:25:22 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 4181019255 (primary_nodes=8)
  1165. Mar 31 01:25:22 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 283636919: 1->3 role( Secondary ) conn( Connected )
  1166. Mar 31 01:25:22 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 283636919 (primary_nodes=8)
  1167. Mar 31 01:25:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Handshake to peer 1 successful: Agreed network protocol version 122
  1168. Mar 31 01:25:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  1169. Mar 31 01:25:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Peer authenticated using 20 bytes HMAC
  1170. Mar 31 01:25:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1804534236: 0->1 role( Secondary ) conn( Connected )
  1171. Mar 31 01:25:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: drbd_sync_handshake:
  1172. Mar 31 01:25:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: self 752996FB4D06CB50:0000000000000000:D8796B7E2A90EF4C:ADB250ED33B11982 bits:0 flags:120
  1173. Mar 31 01:25:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: peer 752996FB4D06CB50:0000000000000000:ADB250ED33B11982:D8796B7E2A90EF4C bits:0 flags:1100
  1174. Mar 31 01:25:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: uuid_compare()=no-sync by rule=reconnected
  1175. Mar 31 01:25:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 1804534236: primary_nodes=8, weak_nodes=0
  1176. Mar 31 01:25:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 1804534236 (36ms)
  1177. Mar 31 01:25:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [connected]
  1178. Mar 31 01:25:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( Off -> Established ) [connected]
  1179. Mar 31 01:25:27 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: cleared bm UUID and bitmap 752996FB4D06CB50:0000000000000000:D8796B7E2A90EF4C:ADB250ED33B11982
  1180. Mar 31 01:45:33 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3130963472: 3->1 conn( Disconnecting ) pdsk( Outdated )
  1181. Mar 31 01:45:33 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Duplicate prepare for remote state change 3130963472
  1182. Mar 31 01:45:33 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 3130963472 (primary_nodes=8)
  1183. Mar 31 01:45:33 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3905303763: 3->2 conn( Disconnecting ) pdsk( Outdated )
  1184. Mar 31 01:45:33 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 3905303763 (primary_nodes=8)
  1185. Mar 31 01:45:40 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Handshake to peer 3 successful: Agreed network protocol version 122
  1186. Mar 31 01:45:40 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  1187. Mar 31 01:45:40 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3269594929: 1->3 role( Secondary ) conn( Connected )
  1188. Mar 31 01:45:40 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Peer authenticated using 20 bytes HMAC
  1189. Mar 31 01:45:40 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 3269594929 (primary_nodes=8)
  1190. Mar 31 01:45:40 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3578569625: 2->3 role( Secondary ) conn( Connected )
  1191. Mar 31 01:45:40 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 3578569625 (primary_nodes=8)
  1192. Mar 31 01:45:40 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 775278204: 0->3 role( Secondary ) conn( Connected )
  1193. Mar 31 01:45:40 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: self 752996FB4D06CB50:28D31133FEBC993B:D8796B7E2A90EF4C:ADB250ED33B11982 bits:0 flags:0
  1194. Mar 31 01:45:40 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: peer's exposed UUID: 28983A7814F88E65
  1195. Mar 31 01:45:40 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 775278204: primary_nodes=8, weak_nodes=0
  1196. Mar 31 01:45:40 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 775278204 (32ms)
  1197. Mar 31 01:45:40 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connecting -> Connected ) peer( Unknown -> Primary ) [connected]
  1198. Mar 31 01:45:40 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( DUnknown -> Diskless ) repl( Off -> Established ) [connected]
  1199. Mar 31 02:29:30 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4051346483: 2->all empty
  1200. Mar 31 02:29:30 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Rejecting concurrent remote state change 3984025428 because of state change 4051346483
  1201. Mar 31 02:29:30 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: PingAck did not arrive in time.
  1202. Mar 31 02:29:30 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )
  1203. Mar 31 02:29:30 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-0: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
  1204. Mar 31 02:29:30 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Aborting remote state change 4051346483
  1205. Mar 31 02:29:30 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Terminating sender thread
  1206. Mar 31 02:29:30 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Starting sender thread (peer-node-id 3)
  1207. Mar 31 02:29:30 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Connection closed
  1208. Mar 31 02:29:30 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( NetworkFailure -> Unconnected ) [disconnected]
  1209. Mar 31 02:29:30 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: Restarting receiver thread
  1210. Mar 31 02:29:30 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-0: conn( Unconnected -> Connecting ) [connecting]
  1211. Mar 31 02:29:30 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3048988671: 1->all empty
  1212. Mar 31 02:29:30 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 3048988671 (primary_nodes=0)
  1213. Mar 31 02:29:30 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3703767319: 0->all empty
  1214. Mar 31 02:29:30 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 3703767319: primary_nodes=0, weak_nodes=0
  1215. Mar 31 02:29:30 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 3703767319 (0ms)
  1216. Mar 31 02:29:30 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4067614425: 2->all empty
  1217. Mar 31 02:29:30 storage-load-test-2 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 4067614425 (primary_nodes=0)
  1218.  
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!