API tools faq
paste
Advertisement
Guest User

dmesg_storage-load-test-0.log

a guest
Mar 30th, 2025
29
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 160.29 KB | None | 0 0
raw download clone embed print report
  1. Mar 31 01:07:58 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Starting worker thread (node-id 3)
  2. Mar 31 01:07:58 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Starting sender thread (peer-node-id 1)
  3. Mar 31 01:07:58 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Starting sender thread (peer-node-id 0)
  4. Mar 31 01:07:58 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Starting sender thread (peer-node-id 2)
  5. Mar 31 01:07:58 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( no -> quorum ) [connect]
  6. Mar 31 01:07:58 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( StandAlone -> Unconnected ) [connect]
  7. Mar 31 01:07:58 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Starting receiver thread (peer-node-id 1)
  8. Mar 31 01:07:58 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Unconnected -> Connecting ) [connecting]
  9. Mar 31 01:07:58 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( StandAlone -> Unconnected ) [connect]
  10. Mar 31 01:07:58 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Starting receiver thread (peer-node-id 0)
  11. Mar 31 01:07:58 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Unconnected -> Connecting ) [connecting]
  12. Mar 31 01:07:58 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( StandAlone -> Unconnected ) [connect]
  13. Mar 31 01:07:58 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Starting receiver thread (peer-node-id 2)
  14. Mar 31 01:07:58 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Unconnected -> Connecting ) [connecting]
  15. Mar 31 01:07:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Handshake to peer 1 successful: Agreed network protocol version 122
  16. Mar 31 01:07:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  17. Mar 31 01:07:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Handshake to peer 2 successful: Agreed network protocol version 122
  18. Mar 31 01:07:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  19. Mar 31 01:07:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Peer authenticated using 20 bytes HMAC
  20. Mar 31 01:07:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Peer authenticated using 20 bytes HMAC
  21. Mar 31 01:07:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4112650981: 2->3 role( Secondary ) conn( Connected )
  22. Mar 31 01:07:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: size = 1028 MB (1052408 KB)
  23. Mar 31 01:07:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: my exposed UUID: 0000000000000000
  24. Mar 31 01:07:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: peer 52F5EAB2DCF34652:FFFFFFFFFFFFFFFF:28D31133FEBC993A:0000000000000000 bits:0 flags:1020
  25. Mar 31 01:07:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: Setting exposed data uuid: 52F5EAB2DCF34652
  26. Mar 31 01:07:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 4112650981 (primary_nodes=0)
  27. Mar 31 01:07:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( quorum -> no ) [remote]
  28. Mar 31 01:07:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  29. Mar 31 01:07:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( no -> yes ) [remote]
  30. Mar 31 01:07:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( DUnknown -> UpToDate ) repl( Off -> Established ) [remote]
  31. Mar 31 01:07:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1056996341: 1->3 role( Secondary ) conn( Connected )
  32. Mar 31 01:07:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: my exposed UUID: 52F5EAB2DCF34652
  33. Mar 31 01:07:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: peer 52F5EAB2DCF34652:FFFFFFFFFFFFFFFF:28D31133FEBC993A:0000000000000000 bits:0 flags:1020
  34. Mar 31 01:07:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Handshake to peer 0 successful: Agreed network protocol version 122
  35. Mar 31 01:07:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  36. Mar 31 01:07:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Peer authenticated using 20 bytes HMAC
  37. Mar 31 01:07:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 1056996341 (primary_nodes=0)
  38. Mar 31 01:07:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  39. Mar 31 01:07:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( DUnknown -> UpToDate ) repl( Off -> Established ) [remote]
  40. Mar 31 01:07:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1852736253: 0->3 role( Secondary ) conn( Connected )
  41. Mar 31 01:07:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: my exposed UUID: 52F5EAB2DCF34652
  42. Mar 31 01:07:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: peer 52F5EAB2DCF34652:FFFFFFFFFFFFFFFF:0000000000000000:0000000000000000 bits:0 flags:1020
  43. Mar 31 01:07:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 1852736253 (primary_nodes=0)
  44. Mar 31 01:07:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  45. Mar 31 01:07:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( DUnknown -> UpToDate ) repl( Off -> Established ) [remote]
  46. Mar 31 01:08:04 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 229711114: 3->all role( Primary )
  47. Mar 31 01:08:04 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 229711114: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF0
  48. Mar 31 01:08:04 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 229711114 (0ms)
  49. Mar 31 01:08:04 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: role( Secondary -> Primary ) [mount:112387 auto-promote]
  50. Mar 31 01:08:04 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: sending new current UUID: 3B36CC1B2C2CCBAD
  51. Mar 31 01:08:40 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3427757018: 0->all empty
  52. Mar 31 01:08:41 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Aborting remote state change 3427757018
  53. Mar 31 01:08:41 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 780112742: 1->all empty
  54. Mar 31 01:08:41 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: PingAck did not arrive in time.
  55. Mar 31 01:08:41 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  56. Mar 31 01:08:41 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
  57. Mar 31 01:08:41 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Terminating sender thread
  58. Mar 31 01:08:41 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Starting sender thread (peer-node-id 2)
  59. Mar 31 01:08:41 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Aborting remote state change 780112742
  60. Mar 31 01:08:41 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1468276786: 3->all empty
  61. Mar 31 01:08:41 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Aborting local state change 1468276786 to yield to remote state change 457764969.
  62. Mar 31 01:08:41 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 192ms
  63. Mar 31 01:08:41 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 457764969: 0->all empty
  64. Mar 31 01:08:41 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Aborting remote state change 457764969
  65. Mar 31 01:08:41 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Connection closed
  66. Mar 31 01:08:41 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( NetworkFailure -> Unconnected ) [disconnected]
  67. Mar 31 01:08:41 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Restarting receiver thread
  68. Mar 31 01:08:41 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Unconnected -> Connecting ) [connecting]
  69. Mar 31 01:08:41 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 999804101: 3->all empty
  70. Mar 31 01:08:41 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 999804101: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF4
  71. Mar 31 01:08:41 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 999804101 (0ms)
  72. Mar 31 01:08:41 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: sending new current UUID: ADB250ED33B11983
  73. Mar 31 01:08:41 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1825593975: 1->all empty
  74. Mar 31 01:08:41 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 1825593975 (primary_nodes=8)
  75. Mar 31 01:08:42 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3548852179: 0->all empty
  76. Mar 31 01:08:42 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 3548852179 (primary_nodes=8)
  77. Mar 31 01:09:24 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 841577807: 0->2 role( Secondary ) conn( Connected )
  78. Mar 31 01:09:24 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 841577807 (primary_nodes=8)
  79. Mar 31 01:09:25 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3079894093: 1->2 role( Secondary ) conn( Connected )
  80. Mar 31 01:09:25 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 3079894093 (primary_nodes=8)
  81. Mar 31 01:09:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Handshake to peer 2 successful: Agreed network protocol version 122
  82. Mar 31 01:09:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  83. Mar 31 01:09:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Peer authenticated using 20 bytes HMAC
  84. Mar 31 01:09:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 97363440: 2->3 role( Secondary ) conn( Connected )
  85. Mar 31 01:09:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: my exposed UUID: ADB250ED33B11983
  86. Mar 31 01:09:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: peer 3B36CC1B2C2CCBAC:28D31133FEBC993A:52F5EAB2DCF34652:28D31133FEBC993A bits:0 flags:1124
  87. Mar 31 01:09:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 97363440 (primary_nodes=8)
  88. Mar 31 01:09:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  89. Mar 31 01:09:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( DUnknown -> Inconsistent ) repl( Off -> Established ) [remote]
  90. Mar 31 01:09:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: resync-susp( no -> peer ) [peer-state]
  91. Mar 31 01:09:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( Inconsistent -> Consistent ) resync-susp( peer -> no ) [peer-state]
  92. Mar 31 01:09:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: Peer is on same UUID now
  93. Mar 31 01:09:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( Consistent -> UpToDate ) [receive-uuids]
  94. Mar 31 01:09:46 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: meta connection shut down by peer.
  95. Mar 31 01:09:46 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  96. Mar 31 01:09:46 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
  97. Mar 31 01:09:46 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1088512057: 3->all empty
  98. Mar 31 01:09:46 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Terminating sender thread
  99. Mar 31 01:09:46 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Starting sender thread (peer-node-id 1)
  100. Mar 31 01:09:46 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 264ms
  101. Mar 31 01:09:46 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3183198224: 2->all empty
  102. Mar 31 01:09:46 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 3183198224 (primary_nodes=8)
  103. Mar 31 01:09:46 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2895404419: 0->all empty
  104. Mar 31 01:09:46 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 2895404419 (primary_nodes=8)
  105. Mar 31 01:09:46 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: sending new current UUID: DD95BBD64DA6BD63
  106. Mar 31 01:09:46 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Connection closed
  107. Mar 31 01:09:46 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( NetworkFailure -> Unconnected ) [disconnected]
  108. Mar 31 01:09:46 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Restarting receiver thread
  109. Mar 31 01:09:46 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Unconnected -> Connecting ) [connecting]
  110. Mar 31 01:09:46 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2955162861: 3->all empty
  111. Mar 31 01:09:46 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 2955162861: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF2
  112. Mar 31 01:09:46 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 2955162861 (0ms)
  113. Mar 31 01:10:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4229506866: 1->2 role( Secondary ) conn( Connected )
  114. Mar 31 01:10:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 4229506866 (primary_nodes=8)
  115. Mar 31 01:10:15 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 461079150: 0->1 role( Secondary ) conn( Connected )
  116. Mar 31 01:10:15 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 461079150 (primary_nodes=8)
  117. Mar 31 01:10:16 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Handshake to peer 1 successful: Agreed network protocol version 122
  118. Mar 31 01:10:16 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  119. Mar 31 01:10:16 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Peer authenticated using 20 bytes HMAC
  120. Mar 31 01:10:16 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4185765531: 1->3 role( Secondary ) conn( Connected )
  121. Mar 31 01:10:16 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: my exposed UUID: DD95BBD64DA6BD63
  122. Mar 31 01:10:16 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: peer ADB250ED33B11982:28D31133FEBC993A:3B36CC1B2C2CCBAC:52F5EAB2DCF34652 bits:0 flags:1124
  123. Mar 31 01:10:16 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 4185765531 (primary_nodes=8)
  124. Mar 31 01:10:16 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  125. Mar 31 01:10:16 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( DUnknown -> Inconsistent ) repl( Off -> Established ) [remote]
  126. Mar 31 01:10:16 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: resync-susp( no -> peer ) [peer-state]
  127. Mar 31 01:10:16 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( Inconsistent -> Consistent ) resync-susp( peer -> no ) [peer-state]
  128. Mar 31 01:10:16 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: Peer is on same UUID now
  129. Mar 31 01:10:16 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( Consistent -> UpToDate ) [receive-uuids]
  130. Mar 31 01:10:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: meta connection shut down by peer.
  131. Mar 31 01:10:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  132. Mar 31 01:10:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
  133. Mar 31 01:10:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3267529743: 3->all empty
  134. Mar 31 01:10:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Terminating sender thread
  135. Mar 31 01:10:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Starting sender thread (peer-node-id 0)
  136. Mar 31 01:10:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: meta connection shut down by peer.
  137. Mar 31 01:10:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  138. Mar 31 01:10:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
  139. Mar 31 01:10:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 56ms
  140. Mar 31 01:10:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Terminating sender thread
  141. Mar 31 01:10:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Starting sender thread (peer-node-id 1)
  142. Mar 31 01:10:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: PingAck did not arrive in time.
  143. Mar 31 01:10:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( no -> no-disk,quorum )
  144. Mar 31 01:10:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  145. Mar 31 01:10:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( yes -> no )
  146. Mar 31 01:10:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
  147. Mar 31 01:10:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Connection closed
  148. Mar 31 01:10:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( NetworkFailure -> Unconnected ) [disconnected]
  149. Mar 31 01:10:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Restarting receiver thread
  150. Mar 31 01:10:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Unconnected -> Connecting ) [connecting]
  151. Mar 31 01:10:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Connection closed
  152. Mar 31 01:10:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( NetworkFailure -> Unconnected ) [disconnected]
  153. Mar 31 01:10:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Restarting receiver thread
  154. Mar 31 01:10:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Unconnected -> Connecting ) [connecting]
  155. Mar 31 01:10:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Terminating sender thread
  156. Mar 31 01:10:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Starting sender thread (peer-node-id 2)
  157. Mar 31 01:10:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3367012559: 3->all empty
  158. Mar 31 01:10:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 3367012559 (0ms)
  159. Mar 31 01:10:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Connection closed
  160. Mar 31 01:10:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( NetworkFailure -> Unconnected ) [disconnected]
  161. Mar 31 01:10:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Restarting receiver thread
  162. Mar 31 01:10:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Unconnected -> Connecting ) [connecting]
  163. Mar 31 01:11:12 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Handshake to peer 1 successful: Agreed network protocol version 122
  164. Mar 31 01:11:12 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  165. Mar 31 01:11:12 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Peer authenticated using 20 bytes HMAC
  166. Mar 31 01:11:12 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1914803476: 1->3 role( Secondary ) conn( Connected )
  167. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: my exposed UUID: DD95BBD64DA6BD63
  168. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: peer DD95BBD64DA6BD62:28D31133FEBC993A:3B36CC1B2C2CCBAC:52F5EAB2DCF34652 bits:0 flags:1120
  169. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 1914803476 (primary_nodes=8)
  170. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( no-disk,quorum -> no ) [remote]
  171. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  172. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( no -> yes ) [remote]
  173. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( DUnknown -> UpToDate ) repl( Off -> Established ) [remote]
  174. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3587873295: 3->all empty
  175. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 3587873295: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF5
  176. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 3587873295 (28ms)
  177. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: sock was shut down by peer
  178. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( no -> no-disk,quorum )
  179. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connected -> BrokenPipe ) peer( Secondary -> Unknown )
  180. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( yes -> no )
  181. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
  182. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: meta connection shut down by peer.
  183. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2143414000: 3->all empty
  184. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 2143414000 (0ms)
  185. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Terminating sender thread
  186. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Starting sender thread (peer-node-id 1)
  187. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d tcp:storage-load-test-3: Closing unexpected connection from 172.17.1.3
  188. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Connection closed
  189. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( BrokenPipe -> Unconnected ) [disconnected]
  190. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Restarting receiver thread
  191. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Unconnected -> Connecting ) [connecting]
  192. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Handshake to peer 1 successful: Agreed network protocol version 122
  193. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  194. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Peer authenticated using 20 bytes HMAC
  195. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1142912854: 1->3 role( Secondary ) conn( Connected )
  196. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: my exposed UUID: DD95BBD64DA6BD63
  197. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: peer DD95BBD64DA6BD62:28D31133FEBC993A:3B36CC1B2C2CCBAC:52F5EAB2DCF34652 bits:0 flags:1120
  198. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 1142912854 (primary_nodes=8)
  199. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( no-disk,quorum -> quorum ) [remote]
  200. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  201. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( DUnknown -> UpToDate ) repl( Off -> Established ) [remote]
  202. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1970256621: 3->all empty
  203. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 1970256621: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF5
  204. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 1970256621 (12ms)
  205. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Handshake to peer 2 successful: Agreed network protocol version 122
  206. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  207. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Peer authenticated using 20 bytes HMAC
  208. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1547400645: 2->3 role( Secondary ) conn( Connected )
  209. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: my exposed UUID: DD95BBD64DA6BD63
  210. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: peer DD95BBD64DA6BD63:28D31133FEBC993A:3B36CC1B2C2CCBAC:ADB250ED33B11982 bits:0 flags:1120
  211. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 1547400645 (primary_nodes=8)
  212. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  213. Mar 31 01:11:13 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( DUnknown -> Outdated ) repl( Off -> Established ) [remote]
  214. Mar 31 01:11:16 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Handshake to peer 0 successful: Agreed network protocol version 122
  215. Mar 31 01:11:16 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  216. Mar 31 01:11:16 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Peer authenticated using 20 bytes HMAC
  217. Mar 31 01:11:16 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1713475212: 0->3 role( Secondary ) conn( Connected )
  218. Mar 31 01:11:16 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: my exposed UUID: DD95BBD64DA6BD63
  219. Mar 31 01:11:16 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: peer DD95BBD64DA6BD63:28D31133FEBC993B:ADB250ED33B11982:3B36CC1B2C2CCBAC bits:0 flags:1120
  220. Mar 31 01:11:16 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 1713475212 (primary_nodes=8)
  221. Mar 31 01:11:16 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  222. Mar 31 01:11:16 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( DUnknown -> Outdated ) repl( Off -> Established ) [remote]
  223. Mar 31 01:11:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: meta connection shut down by peer.
  224. Mar 31 01:11:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  225. Mar 31 01:11:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( Established -> Off )
  226. Mar 31 01:11:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1652681402: 0->all empty
  227. Mar 31 01:11:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Aborting remote state change 1652681402
  228. Mar 31 01:11:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4152775671: 2->all empty
  229. Mar 31 01:11:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Terminating sender thread
  230. Mar 31 01:11:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Starting sender thread (peer-node-id 0)
  231. Mar 31 01:11:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 4152775671 (primary_nodes=8)
  232. Mar 31 01:11:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1902223197: 3->all empty
  233. Mar 31 01:11:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Aborting local state change 1902223197 to yield to remote state change 797125860.
  234. Mar 31 01:11:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 352ms
  235. Mar 31 01:11:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 797125860: 1->all empty
  236. Mar 31 01:11:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 797125860 (primary_nodes=8)
  237. Mar 31 01:11:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Connection closed
  238. Mar 31 01:11:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( NetworkFailure -> Unconnected ) [disconnected]
  239. Mar 31 01:11:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Restarting receiver thread
  240. Mar 31 01:11:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Unconnected -> Connecting ) [connecting]
  241. Mar 31 01:11:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3924326760: 3->all empty
  242. Mar 31 01:11:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 3924326760: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF1
  243. Mar 31 01:11:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 3924326760 (0ms)
  244. Mar 31 01:12:03 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2217267134: 0->2 role( Secondary ) conn( Connected )
  245. Mar 31 01:12:03 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 2217267134 (primary_nodes=8)
  246. Mar 31 01:12:08 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Handshake to peer 0 successful: Agreed network protocol version 122
  247. Mar 31 01:12:08 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  248. Mar 31 01:12:08 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Peer authenticated using 20 bytes HMAC
  249. Mar 31 01:12:08 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2130346017: 0->3 role( Secondary ) conn( Connected )
  250. Mar 31 01:12:08 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: my exposed UUID: DD95BBD64DA6BD63
  251. Mar 31 01:12:08 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: peer DD95BBD64DA6BD63:28D31133FEBC993B:ADB250ED33B11982:3B36CC1B2C2CCBAC bits:0 flags:1120
  252. Mar 31 01:12:08 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 2130346017 (primary_nodes=8)
  253. Mar 31 01:12:08 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  254. Mar 31 01:12:08 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( Off -> Established ) [remote]
  255. Mar 31 01:12:08 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( quorum -> no ) [peer-state]
  256. Mar 31 01:12:08 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( no -> yes ) [peer-state]
  257. Mar 31 01:12:08 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( Outdated -> UpToDate ) [peer-state]
  258. Mar 31 01:12:11 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1527465761: 0->1 role( Secondary ) conn( Connected )
  259. Mar 31 01:12:11 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 1527465761 (primary_nodes=8)
  260. Mar 31 01:12:24 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: meta connection shut down by peer.
  261. Mar 31 01:12:24 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  262. Mar 31 01:12:24 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( Established -> Off )
  263. Mar 31 01:12:24 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1389696837: 3->all empty
  264. Mar 31 01:12:24 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Terminating sender thread
  265. Mar 31 01:12:24 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Starting sender thread (peer-node-id 2)
  266. Mar 31 01:12:24 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 64ms
  267. Mar 31 01:12:24 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Connection closed
  268. Mar 31 01:12:24 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( NetworkFailure -> Unconnected ) [disconnected]
  269. Mar 31 01:12:24 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Restarting receiver thread
  270. Mar 31 01:12:24 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Unconnected -> Connecting ) [connecting]
  271. Mar 31 01:12:24 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2149326564: 3->all empty
  272. Mar 31 01:12:24 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 1052ms
  273. Mar 31 01:12:25 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3714550546: 1->all empty
  274. Mar 31 01:12:25 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Aborting remote state change 3714550546
  275. Mar 31 01:12:25 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2301440963: 0->all empty
  276. Mar 31 01:12:25 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 2301440963 (primary_nodes=8)
  277. Mar 31 01:12:25 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3306662247: 1->all empty
  278. Mar 31 01:12:25 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 3306662247 (primary_nodes=8)
  279. Mar 31 01:12:25 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 4059018192: 3->all empty
  280. Mar 31 01:12:25 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 4059018192: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF4
  281. Mar 31 01:12:25 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 4059018192 (0ms)
  282. Mar 31 01:12:47 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Handshake to peer 2 successful: Agreed network protocol version 122
  283. Mar 31 01:12:47 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  284. Mar 31 01:12:47 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Peer authenticated using 20 bytes HMAC
  285. Mar 31 01:12:47 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4282827581: 2->3 role( Secondary ) conn( Connected )
  286. Mar 31 01:12:47 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: my exposed UUID: DD95BBD64DA6BD63
  287. Mar 31 01:12:47 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: peer DD95BBD64DA6BD63:28D31133FEBC993A:3B36CC1B2C2CCBAC:ADB250ED33B11982 bits:0 flags:1120
  288. Mar 31 01:12:47 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 4282827581 (primary_nodes=8)
  289. Mar 31 01:12:47 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  290. Mar 31 01:12:47 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( Off -> Established ) [remote]
  291. Mar 31 01:12:47 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( Outdated -> UpToDate ) [peer-state]
  292. Mar 31 01:12:47 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3429861694: 0->2 role( Secondary ) conn( Connected )
  293. Mar 31 01:12:47 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 3429861694 (primary_nodes=8)
  294. Mar 31 01:12:47 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1152771459: 1->2 role( Secondary ) conn( Connected )
  295. Mar 31 01:12:47 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 1152771459 (primary_nodes=8)
  296. Mar 31 01:13:05 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: meta connection shut down by peer.
  297. Mar 31 01:13:05 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  298. Mar 31 01:13:05 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
  299. Mar 31 01:13:05 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3602058191: 3->all empty
  300. Mar 31 01:13:05 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Terminating sender thread
  301. Mar 31 01:13:05 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Starting sender thread (peer-node-id 0)
  302. Mar 31 01:13:05 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: meta connection shut down by peer.
  303. Mar 31 01:13:05 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  304. Mar 31 01:13:05 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
  305. Mar 31 01:13:05 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 88ms
  306. Mar 31 01:13:05 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Terminating sender thread
  307. Mar 31 01:13:05 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Starting sender thread (peer-node-id 1)
  308. Mar 31 01:13:05 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: meta connection shut down by peer.
  309. Mar 31 01:13:05 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( no -> no-disk,quorum )
  310. Mar 31 01:13:05 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  311. Mar 31 01:13:05 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( yes -> no )
  312. Mar 31 01:13:05 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
  313. Mar 31 01:13:05 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Connection closed
  314. Mar 31 01:13:05 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( NetworkFailure -> Unconnected ) [disconnected]
  315. Mar 31 01:13:05 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Restarting receiver thread
  316. Mar 31 01:13:05 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Unconnected -> Connecting ) [connecting]
  317. Mar 31 01:13:05 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Terminating sender thread
  318. Mar 31 01:13:05 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Starting sender thread (peer-node-id 2)
  319. Mar 31 01:13:05 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Connection closed
  320. Mar 31 01:13:05 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( NetworkFailure -> Unconnected ) [disconnected]
  321. Mar 31 01:13:05 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Restarting receiver thread
  322. Mar 31 01:13:05 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Unconnected -> Connecting ) [connecting]
  323. Mar 31 01:13:05 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Connection closed
  324. Mar 31 01:13:05 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( NetworkFailure -> Unconnected ) [disconnected]
  325. Mar 31 01:13:05 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Restarting receiver thread
  326. Mar 31 01:13:05 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Unconnected -> Connecting ) [connecting]
  327. Mar 31 01:13:05 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2163763147: 3->all empty
  328. Mar 31 01:13:05 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 2163763147 (0ms)
  329. Mar 31 01:13:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Handshake to peer 1 successful: Agreed network protocol version 122
  330. Mar 31 01:13:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  331. Mar 31 01:13:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Peer authenticated using 20 bytes HMAC
  332. Mar 31 01:13:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3125836916: 1->3 role( Secondary ) conn( Connected )
  333. Mar 31 01:13:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: my exposed UUID: DD95BBD64DA6BD63
  334. Mar 31 01:13:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: peer DD95BBD64DA6BD62:28D31133FEBC993A:3B36CC1B2C2CCBAC:52F5EAB2DCF34652 bits:0 flags:1120
  335. Mar 31 01:13:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 3125836916 (primary_nodes=8)
  336. Mar 31 01:13:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( no-disk,quorum -> no ) [remote]
  337. Mar 31 01:13:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  338. Mar 31 01:13:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( no -> yes ) [remote]
  339. Mar 31 01:13:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( DUnknown -> UpToDate ) repl( Off -> Established ) [remote]
  340. Mar 31 01:13:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2402789986: 3->all empty
  341. Mar 31 01:13:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 2402789986: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF5
  342. Mar 31 01:13:43 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 2402789986 (20ms)
  343. Mar 31 01:13:46 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Handshake to peer 0 successful: Agreed network protocol version 122
  344. Mar 31 01:13:46 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  345. Mar 31 01:13:46 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Peer authenticated using 20 bytes HMAC
  346. Mar 31 01:13:46 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1570761256: 0->3 role( Secondary ) conn( Connected )
  347. Mar 31 01:13:46 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change failed: No quorum (-25)
  348. Mar 31 01:13:46 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: my exposed UUID: DD95BBD64DA6BD63
  349. Mar 31 01:13:46 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: peer DD95BBD64DA6BD63:28D31133FEBC993B:ADB250ED33B11982:3B36CC1B2C2CCBAC bits:0 flags:1120
  350. Mar 31 01:13:46 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Aborting remote state change 1570761256
  351. Mar 31 01:13:47 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3138477124: 0->3 role( Secondary ) conn( Connected )
  352. Mar 31 01:13:47 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change failed: No quorum (-25)
  353. Mar 31 01:13:47 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: my exposed UUID: DD95BBD64DA6BD63
  354. Mar 31 01:13:47 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: peer DD95BBD64DA6BD63:28D31133FEBC993B:ADB250ED33B11982:3B36CC1B2C2CCBAC bits:0 flags:1120
  355. Mar 31 01:13:47 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Aborting remote state change 3138477124
  356. Mar 31 01:13:48 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3535667735: 0->3 role( Secondary ) conn( Connected )
  357. Mar 31 01:13:48 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change failed: No quorum (-25)
  358. Mar 31 01:13:48 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: my exposed UUID: DD95BBD64DA6BD63
  359. Mar 31 01:13:48 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: peer DD95BBD64DA6BD63:28D31133FEBC993B:ADB250ED33B11982:3B36CC1B2C2CCBAC bits:0 flags:1120
  360. Mar 31 01:13:48 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Aborting remote state change 3535667735
  361. Mar 31 01:13:48 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Handshake to peer 2 successful: Agreed network protocol version 122
  362. Mar 31 01:13:48 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  363. Mar 31 01:13:48 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Peer authenticated using 20 bytes HMAC
  364. Mar 31 01:13:48 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2072727487: 2->3 role( Secondary ) conn( Connected )
  365. Mar 31 01:13:48 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change failed: No quorum (-25)
  366. Mar 31 01:13:48 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: my exposed UUID: DD95BBD64DA6BD63
  367. Mar 31 01:13:48 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: peer DD95BBD64DA6BD63:28D31133FEBC993A:3B36CC1B2C2CCBAC:ADB250ED33B11982 bits:0 flags:1120
  368. Mar 31 01:13:48 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Aborting remote state change 2072727487
  369. Mar 31 01:13:49 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3417612: 0->3 role( Secondary ) conn( Connected )
  370. Mar 31 01:13:49 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change failed: No quorum (-25)
  371. Mar 31 01:13:49 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: my exposed UUID: DD95BBD64DA6BD63
  372. Mar 31 01:13:49 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: peer DD95BBD64DA6BD63:28D31133FEBC993B:ADB250ED33B11982:3B36CC1B2C2CCBAC bits:0 flags:1120
  373. Mar 31 01:13:49 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Aborting remote state change 3417612
  374. Mar 31 01:13:49 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 204289545: 2->3 role( Secondary ) conn( Connected )
  375. Mar 31 01:13:49 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change failed: No quorum (-25)
  376. Mar 31 01:13:49 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: my exposed UUID: DD95BBD64DA6BD63
  377. Mar 31 01:13:49 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: peer DD95BBD64DA6BD63:28D31133FEBC993A:3B36CC1B2C2CCBAC:ADB250ED33B11982 bits:0 flags:1120
  378. Mar 31 01:13:49 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Aborting remote state change 204289545
  379. Mar 31 01:13:50 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2615676757: 0->3 role( Secondary ) conn( Connected )
  380. Mar 31 01:13:50 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change failed: No quorum (-25)
  381. Mar 31 01:13:50 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: my exposed UUID: DD95BBD64DA6BD63
  382. Mar 31 01:13:50 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: peer DD95BBD64DA6BD63:28D31133FEBC993B:ADB250ED33B11982:3B36CC1B2C2CCBAC bits:0 flags:1120
  383. Mar 31 01:13:50 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Aborting remote state change 2615676757
  384. Mar 31 01:13:50 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2561739627: 2->3 role( Secondary ) conn( Connected )
  385. Mar 31 01:13:50 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change failed: No quorum (-25)
  386. Mar 31 01:13:50 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: my exposed UUID: DD95BBD64DA6BD63
  387. Mar 31 01:13:50 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: peer DD95BBD64DA6BD63:28D31133FEBC993A:3B36CC1B2C2CCBAC:ADB250ED33B11982 bits:0 flags:1120
  388. Mar 31 01:13:50 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Aborting remote state change 2561739627
  389. Mar 31 01:13:51 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2363701962: 0->3 role( Secondary ) conn( Connected )
  390. Mar 31 01:13:51 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change failed: No quorum (-25)
  391. Mar 31 01:13:51 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: my exposed UUID: DD95BBD64DA6BD63
  392. Mar 31 01:13:51 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: peer DD95BBD64DA6BD63:28D31133FEBC993B:ADB250ED33B11982:3B36CC1B2C2CCBAC bits:0 flags:1120
  393. Mar 31 01:13:51 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Aborting remote state change 2363701962
  394. Mar 31 01:13:51 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1348648508: 2->3 role( Secondary ) conn( Connected )
  395. Mar 31 01:13:51 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change failed: No quorum (-25)
  396. Mar 31 01:13:51 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: my exposed UUID: DD95BBD64DA6BD63
  397. Mar 31 01:13:51 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: peer DD95BBD64DA6BD63:28D31133FEBC993A:3B36CC1B2C2CCBAC:ADB250ED33B11982 bits:0 flags:1120
  398. Mar 31 01:13:51 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Aborting remote state change 1348648508
  399. Mar 31 01:13:52 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3115004085: 0->3 role( Secondary ) conn( Connected )
  400. Mar 31 01:13:52 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change failed: No quorum (-25)
  401. Mar 31 01:13:52 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: my exposed UUID: DD95BBD64DA6BD63
  402. Mar 31 01:13:52 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: peer DD95BBD64DA6BD63:28D31133FEBC993B:ADB250ED33B11982:3B36CC1B2C2CCBAC bits:0 flags:1120
  403. Mar 31 01:13:52 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Aborting remote state change 3115004085
  404. Mar 31 01:13:52 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: meta connection shut down by peer.
  405. Mar 31 01:13:52 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( no -> no-disk,quorum )
  406. Mar 31 01:13:52 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  407. Mar 31 01:13:52 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( yes -> no )
  408. Mar 31 01:13:52 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
  409. Mar 31 01:13:52 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 91655561: 3->all empty
  410. Mar 31 01:13:52 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 91655561 (0ms)
  411. Mar 31 01:13:52 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Terminating sender thread
  412. Mar 31 01:13:52 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Starting sender thread (peer-node-id 1)
  413. Mar 31 01:13:52 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Connection closed
  414. Mar 31 01:13:52 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( NetworkFailure -> Unconnected ) [disconnected]
  415. Mar 31 01:13:52 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Restarting receiver thread
  416. Mar 31 01:13:52 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Unconnected -> Connecting ) [connecting]
  417. Mar 31 01:13:52 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4263931465: 2->3 role( Secondary ) conn( Connected )
  418. Mar 31 01:13:52 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: my exposed UUID: DD95BBD64DA6BD63
  419. Mar 31 01:13:52 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: peer DD95BBD64DA6BD63:28D31133FEBC993A:3B36CC1B2C2CCBAC:ADB250ED33B11982 bits:0 flags:1120
  420. Mar 31 01:13:52 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 4263931465 (primary_nodes=8)
  421. Mar 31 01:13:52 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  422. Mar 31 01:13:52 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( DUnknown -> Outdated ) repl( Off -> Established ) [remote]
  423. Mar 31 01:13:52 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( no-disk,quorum -> quorum ) [peer-state]
  424. Mar 31 01:13:52 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( Outdated -> UpToDate ) [peer-state]
  425. Mar 31 01:13:52 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3928376880: 3->all empty
  426. Mar 31 01:13:52 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Aborting local state change 3928376880 to yield to remote state change 4025883806.
  427. Mar 31 01:13:52 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 224ms
  428. Mar 31 01:13:52 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4025883806: 0->all empty
  429. Mar 31 01:13:52 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 4025883806 (primary_nodes=8)
  430. Mar 31 01:13:52 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( DUnknown -> Outdated ) [far-away]
  431. Mar 31 01:13:53 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3773917208: 3->all empty
  432. Mar 31 01:13:53 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 3773917208: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF3
  433. Mar 31 01:13:53 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 3773917208 (0ms)
  434. Mar 31 01:13:53 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 417657209: 0->3 role( Secondary ) conn( Connected )
  435. Mar 31 01:13:53 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: my exposed UUID: DD95BBD64DA6BD63
  436. Mar 31 01:13:53 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: peer DD95BBD64DA6BD63:28D31133FEBC993B:ADB250ED33B11982:3B36CC1B2C2CCBAC bits:0 flags:1120
  437. Mar 31 01:13:53 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 417657209 (primary_nodes=8)
  438. Mar 31 01:13:53 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  439. Mar 31 01:13:53 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( Off -> Established ) [remote]
  440. Mar 31 01:13:53 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( quorum -> no ) [peer-state]
  441. Mar 31 01:13:53 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( no -> yes ) [peer-state]
  442. Mar 31 01:13:53 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( Outdated -> UpToDate ) [peer-state]
  443. Mar 31 01:14:34 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Handshake to peer 1 successful: Agreed network protocol version 122
  444. Mar 31 01:14:34 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  445. Mar 31 01:14:34 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Peer authenticated using 20 bytes HMAC
  446. Mar 31 01:14:35 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3446771404: 1->3 role( Secondary ) conn( Connected )
  447. Mar 31 01:14:35 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: my exposed UUID: DD95BBD64DA6BD63
  448. Mar 31 01:14:35 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: peer DD95BBD64DA6BD62:28D31133FEBC993A:3B36CC1B2C2CCBAC:52F5EAB2DCF34652 bits:0 flags:1120
  449. Mar 31 01:14:35 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 3446771404 (primary_nodes=8)
  450. Mar 31 01:14:35 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  451. Mar 31 01:14:35 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( DUnknown -> UpToDate ) repl( Off -> Established ) [remote]
  452. Mar 31 01:14:36 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 864936807: 0->1 role( Secondary ) conn( Connected )
  453. Mar 31 01:14:36 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 864936807 (primary_nodes=8)
  454. Mar 31 01:14:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1444235534: 1->2 role( Secondary ) conn( Connected )
  455. Mar 31 01:14:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 1444235534 (primary_nodes=8)
  456. Mar 31 01:14:56 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3111857493: 2->all empty
  457. Mar 31 01:14:56 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 3111857493 (primary_nodes=8)
  458. Mar 31 01:14:57 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: PingAck did not arrive in time.
  459. Mar 31 01:14:57 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  460. Mar 31 01:14:57 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
  461. Mar 31 01:14:57 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1421634337: 3->all empty
  462. Mar 31 01:14:57 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Terminating sender thread
  463. Mar 31 01:14:57 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Starting sender thread (peer-node-id 0)
  464. Mar 31 01:14:57 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Connection closed
  465. Mar 31 01:14:57 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( NetworkFailure -> Unconnected ) [disconnected]
  466. Mar 31 01:14:57 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Restarting receiver thread
  467. Mar 31 01:14:57 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Unconnected -> Connecting ) [connecting]
  468. Mar 31 01:14:57 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 420ms
  469. Mar 31 01:14:57 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3701488114: 1->all empty
  470. Mar 31 01:14:57 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 3701488114 (primary_nodes=8)
  471. Mar 31 01:14:57 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 216853111: 3->all empty
  472. Mar 31 01:14:57 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 216853111: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF1
  473. Mar 31 01:14:57 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 216853111 (0ms)
  474. Mar 31 01:14:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: sending new current UUID: D8796B7E2A90EF4D
  475. Mar 31 01:15:16 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Handshake to peer 0 successful: Agreed network protocol version 122
  476. Mar 31 01:15:16 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  477. Mar 31 01:15:16 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Peer authenticated using 20 bytes HMAC
  478. Mar 31 01:15:16 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1653285823: 0->3 role( Secondary ) conn( Connected )
  479. Mar 31 01:15:16 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: my exposed UUID: D8796B7E2A90EF4D
  480. Mar 31 01:15:16 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: peer DD95BBD64DA6BD62:28D31133FEBC993B:ADB250ED33B11982:3B36CC1B2C2CCBAC bits:0 flags:1120
  481. Mar 31 01:15:16 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: Downgrading joining peer's disk as its data is older
  482. Mar 31 01:15:16 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 1653285823 (primary_nodes=8)
  483. Mar 31 01:15:16 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  484. Mar 31 01:15:16 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( DUnknown -> Outdated ) repl( Off -> Established ) [remote]
  485. Mar 31 01:15:16 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3579449599: 0->1 role( Secondary ) conn( Connected )
  486. Mar 31 01:15:17 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 3579449599 (primary_nodes=8)
  487. Mar 31 01:15:17 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( Outdated -> Inconsistent ) resync-susp( no -> peer ) [peer-state]
  488. Mar 31 01:15:17 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( Inconsistent -> Consistent ) resync-susp( peer -> no ) [peer-state]
  489. Mar 31 01:15:17 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: Peer is on same UUID now
  490. Mar 31 01:15:17 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( Consistent -> UpToDate ) [receive-uuids]
  491. Mar 31 01:15:19 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2130924816: 0->2 role( Secondary ) conn( Connected )
  492. Mar 31 01:15:19 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 2130924816 (primary_nodes=8)
  493. Mar 31 01:15:29 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4158750261: 0->all empty
  494. Mar 31 01:15:30 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: PingAck did not arrive in time.
  495. Mar 31 01:15:30 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  496. Mar 31 01:15:30 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
  497. Mar 31 01:15:30 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Terminating sender thread
  498. Mar 31 01:15:30 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Starting sender thread (peer-node-id 2)
  499. Mar 31 01:15:30 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Aborting remote state change 4158750261
  500. Mar 31 01:15:30 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2173774204: 3->all empty
  501. Mar 31 01:15:30 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Aborting local state change 2173774204 to yield to remote state change 2259277806.
  502. Mar 31 01:15:30 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 296ms
  503. Mar 31 01:15:30 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2259277806: 1->all empty
  504. Mar 31 01:15:30 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 2259277806 (primary_nodes=8)
  505. Mar 31 01:15:30 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: sending new current UUID: 752996FB4D06CB51
  506. Mar 31 01:15:30 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Connection closed
  507. Mar 31 01:15:30 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( NetworkFailure -> Unconnected ) [disconnected]
  508. Mar 31 01:15:30 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Restarting receiver thread
  509. Mar 31 01:15:30 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Unconnected -> Connecting ) [connecting]
  510. Mar 31 01:15:30 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 567398661: 0->all empty
  511. Mar 31 01:15:30 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 567398661 (primary_nodes=8)
  512. Mar 31 01:15:30 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3559347635: 3->all empty
  513. Mar 31 01:15:30 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 3559347635: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF4
  514. Mar 31 01:15:30 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 3559347635 (0ms)
  515. Mar 31 01:16:01 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Handshake to peer 2 successful: Agreed network protocol version 122
  516. Mar 31 01:16:01 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  517. Mar 31 01:16:01 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Peer authenticated using 20 bytes HMAC
  518. Mar 31 01:16:01 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1631962529: 2->3 role( Secondary ) conn( Connected )
  519. Mar 31 01:16:02 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: my exposed UUID: 752996FB4D06CB51
  520. Mar 31 01:16:02 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: peer D8796B7E2A90EF4C:28D31133FEBC993A:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC bits:0 flags:1120
  521. Mar 31 01:16:02 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: Downgrading joining peer's disk as its data is older
  522. Mar 31 01:16:02 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 1631962529 (primary_nodes=8)
  523. Mar 31 01:16:02 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  524. Mar 31 01:16:02 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( DUnknown -> Outdated ) repl( Off -> Established ) [remote]
  525. Mar 31 01:16:02 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3369737057: 0->2 role( Secondary ) conn( Connected )
  526. Mar 31 01:16:02 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 3369737057 (primary_nodes=8)
  527. Mar 31 01:16:02 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( Outdated -> Inconsistent ) resync-susp( no -> peer ) [peer-state]
  528. Mar 31 01:16:02 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( Inconsistent -> Consistent ) resync-susp( peer -> no ) [peer-state]
  529. Mar 31 01:16:02 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: Peer is on same UUID now
  530. Mar 31 01:16:02 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( Consistent -> UpToDate ) [receive-uuids]
  531. Mar 31 01:16:04 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2284281406: 1->2 role( Secondary ) conn( Connected )
  532. Mar 31 01:16:04 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 2284281406 (primary_nodes=8)
  533. Mar 31 01:16:26 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: meta connection shut down by peer.
  534. Mar 31 01:16:26 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  535. Mar 31 01:16:26 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
  536. Mar 31 01:16:26 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1589683345: 3->all empty
  537. Mar 31 01:16:26 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Terminating sender thread
  538. Mar 31 01:16:26 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Starting sender thread (peer-node-id 0)
  539. Mar 31 01:16:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: meta connection shut down by peer.
  540. Mar 31 01:16:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  541. Mar 31 01:16:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
  542. Mar 31 01:16:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 36ms
  543. Mar 31 01:16:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Terminating sender thread
  544. Mar 31 01:16:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Starting sender thread (peer-node-id 1)
  545. Mar 31 01:16:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: PingAck did not arrive in time.
  546. Mar 31 01:16:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( no -> no-disk,quorum )
  547. Mar 31 01:16:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  548. Mar 31 01:16:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( yes -> no )
  549. Mar 31 01:16:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
  550. Mar 31 01:16:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Connection closed
  551. Mar 31 01:16:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( NetworkFailure -> Unconnected ) [disconnected]
  552. Mar 31 01:16:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Restarting receiver thread
  553. Mar 31 01:16:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Unconnected -> Connecting ) [connecting]
  554. Mar 31 01:16:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Terminating sender thread
  555. Mar 31 01:16:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Starting sender thread (peer-node-id 2)
  556. Mar 31 01:16:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: meta connection shut down by peer.
  557. Mar 31 01:16:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Connection closed
  558. Mar 31 01:16:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( NetworkFailure -> Unconnected ) [disconnected]
  559. Mar 31 01:16:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Restarting receiver thread
  560. Mar 31 01:16:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Unconnected -> Connecting ) [connecting]
  561. Mar 31 01:16:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 774757727: 3->all empty
  562. Mar 31 01:16:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 774757727 (0ms)
  563. Mar 31 01:16:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Connection closed
  564. Mar 31 01:16:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( NetworkFailure -> Unconnected ) [disconnected]
  565. Mar 31 01:16:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Restarting receiver thread
  566. Mar 31 01:16:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Unconnected -> Connecting ) [connecting]
  567. Mar 31 01:16:37 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Handshake to peer 0 successful: Agreed network protocol version 122
  568. Mar 31 01:16:37 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  569. Mar 31 01:16:37 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Peer authenticated using 20 bytes HMAC
  570. Mar 31 01:16:37 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3768514093: 0->3 role( Secondary ) conn( Connected )
  571. Mar 31 01:16:37 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: my exposed UUID: 752996FB4D06CB51
  572. Mar 31 01:16:37 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: peer 752996FB4D06CB51:28D31133FEBC993B:D8796B7E2A90EF4C:ADB250ED33B11982 bits:0 flags:1120
  573. Mar 31 01:16:37 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 3768514093 (primary_nodes=8)
  574. Mar 31 01:16:37 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( no-disk,quorum -> uuid ) [remote]
  575. Mar 31 01:16:37 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  576. Mar 31 01:16:37 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( no -> yes ) [remote]
  577. Mar 31 01:16:37 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( DUnknown -> UpToDate ) repl( Off -> Established ) [remote]
  578. Mar 31 01:16:37 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: sending new current UUID: 28983A7814F88E65
  579. Mar 31 01:16:37 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( uuid -> no ) [susp-uuid]
  580. Mar 31 01:16:37 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2217205021: 3->all empty
  581. Mar 31 01:16:37 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: sock was shut down by peer
  582. Mar 31 01:16:37 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( no -> no-disk,quorum )
  583. Mar 31 01:16:37 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connected -> BrokenPipe ) peer( Secondary -> Unknown )
  584. Mar 31 01:16:37 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( yes -> no )
  585. Mar 31 01:16:37 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
  586. Mar 31 01:16:37 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: meta connection shut down by peer.
  587. Mar 31 01:16:37 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 372ms
  588. Mar 31 01:16:37 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Terminating sender thread
  589. Mar 31 01:16:37 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Starting sender thread (peer-node-id 0)
  590. Mar 31 01:16:37 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Connection closed
  591. Mar 31 01:16:37 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( BrokenPipe -> Unconnected ) [disconnected]
  592. Mar 31 01:16:37 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Restarting receiver thread
  593. Mar 31 01:16:37 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Unconnected -> Connecting ) [connecting]
  594. Mar 31 01:16:37 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Handshake to peer 1 successful: Agreed network protocol version 122
  595. Mar 31 01:16:37 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  596. Mar 31 01:16:37 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Peer authenticated using 20 bytes HMAC
  597. Mar 31 01:16:37 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 778947052: 1->3 role( Secondary ) conn( Connected )
  598. Mar 31 01:16:37 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: my exposed UUID: 28983A7814F88E65
  599. Mar 31 01:16:37 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: peer 752996FB4D06CB50:28D31133FEBC993A:ADB250ED33B11982:D8796B7E2A90EF4C bits:0 flags:1120
  600. Mar 31 01:16:37 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 778947052 (primary_nodes=8)
  601. Mar 31 01:16:37 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  602. Mar 31 01:16:37 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: pdsk( DUnknown -> Outdated ) repl( Off -> Established ) [remote]
  603. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Handshake to peer 2 successful: Agreed network protocol version 122
  604. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  605. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Peer authenticated using 20 bytes HMAC
  606. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2467925668: 2->3 role( Secondary ) conn( Connected )
  607. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: my exposed UUID: 28983A7814F88E65
  608. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: peer 752996FB4D06CB50:28D31133FEBC993A:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC bits:0 flags:1120
  609. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 2467925668 (primary_nodes=8)
  610. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  611. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: pdsk( DUnknown -> Outdated ) repl( Off -> Established ) [remote]
  612. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 433044289: 3->all empty
  613. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 433044289: primary_nodes=8, weak_nodes=0
  614. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 433044289 (16ms)
  615. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 722555851: 3->all empty
  616. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 722555851: primary_nodes=8, weak_nodes=0
  617. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 722555851 (4ms)
  618. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Handshake to peer 0 successful: Agreed network protocol version 122
  619. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  620. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Peer authenticated using 20 bytes HMAC
  621. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2386366568: 0->3 role( Secondary ) conn( Connected )
  622. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: my exposed UUID: 28983A7814F88E65
  623. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: peer 28983A7814F88E65:28D31133FEBC993B:D8796B7E2A90EF4C:ADB250ED33B11982 bits:0 flags:1120
  624. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 2386366568 (primary_nodes=8)
  625. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( no-disk,quorum -> quorum ) [remote]
  626. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  627. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( DUnknown -> UpToDate ) repl( Off -> Established ) [remote]
  628. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1209087830: 3->all empty
  629. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change failed: Need access to UpToDate data (-2)
  630. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Failed: susp-io( quorum -> no-disk,quorum ) [peer-state]
  631. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: Failed: pdsk( UpToDate -> Outdated ) [peer-state]
  632. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( quorum -> no-disk,quorum )
  633. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connected -> Disconnecting ) peer( Secondary -> Unknown )
  634. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
  635. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: error receiving P_STATE, e: -5 l: 0!
  636. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 248ms
  637. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Terminating sender thread
  638. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Starting sender thread (peer-node-id 0)
  639. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Connection closed
  640. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Disconnecting -> StandAlone ) [disconnected]
  641. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Terminating receiver thread
  642. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3472239397: 0->all empty
  643. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Duplicate prepare for remote state change 3472239397
  644. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 3472239397 (primary_nodes=8)
  645. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: pdsk( DUnknown -> Outdated ) [far-away]
  646. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2761915777: 3->all empty
  647. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 2761915777: primary_nodes=8, weak_nodes=0
  648. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 2761915777 (0ms)
  649. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1779301481: 3->all empty
  650. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 1779301481: primary_nodes=8, weak_nodes=0
  651. Mar 31 01:16:38 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 1779301481 (0ms)
  652. Mar 31 01:16:58 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1400865519: 2->all empty
  653. Mar 31 01:16:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Aborting remote state change 1400865519
  654. Mar 31 01:16:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2703312900: 0->all empty
  655. Mar 31 01:16:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: meta connection shut down by peer.
  656. Mar 31 01:16:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  657. Mar 31 01:16:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( Established -> Off )
  658. Mar 31 01:16:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Terminating sender thread
  659. Mar 31 01:16:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Starting sender thread (peer-node-id 1)
  660. Mar 31 01:16:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Aborting remote state change 2703312900
  661. Mar 31 01:16:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 525779288: 3->all empty
  662. Mar 31 01:16:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Aborting local state change 525779288 to yield to remote state change 2293185259.
  663. Mar 31 01:16:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 136ms
  664. Mar 31 01:16:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2293185259: 2->all empty
  665. Mar 31 01:16:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 2293185259 (primary_nodes=8)
  666. Mar 31 01:16:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Connection closed
  667. Mar 31 01:16:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( NetworkFailure -> Unconnected ) [disconnected]
  668. Mar 31 01:16:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Restarting receiver thread
  669. Mar 31 01:16:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Unconnected -> Connecting ) [connecting]
  670. Mar 31 01:16:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 4138258921: 3->all empty
  671. Mar 31 01:16:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 4138258921: primary_nodes=8, weak_nodes=0
  672. Mar 31 01:16:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 4138258921 (0ms)
  673. Mar 31 01:16:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 204760639: 0->all empty
  674. Mar 31 01:16:59 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 204760639 (primary_nodes=8)
  675. Mar 31 01:17:09 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d tcp:storage-load-test-1: Closing unexpected connection from 172.17.1.4
  676. Mar 31 01:17:18 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d tcp:storage-load-test-1: Closing unexpected connection from 172.17.1.4
  677. Mar 31 01:17:28 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 332174034: 1->2 role( Secondary ) conn( Connected )
  678. Mar 31 01:17:28 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 332174034 (primary_nodes=8)
  679. Mar 31 01:17:28 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3438731863: 0->1 role( Secondary ) conn( Connected )
  680. Mar 31 01:17:28 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 3438731863 (primary_nodes=8)
  681. Mar 31 01:17:28 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Handshake to peer 1 successful: Agreed network protocol version 122
  682. Mar 31 01:17:28 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  683. Mar 31 01:17:28 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Peer authenticated using 20 bytes HMAC
  684. Mar 31 01:17:28 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 369871165: 1->3 role( Secondary ) conn( Connected )
  685. Mar 31 01:17:28 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: my exposed UUID: 28983A7814F88E65
  686. Mar 31 01:17:28 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: peer 752996FB4D06CB50:28D31133FEBC993A:ADB250ED33B11982:D8796B7E2A90EF4C bits:0 flags:1120
  687. Mar 31 01:17:28 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 369871165 (primary_nodes=8)
  688. Mar 31 01:17:28 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  689. Mar 31 01:17:28 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( Off -> Established ) [remote]
  690. Mar 31 01:17:49 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 581687972: 1->all empty
  691. Mar 31 01:17:50 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Aborting remote state change 581687972
  692. Mar 31 01:17:50 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3634414869: 2->all empty
  693. Mar 31 01:17:50 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 3634414869 (primary_nodes=8)
  694. Mar 31 01:17:50 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1347114405: 1->all empty
  695. Mar 31 01:17:50 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 1347114405 (primary_nodes=8)
  696. Mar 31 01:18:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1603385043: 0->2 role( Secondary ) conn( Connected )
  697. Mar 31 01:18:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 1603385043 (primary_nodes=8)
  698. Mar 31 01:18:29 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2458447738: 0->1 role( Secondary ) conn( Connected )
  699. Mar 31 01:18:29 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Duplicate prepare for remote state change 2458447738
  700. Mar 31 01:18:29 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 2458447738 (primary_nodes=8)
  701. Mar 31 01:18:39 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2710131752: 0->all empty
  702. Mar 31 01:18:39 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 2710131752 (primary_nodes=8)
  703. Mar 31 01:18:39 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: meta connection shut down by peer.
  704. Mar 31 01:18:39 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  705. Mar 31 01:18:39 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( Established -> Off )
  706. Mar 31 01:18:39 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 1561974972: 3->all empty
  707. Mar 31 01:18:39 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Terminating sender thread
  708. Mar 31 01:18:39 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Starting sender thread (peer-node-id 2)
  709. Mar 31 01:18:39 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 1561974972: primary_nodes=8, weak_nodes=0
  710. Mar 31 01:18:39 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 1561974972 (0ms)
  711. Mar 31 01:18:39 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Connection closed
  712. Mar 31 01:18:39 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( NetworkFailure -> Unconnected ) [disconnected]
  713. Mar 31 01:18:39 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Restarting receiver thread
  714. Mar 31 01:18:39 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Unconnected -> Connecting ) [connecting]
  715. Mar 31 01:18:40 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 307708718: 1->all empty
  716. Mar 31 01:18:40 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 307708718 (primary_nodes=8)
  717. Mar 31 01:18:49 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d tcp:storage-load-test-3: Closing unexpected connection from 172.17.1.4
  718. Mar 31 01:18:52 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d tcp:storage-load-test-3: Closing unexpected connection from 172.17.1.4
  719. Mar 31 01:19:04 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4181492047: 0->2 role( Secondary ) conn( Connected )
  720. Mar 31 01:19:05 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 4181492047 (primary_nodes=8)
  721. Mar 31 01:19:05 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Handshake to peer 2 successful: Agreed network protocol version 122
  722. Mar 31 01:19:05 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  723. Mar 31 01:19:05 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Peer authenticated using 20 bytes HMAC
  724. Mar 31 01:19:05 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2159242530: 2->3 role( Secondary ) conn( Connected )
  725. Mar 31 01:19:05 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: my exposed UUID: 28983A7814F88E65
  726. Mar 31 01:19:05 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: peer 752996FB4D06CB50:28D31133FEBC993A:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC bits:0 flags:1120
  727. Mar 31 01:19:05 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 2159242530 (primary_nodes=8)
  728. Mar 31 01:19:05 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  729. Mar 31 01:19:05 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( Off -> Established ) [remote]
  730. Mar 31 01:19:05 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3871126125: 1->2 role( Secondary ) conn( Connected )
  731. Mar 31 01:19:05 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 3871126125 (primary_nodes=8)
  732. Mar 31 01:19:26 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: PingAck did not arrive in time.
  733. Mar 31 01:19:26 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  734. Mar 31 01:19:26 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( Established -> Off )
  735. Mar 31 01:19:26 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 732383613: 3->all empty
  736. Mar 31 01:19:26 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 196ms
  737. Mar 31 01:19:26 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Terminating sender thread
  738. Mar 31 01:19:26 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Starting sender thread (peer-node-id 2)
  739. Mar 31 01:19:26 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Connection closed
  740. Mar 31 01:19:26 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( NetworkFailure -> Unconnected ) [disconnected]
  741. Mar 31 01:19:26 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Restarting receiver thread
  742. Mar 31 01:19:26 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Unconnected -> Connecting ) [connecting]
  743. Mar 31 01:19:26 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3213501202: 3->all empty
  744. Mar 31 01:19:26 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: meta connection shut down by peer.
  745. Mar 31 01:19:26 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  746. Mar 31 01:19:26 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( Established -> Off )
  747. Mar 31 01:19:26 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 68ms
  748. Mar 31 01:19:26 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Terminating sender thread
  749. Mar 31 01:19:26 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Starting sender thread (peer-node-id 1)
  750. Mar 31 01:19:26 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Connection closed
  751. Mar 31 01:19:26 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( NetworkFailure -> Unconnected ) [disconnected]
  752. Mar 31 01:19:26 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Restarting receiver thread
  753. Mar 31 01:19:26 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Unconnected -> Connecting ) [connecting]
  754. Mar 31 01:19:26 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3100446467: 3->all empty
  755. Mar 31 01:19:26 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 3100446467 (0ms)
  756. Mar 31 01:20:00 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Handshake to peer 2 successful: Agreed network protocol version 122
  757. Mar 31 01:20:00 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  758. Mar 31 01:20:00 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Peer authenticated using 20 bytes HMAC
  759. Mar 31 01:20:00 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1789966072: 2->3 role( Secondary ) conn( Connected )
  760. Mar 31 01:20:00 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: my exposed UUID: 28983A7814F88E65
  761. Mar 31 01:20:00 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: peer 752996FB4D06CB50:28D31133FEBC993A:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC bits:0 flags:1120
  762. Mar 31 01:20:00 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 1789966072 (primary_nodes=8)
  763. Mar 31 01:20:00 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  764. Mar 31 01:20:00 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( Off -> Established ) [remote]
  765. Mar 31 01:20:01 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d tcp:storage-load-test-1: Closing unexpected connection from 172.17.1.4
  766. Mar 31 01:20:07 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Handshake to peer 1 successful: Agreed network protocol version 122
  767. Mar 31 01:20:07 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  768. Mar 31 01:20:07 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Peer authenticated using 20 bytes HMAC
  769. Mar 31 01:20:07 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 178417790: 1->3 role( Secondary ) conn( Connected )
  770. Mar 31 01:20:07 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: my exposed UUID: 28983A7814F88E65
  771. Mar 31 01:20:07 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: peer 752996FB4D06CB50:28D31133FEBC993A:ADB250ED33B11982:D8796B7E2A90EF4C bits:0 flags:1120
  772. Mar 31 01:20:07 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 178417790 (primary_nodes=8)
  773. Mar 31 01:20:07 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  774. Mar 31 01:20:07 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( Off -> Established ) [remote]
  775. Mar 31 01:20:18 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1409325880: 0->all empty
  776. Mar 31 01:20:18 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Aborting remote state change 1409325880
  777. Mar 31 01:20:18 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3320924655: 1->all empty
  778. Mar 31 01:20:18 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Aborting remote state change 3320924655
  779. Mar 31 01:20:18 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3305118617: 2->all empty
  780. Mar 31 01:20:18 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 3305118617 (primary_nodes=8)
  781. Mar 31 01:20:19 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3950734662: 1->all empty
  782. Mar 31 01:20:19 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 3950734662 (primary_nodes=8)
  783. Mar 31 01:20:56 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 26180170: 0->1 role( Secondary ) conn( Connected )
  784. Mar 31 01:20:56 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 26180170 (primary_nodes=8)
  785. Mar 31 01:21:00 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 113751511: 0->2 role( Secondary ) conn( Connected )
  786. Mar 31 01:21:00 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Duplicate prepare for remote state change 113751511
  787. Mar 31 01:21:00 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 113751511 (primary_nodes=8)
  788. Mar 31 01:21:07 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3394754177: 2->all empty
  789. Mar 31 01:21:07 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: PingAck did not arrive in time.
  790. Mar 31 01:21:07 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  791. Mar 31 01:21:07 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( Established -> Off )
  792. Mar 31 01:21:07 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Terminating sender thread
  793. Mar 31 01:21:07 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Starting sender thread (peer-node-id 1)
  794. Mar 31 01:21:07 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Aborting remote state change 3394754177
  795. Mar 31 01:21:07 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2936172748: 3->all empty
  796. Mar 31 01:21:07 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 2936172748: primary_nodes=8, weak_nodes=0
  797. Mar 31 01:21:07 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 2936172748 (0ms)
  798. Mar 31 01:21:07 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Connection closed
  799. Mar 31 01:21:07 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( NetworkFailure -> Unconnected ) [disconnected]
  800. Mar 31 01:21:07 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Restarting receiver thread
  801. Mar 31 01:21:07 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Unconnected -> Connecting ) [connecting]
  802. Mar 31 01:21:08 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 191871833: 0->all empty
  803. Mar 31 01:21:08 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 191871833 (primary_nodes=8)
  804. Mar 31 01:21:08 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1983572376: 2->all empty
  805. Mar 31 01:21:08 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 1983572376 (primary_nodes=8)
  806. Mar 31 01:21:17 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d tcp:storage-load-test-1: Closing unexpected connection from 172.17.1.4
  807. Mar 31 01:21:20 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d tcp:storage-load-test-1: Closing unexpected connection from 172.17.1.4
  808. Mar 31 01:21:32 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1146113168: 1->2 role( Secondary ) conn( Connected )
  809. Mar 31 01:21:32 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 1146113168 (primary_nodes=8)
  810. Mar 31 01:21:33 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d tcp:storage-load-test-1: Closing unexpected connection from 172.17.1.4
  811. Mar 31 01:21:34 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Handshake to peer 1 successful: Agreed network protocol version 122
  812. Mar 31 01:21:34 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  813. Mar 31 01:21:34 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Peer authenticated using 20 bytes HMAC
  814. Mar 31 01:21:34 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4164491876: 1->3 role( Secondary ) conn( Connected )
  815. Mar 31 01:21:34 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: my exposed UUID: 28983A7814F88E65
  816. Mar 31 01:21:34 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: peer 752996FB4D06CB50:28D31133FEBC993A:ADB250ED33B11982:D8796B7E2A90EF4C bits:0 flags:1120
  817. Mar 31 01:21:34 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 4164491876 (primary_nodes=8)
  818. Mar 31 01:21:34 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  819. Mar 31 01:21:34 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( Off -> Established ) [remote]
  820. Mar 31 01:21:34 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1675388973: 0->1 role( Secondary ) conn( Connected )
  821. Mar 31 01:21:34 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Duplicate prepare for remote state change 1675388973
  822. Mar 31 01:21:34 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 1675388973 (primary_nodes=8)
  823. Mar 31 01:21:55 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: PingAck did not arrive in time.
  824. Mar 31 01:21:55 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  825. Mar 31 01:21:55 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( Established -> Off )
  826. Mar 31 01:21:55 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: meta connection shut down by peer.
  827. Mar 31 01:21:55 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  828. Mar 31 01:21:55 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( Established -> Off )
  829. Mar 31 01:21:55 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 945640989: 3->all empty
  830. Mar 31 01:21:55 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 945640989 (0ms)
  831. Mar 31 01:21:55 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Terminating sender thread
  832. Mar 31 01:21:55 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Terminating sender thread
  833. Mar 31 01:21:55 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Starting sender thread (peer-node-id 2)
  834. Mar 31 01:21:55 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Starting sender thread (peer-node-id 1)
  835. Mar 31 01:21:55 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Connection closed
  836. Mar 31 01:21:55 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( NetworkFailure -> Unconnected ) [disconnected]
  837. Mar 31 01:21:55 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Restarting receiver thread
  838. Mar 31 01:21:55 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Unconnected -> Connecting ) [connecting]
  839. Mar 31 01:21:55 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Connection closed
  840. Mar 31 01:21:55 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( NetworkFailure -> Unconnected ) [disconnected]
  841. Mar 31 01:21:55 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Restarting receiver thread
  842. Mar 31 01:21:55 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Unconnected -> Connecting ) [connecting]
  843. Mar 31 01:22:30 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d tcp:storage-load-test-1: Closing unexpected connection from 172.17.1.4
  844. Mar 31 01:22:33 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Handshake to peer 2 successful: Agreed network protocol version 122
  845. Mar 31 01:22:33 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  846. Mar 31 01:22:33 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Peer authenticated using 20 bytes HMAC
  847. Mar 31 01:22:33 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3835743607: 2->3 role( Secondary ) conn( Connected )
  848. Mar 31 01:22:33 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Handshake to peer 1 successful: Agreed network protocol version 122
  849. Mar 31 01:22:33 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  850. Mar 31 01:22:33 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Peer authenticated using 20 bytes HMAC
  851. Mar 31 01:22:33 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: my exposed UUID: 28983A7814F88E65
  852. Mar 31 01:22:33 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: peer 752996FB4D06CB50:28D31133FEBC993A:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC bits:0 flags:1120
  853. Mar 31 01:22:33 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 3835743607 (primary_nodes=8)
  854. Mar 31 01:22:33 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  855. Mar 31 01:22:33 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( Off -> Established ) [remote]
  856. Mar 31 01:22:33 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1248691193: 1->3 role( Secondary ) conn( Connected )
  857. Mar 31 01:22:33 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: my exposed UUID: 28983A7814F88E65
  858. Mar 31 01:22:33 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: peer 752996FB4D06CB50:28D31133FEBC993A:ADB250ED33B11982:D8796B7E2A90EF4C bits:0 flags:1120
  859. Mar 31 01:22:33 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 1248691193 (primary_nodes=8)
  860. Mar 31 01:22:33 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  861. Mar 31 01:22:33 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( Off -> Established ) [remote]
  862. Mar 31 01:22:54 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 449856471: 1->all empty
  863. Mar 31 01:22:55 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Aborting remote state change 449856471
  864. Mar 31 01:22:55 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4094485826: 2->all empty
  865. Mar 31 01:22:55 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 4094485826 (primary_nodes=8)
  866. Mar 31 01:22:55 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1321179860: 1->all empty
  867. Mar 31 01:22:55 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 1321179860 (primary_nodes=8)
  868. Mar 31 01:23:21 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4104775491: 0->1 role( Secondary ) conn( Connected )
  869. Mar 31 01:23:21 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 4104775491 (primary_nodes=8)
  870. Mar 31 01:23:21 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 8244816: 0->2 role( Secondary ) conn( Connected )
  871. Mar 31 01:23:21 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Duplicate prepare for remote state change 8244816
  872. Mar 31 01:23:21 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 8244816 (primary_nodes=8)
  873. Mar 31 01:23:31 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2945318694: 1->all empty
  874. Mar 31 01:23:31 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 2945318694 (primary_nodes=8)
  875. Mar 31 01:23:32 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: PingAck did not arrive in time.
  876. Mar 31 01:23:32 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  877. Mar 31 01:23:32 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( Established -> Off )
  878. Mar 31 01:23:32 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2793566032: 3->all empty
  879. Mar 31 01:23:32 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Terminating sender thread
  880. Mar 31 01:23:32 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Starting sender thread (peer-node-id 2)
  881. Mar 31 01:23:32 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Connection closed
  882. Mar 31 01:23:32 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( NetworkFailure -> Unconnected ) [disconnected]
  883. Mar 31 01:23:32 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Restarting receiver thread
  884. Mar 31 01:23:32 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Unconnected -> Connecting ) [connecting]
  885. Mar 31 01:23:32 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 144ms
  886. Mar 31 01:23:32 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 532498931: 0->all empty
  887. Mar 31 01:23:32 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 532498931 (primary_nodes=8)
  888. Mar 31 01:23:32 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 2699842262: 3->all empty
  889. Mar 31 01:23:32 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 2699842262: primary_nodes=8, weak_nodes=0
  890. Mar 31 01:23:32 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 2699842262 (0ms)
  891. Mar 31 01:23:42 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d tcp:storage-load-test-3: Closing unexpected connection from 172.17.1.4
  892. Mar 31 01:23:48 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d tcp:storage-load-test-3: Closing unexpected connection from 172.17.1.4
  893. Mar 31 01:24:01 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d tcp:storage-load-test-3: Closing unexpected connection from 172.17.1.4
  894. Mar 31 01:24:09 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d tcp:storage-load-test-3: Closing unexpected connection from 172.17.1.4
  895. Mar 31 01:24:17 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Handshake to peer 2 successful: Agreed network protocol version 122
  896. Mar 31 01:24:17 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  897. Mar 31 01:24:17 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Peer authenticated using 20 bytes HMAC
  898. Mar 31 01:24:17 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 664435784: 2->3 role( Secondary ) conn( Connected )
  899. Mar 31 01:24:17 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: my exposed UUID: 28983A7814F88E65
  900. Mar 31 01:24:17 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: peer 752996FB4D06CB50:28D31133FEBC993A:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC bits:0 flags:1120
  901. Mar 31 01:24:17 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 664435784 (primary_nodes=8)
  902. Mar 31 01:24:17 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  903. Mar 31 01:24:17 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( Off -> Established ) [remote]
  904. Mar 31 01:24:18 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 426807218: 0->2 role( Secondary ) conn( Connected )
  905. Mar 31 01:24:18 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Duplicate prepare for remote state change 426807218
  906. Mar 31 01:24:18 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 426807218 (primary_nodes=8)
  907. Mar 31 01:24:20 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 2001482756: 1->2 role( Secondary ) conn( Connected )
  908. Mar 31 01:24:20 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 2001482756 (primary_nodes=8)
  909. Mar 31 01:24:30 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1784832376: 0->all empty
  910. Mar 31 01:24:30 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: PingAck did not arrive in time.
  911. Mar 31 01:24:30 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
  912. Mar 31 01:24:30 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( Established -> Off )
  913. Mar 31 01:24:30 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Terminating sender thread
  914. Mar 31 01:24:30 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Starting sender thread (peer-node-id 1)
  915. Mar 31 01:24:30 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Connection closed
  916. Mar 31 01:24:30 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( NetworkFailure -> Unconnected ) [disconnected]
  917. Mar 31 01:24:30 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Restarting receiver thread
  918. Mar 31 01:24:30 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Unconnected -> Connecting ) [connecting]
  919. Mar 31 01:24:31 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Aborting remote state change 1784832376
  920. Mar 31 01:24:31 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 4144099619: 3->all empty
  921. Mar 31 01:24:31 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Aborting local state change 4144099619 to yield to remote state change 4218099027.
  922. Mar 31 01:24:31 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Retrying cluster-wide state change after 260ms
  923. Mar 31 01:24:31 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4218099027: 2->all empty
  924. Mar 31 01:24:31 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 4218099027 (primary_nodes=8)
  925. Mar 31 01:24:31 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1696622909: 0->all empty
  926. Mar 31 01:24:31 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 1696622909 (primary_nodes=8)
  927. Mar 31 01:24:31 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3061440329: 3->all empty
  928. Mar 31 01:24:31 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 3061440329: primary_nodes=8, weak_nodes=0
  929. Mar 31 01:24:31 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 3061440329 (0ms)
  930. Mar 31 01:24:41 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d tcp:storage-load-test-1: Closing unexpected connection from 172.17.1.4
  931. Mar 31 01:24:44 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d tcp:storage-load-test-1: Closing unexpected connection from 172.17.1.4
  932. Mar 31 01:25:00 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d tcp:storage-load-test-1: Closing unexpected connection from 172.17.1.4
  933. Mar 31 01:25:02 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d tcp:storage-load-test-1: Closing unexpected connection from 172.17.1.4
  934. Mar 31 01:25:21 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d tcp:storage-load-test-1: Closing unexpected connection from 172.17.1.4
  935. Mar 31 01:25:22 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 4181019255: 1->2 role( Secondary ) conn( Connected )
  936. Mar 31 01:25:22 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 4181019255 (primary_nodes=8)
  937. Mar 31 01:25:22 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Handshake to peer 1 successful: Agreed network protocol version 122
  938. Mar 31 01:25:22 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  939. Mar 31 01:25:22 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Peer authenticated using 20 bytes HMAC
  940. Mar 31 01:25:22 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 283636919: 1->3 role( Secondary ) conn( Connected )
  941. Mar 31 01:25:22 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: my exposed UUID: 28983A7814F88E65
  942. Mar 31 01:25:22 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: peer 752996FB4D06CB50:28D31133FEBC993A:ADB250ED33B11982:D8796B7E2A90EF4C bits:0 flags:1120
  943. Mar 31 01:25:22 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 283636919 (primary_nodes=8)
  944. Mar 31 01:25:22 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  945. Mar 31 01:25:22 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( Off -> Established ) [remote]
  946. Mar 31 01:25:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 1804534236: 0->1 role( Secondary ) conn( Connected )
  947. Mar 31 01:25:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Duplicate prepare for remote state change 1804534236
  948. Mar 31 01:25:27 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 1804534236 (primary_nodes=8)
  949. Mar 31 01:45:33 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3130963472: 3->1 conn( Disconnecting ) pdsk( Outdated )
  950. Mar 31 01:45:33 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 3130963472: primary_nodes=8, weak_nodes=0
  951. Mar 31 01:45:33 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 3130963472 (0ms)
  952. Mar 31 01:45:33 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connected -> Disconnecting ) peer( Secondary -> Unknown ) [disconnect]
  953. Mar 31 01:45:33 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( Established -> Off ) [disconnect]
  954. Mar 31 01:45:33 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Terminating sender thread
  955. Mar 31 01:45:33 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Starting sender thread (peer-node-id 1)
  956. Mar 31 01:45:33 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Connection closed
  957. Mar 31 01:45:33 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Disconnecting -> StandAlone ) [disconnected]
  958. Mar 31 01:45:33 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Terminating receiver thread
  959. Mar 31 01:45:33 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change failed: Can not disconnect a StandAlone device (-9)
  960. Mar 31 01:45:33 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Failed: conn( StandAlone -> Disconnecting ) [disconnect]
  961. Mar 31 01:45:33 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing cluster-wide state change 3905303763: 3->2 conn( Disconnecting ) pdsk( Outdated )
  962. Mar 31 01:45:33 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: State change 3905303763: primary_nodes=8, weak_nodes=0
  963. Mar 31 01:45:33 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Cluster is now split
  964. Mar 31 01:45:33 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Committing cluster-wide state change 3905303763 (4ms)
  965. Mar 31 01:45:33 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connected -> Disconnecting ) peer( Secondary -> Unknown ) [disconnect]
  966. Mar 31 01:45:33 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( Established -> Off ) [disconnect]
  967. Mar 31 01:45:33 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Terminating sender thread
  968. Mar 31 01:45:33 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Starting sender thread (peer-node-id 2)
  969. Mar 31 01:45:34 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Connection closed
  970. Mar 31 01:45:34 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Disconnecting -> StandAlone ) [disconnected]
  971. Mar 31 01:45:34 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Terminating receiver thread
  972. Mar 31 01:45:39 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( StandAlone -> Unconnected ) [connect]
  973. Mar 31 01:45:39 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Starting receiver thread (peer-node-id 1)
  974. Mar 31 01:45:39 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Unconnected -> Connecting ) [connecting]
  975. Mar 31 01:45:39 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( StandAlone -> Unconnected ) [connect]
  976. Mar 31 01:45:39 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Starting receiver thread (peer-node-id 0)
  977. Mar 31 01:45:39 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Unconnected -> Connecting ) [connecting]
  978. Mar 31 01:45:39 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( StandAlone -> Unconnected ) [connect]
  979. Mar 31 01:45:39 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Starting receiver thread (peer-node-id 2)
  980. Mar 31 01:45:39 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Unconnected -> Connecting ) [connecting]
  981. Mar 31 01:45:40 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Handshake to peer 1 successful: Agreed network protocol version 122
  982. Mar 31 01:45:40 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  983. Mar 31 01:45:40 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Peer authenticated using 20 bytes HMAC
  984. Mar 31 01:45:40 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Handshake to peer 0 successful: Agreed network protocol version 122
  985. Mar 31 01:45:40 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  986. Mar 31 01:45:40 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3269594929: 1->3 role( Secondary ) conn( Connected )
  987. Mar 31 01:45:40 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Peer authenticated using 20 bytes HMAC
  988. Mar 31 01:45:40 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Handshake to peer 2 successful: Agreed network protocol version 122
  989. Mar 31 01:45:40 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
  990. Mar 31 01:45:40 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Peer authenticated using 20 bytes HMAC
  991. Mar 31 01:45:40 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: my exposed UUID: 28983A7814F88E65
  992. Mar 31 01:45:40 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: peer 752996FB4D06CB50:28D31133FEBC993A:ADB250ED33B11982:D8796B7E2A90EF4C bits:0 flags:1120
  993. Mar 31 01:45:40 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: Committing remote state change 3269594929 (primary_nodes=8)
  994. Mar 31 01:45:40 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-1: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  995. Mar 31 01:45:40 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-1: repl( Off -> Established ) [remote]
  996. Mar 31 01:45:40 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 3578569625: 2->3 role( Secondary ) conn( Connected )
  997. Mar 31 01:45:40 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: my exposed UUID: 28983A7814F88E65
  998. Mar 31 01:45:40 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: peer 752996FB4D06CB50:28D31133FEBC993A:DD95BBD64DA6BD62:3B36CC1B2C2CCBAC bits:0 flags:1120
  999. Mar 31 01:45:40 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: Committing remote state change 3578569625 (primary_nodes=8)
  1000. Mar 31 01:45:40 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-3: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  1001. Mar 31 01:45:40 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-3: repl( Off -> Established ) [remote]
  1002. Mar 31 01:45:40 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: Preparing remote state change 775278204: 0->3 role( Secondary ) conn( Connected )
  1003. Mar 31 01:45:40 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: my exposed UUID: 28983A7814F88E65
  1004. Mar 31 01:45:40 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: peer 28983A7814F88E65:28D31133FEBC993B:D8796B7E2A90EF4C:ADB250ED33B11982 bits:0 flags:1120
  1005. Mar 31 01:45:40 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: Committing remote state change 775278204 (primary_nodes=8)
  1006. Mar 31 01:45:40 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d storage-load-test-2: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [remote]
  1007. Mar 31 01:45:40 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002 storage-load-test-2: repl( Off -> Established ) [remote]
  1008. Mar 31 01:46:04 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d: susp-io( no-disk,quorum -> no-disk ) [resource-options]
  1009. Mar 31 01:46:04 storage-load-test-0 kernel: drbd pvc-742a2fd2-a6de-4450-a600-dac7693c758d/0 drbd1002: quorum( no -> yes ) [resource-options]
  1010.  
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!