########## Logs from srv2-1 ########## // Online verification started Feb 16 04:00:01 srv2-1 /USR/SBIN/CRON[1426]: (root) CMD (/sbin/drbdadm verify all >/dev/null 2>&1 #SLAVE) Feb 16 04:00:01 srv2-1 kernel: block drbd1: conn( Connected -> VerifyS ) Feb 16 04:00:01 srv2-1 kernel: block drbd1: Starting Online Verify from sector 0 // Many many many errors during 3 days ! Feb 17 20:31:11 srv2-1 kernel: block drbd1: [drbd1_worker/3083] sock_sendmsg time expired, ko = 4294967295 Feb 17 20:31:17 srv2-1 kernel: block drbd1: [drbd1_worker/3083] sock_sendmsg time expired, ko = 4294967294 (...) Feb 19 19:20:47 srv2-1 kernel: block drbd1: [drbd1_worker/3083] sock_sendmsg time expired, ko = 4294939199 Feb 19 19:20:53 srv2-1 kernel: block drbd1: [drbd1_worker/3083] sock_sendmsg time expired, ko = 4294939198 // Close network communication between srv2-1 and srv2-2 // srv2-2 can now be reached ! // Change srv2-1 from Secondary to Primary Feb 19 19:20:53 srv2-1 kernel: block drbd1: PingAck did not arrive in time. Feb 19 19:20:53 srv2-1 kernel: block drbd1: peer( Primary -> Unknown ) conn( VerifyS -> NetworkFailure ) pdsk( UpToDate -> DUnknown ) Feb 19 19:20:53 srv2-1 kernel: block drbd1: Online Verify reached sector 2983106184 Feb 19 19:20:53 srv2-1 kernel: block drbd1: drbd_pp_alloc interrupted! Feb 19 19:20:53 srv2-1 kernel: block drbd1: error receiving Data, l: 4120! Feb 19 19:20:53 srv2-1 kernel: block drbd1: asender terminated Feb 19 19:20:53 srv2-1 kernel: block drbd1: Terminating asender thread Feb 19 19:20:55 srv2-1 kernel: block drbd1: role( Secondary -> Primary ) Feb 19 19:20:59 srv2-1 kernel: block drbd1: short sent OVRequest size=32 sent=8 Feb 19 19:20:59 srv2-1 kernel: block drbd1: Connection closed Feb 19 19:20:59 srv2-1 kernel: block drbd1: new current UUID 892066CDDBD054B9:78A03E42A1044DAC:1F43CD451DB9AFCA:1F42CD451DB9AFCB Feb 19 19:20:59 srv2-1 kernel: block drbd1: bitmap WRITE of 0 pages took 0 jiffies Feb 19 19:20:59 srv2-1 kernel: block drbd1: conn( NetworkFailure -> Unconnected ) Feb 19 19:20:59 srv2-1 kernel: block drbd1: 0 KB (0 bits) marked out-of-sync by on disk bit-map. Feb 19 19:20:59 srv2-1 kernel: block drbd1: receiver terminated Feb 19 19:20:59 srv2-1 kernel: block drbd1: Restarting receiver thread Feb 19 19:20:59 srv2-1 kernel: block drbd1: receiver (re)started Feb 19 19:20:59 srv2-1 kernel: block drbd1: conn( Unconnected -> WFConnection ) // Reboot of srv2-2 // Open network communication between srv2-1 and srv2-2 Feb 19 19:34:29 srv2-1 kernel: block drbd1: Handshake successful: Agreed network protocol version 97 Feb 19 19:34:29 srv2-1 kernel: block drbd1: conn( WFConnection -> WFReportParams ) Feb 19 19:34:29 srv2-1 kernel: block drbd1: Starting asender thread (from drbd1_receiver [3093]) Feb 19 19:34:29 srv2-1 kernel: block drbd1: data-integrity-alg: Feb 19 19:34:29 srv2-1 kernel: block drbd1: drbd_sync_handshake: Feb 19 19:34:29 srv2-1 kernel: block drbd1: self 892066CDDBD054B9:78A03E42A1044DAC:1F43CD451DB9AFCA:1F42CD451DB9AFCB bits:38627 flags:0 Feb 19 19:34:29 srv2-1 kernel: block drbd1: peer 7033A3893BFC813E:78A03E42A1044DAD:1F43CD451DB9AFCB:1F42CD451DB9AFCB bits:160811 flags:2 Feb 19 19:34:29 srv2-1 kernel: block drbd1: uuid_compare()=100 by rule 90 Feb 19 19:34:29 srv2-1 kernel: block drbd1: helper command: /sbin/drbdadm initial-split-brain minor-1 Feb 19 19:34:29 srv2-1 kernel: block drbd1: helper command: /sbin/drbdadm initial-split-brain minor-1 exit code 0 (0x0) Feb 19 19:34:29 srv2-1 kernel: block drbd1: Split-Brain detected, 1 primaries, automatically solved. Sync from this node Feb 19 19:34:29 srv2-1 kernel: block drbd1: peer( Unknown -> Secondary ) conn( WFReportParams -> WFBitMapS ) pdsk( DUnknown -> Consistent ) Feb 19 19:34:42 srv2-1 kernel: block drbd1: helper command: /sbin/drbdadm before-resync-source minor-1 Feb 19 19:34:42 srv2-1 kernel: block drbd1: helper command: /sbin/drbdadm before-resync-source minor-1 exit code 0 (0x0) Feb 19 19:34:42 srv2-1 kernel: block drbd1: conn( WFBitMapS -> SyncSource ) pdsk( Consistent -> Inconsistent ) Feb 19 19:34:42 srv2-1 kernel: block drbd1: Began resync as SyncSource (will sync 786200 KB [196550 bits set]). Feb 19 19:34:42 srv2-1 kernel: block drbd1: updated sync UUID 892066CDDBD054B9:78A13E42A1044DAC:78A03E42A1044DAC:1F43CD451DB9AFCA Feb 19 19:35:59 srv2-1 kernel: block drbd1: Resync done (total 77 sec; paused 0 sec; 10208 K/sec) Feb 19 19:35:59 srv2-1 kernel: block drbd1: updated UUIDs 892066CDDBD054B9:0000000000000000:78A13E42A1044DAC:78A03E42A1044DAC Feb 19 19:35:59 srv2-1 kernel: block drbd1: conn( SyncSource -> Connected ) pdsk( Inconsistent -> UpToDate ) Feb 19 19:36:01 srv2-1 kernel: block drbd1: bitmap WRITE of 14340 pages took 484 jiffies Feb 19 19:36:01 srv2-1 kernel: block drbd1: 0 KB (0 bits) marked out-of-sync by on disk bit-map.