Apr 30 13:26:44 athos kernel: DLM (built Mar 24 2009 12:06:38) installed Apr 30 13:26:44 athos kernel: GFS2 Overlay (built May 29 2008 16:48:00) installed Apr 30 13:26:44 athos kernel: Lock_DLM (built Mar 24 2009 12:07:36) installed Apr 30 13:26:44 athos ccsd[8558]: Starting ccsd 2.0.98: Apr 30 13:26:44 athos ccsd[8558]: Built: Mar 5 2009 16:16:06 Apr 30 13:26:44 athos ccsd[8558]: Copyright (C) Red Hat, Inc. 2004 All rights reserved. Apr 30 13:26:44 athos ccsd[8558]: cluster.conf (cluster name = MUSKETEER, version = 18) found. Apr 30 13:26:46 athos openais[8569]: [MAIN ] AIS Executive Service RELEASE 'subrev 1358 version 0.80.3' Apr 30 13:26:46 athos openais[8569]: [MAIN ] Copyright (C) 2002-2006 MontaVista Software, Inc and contributors. Apr 30 13:26:46 athos openais[8569]: [MAIN ] Copyright (C) 2006 Red Hat, Inc. Apr 30 13:26:46 athos openais[8569]: [MAIN ] AIS Executive Service: started and ready to provide service. Apr 30 13:26:46 athos openais[8569]: [MAIN ] Using default multicast address of 239.192.158.31 Apr 30 13:26:46 athos openais[8569]: [MAIN ] openais component openais_cpg loaded. Apr 30 13:26:46 athos openais[8569]: [MAIN ] Registering service handler 'openais cluster closed process group service v1.01' Apr 30 13:26:46 athos openais[8569]: [MAIN ] openais component openais_cfg loaded. Apr 30 13:26:46 athos openais[8569]: [MAIN ] Registering service handler 'openais configuration service' Apr 30 13:26:46 athos openais[8569]: [MAIN ] openais component openais_msg loaded. Apr 30 13:26:46 athos openais[8569]: [MAIN ] Registering service handler 'openais message service B.01.01' Apr 30 13:26:46 athos openais[8569]: [MAIN ] openais component openais_lck loaded. Apr 30 13:26:46 athos openais[8569]: [MAIN ] Registering service handler 'openais distributed locking service B.01.01' Apr 30 13:26:46 athos openais[8569]: [MAIN ] openais component openais_evt loaded. Apr 30 13:26:46 athos openais[8569]: [MAIN ] Registering service handler 'openais event service B.01.01' Apr 30 13:26:46 athos openais[8569]: [MAIN ] openais component openais_ckpt loaded. Apr 30 13:26:46 athos openais[8569]: [MAIN ] Registering service handler 'openais checkpoint service B.01.01' Apr 30 13:26:46 athos openais[8569]: [MAIN ] openais component openais_amf loaded. Apr 30 13:26:46 athos openais[8569]: [MAIN ] Registering service handler 'openais availability management framework B.01.01' Apr 30 13:26:46 athos openais[8569]: [MAIN ] openais component openais_clm loaded. Apr 30 13:26:46 athos openais[8569]: [MAIN ] Registering service handler 'openais cluster membership service B.01.01' Apr 30 13:26:46 athos openais[8569]: [MAIN ] openais component openais_evs loaded. Apr 30 13:26:46 athos openais[8569]: [MAIN ] Registering service handler 'openais extended virtual synchrony service' Apr 30 13:26:46 athos openais[8569]: [MAIN ] openais component openais_cman loaded. Apr 30 13:26:46 athos openais[8569]: [MAIN ] Registering service handler 'openais CMAN membership service 2.01' Apr 30 13:26:46 athos openais[8569]: [TOTEM] Token Timeout (10000 ms) retransmit timeout (495 ms) Apr 30 13:26:46 athos openais[8569]: [TOTEM] token hold (386 ms) retransmits before loss (20 retrans) Apr 30 13:26:46 athos openais[8569]: [TOTEM] join (60 ms) send_join (0 ms) consensus (4800 ms) merge (200 ms) Apr 30 13:26:46 athos openais[8569]: [TOTEM] downcheck (1000 ms) fail to recv const (50 msgs) Apr 30 13:26:46 athos openais[8569]: [TOTEM] seqno unchanged const (30 rotations) Maximum network MTU 1500 Apr 30 13:26:46 athos openais[8569]: [TOTEM] window size per rotation (50 messages) maximum messages per rotation (17 messages) Apr 30 13:26:46 athos openais[8569]: [TOTEM] send threads (0 threads) Apr 30 13:26:46 athos openais[8569]: [TOTEM] RRP token expired timeout (495 ms) Apr 30 13:26:46 athos openais[8569]: [TOTEM] RRP token problem counter (2000 ms) Apr 30 13:26:46 athos openais[8569]: [TOTEM] RRP threshold (10 problem count) Apr 30 13:26:46 athos openais[8569]: [TOTEM] RRP mode set to none. Apr 30 13:26:46 athos openais[8569]: [TOTEM] heartbeat_failures_allowed (0) Apr 30 13:26:46 athos openais[8569]: [TOTEM] max_network_delay (50 ms) Apr 30 13:26:46 athos openais[8569]: [TOTEM] HeartBeat is Disabled. To enable set heartbeat_failures_allowed > 0 Apr 30 13:26:46 athos openais[8569]: [TOTEM] Receive multicast socket recv buffer size (262142 bytes). Apr 30 13:26:46 athos openais[8569]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). Apr 30 13:26:46 athos openais[8569]: [TOTEM] The network interface [172.16.1.1] is now up. Apr 30 13:26:46 athos openais[8569]: [TOTEM] Created or loaded sequence id 9984.172.16.1.1 for this ring. Apr 30 13:26:46 athos openais[8569]: [TOTEM] entering GATHER state from 15. Apr 30 13:26:46 athos openais[8569]: [SERV ] Initialising service handler 'openais extended virtual synchrony service' Apr 30 13:26:46 athos openais[8569]: [SERV ] Initialising service handler 'openais cluster membership service B.01.01' Apr 30 13:26:46 athos openais[8569]: [SERV ] Initialising service handler 'openais availability management framework B.01.01' Apr 30 13:26:46 athos openais[8569]: [SERV ] Initialising service handler 'openais checkpoint service B.01.01' Apr 30 13:26:46 athos openais[8569]: [SERV ] Initialising service handler 'openais event service B.01.01' Apr 30 13:26:46 athos openais[8569]: [SERV ] Initialising service handler 'openais distributed locking service B.01.01' Apr 30 13:26:46 athos ccsd[8558]: Initial status:: Inquorate Apr 30 13:26:46 athos openais[8569]: [SERV ] Initialising service handler 'openais message service B.01.01' Apr 30 13:26:46 athos openais[8569]: [SERV ] Initialising service handler 'openais configuration service' Apr 30 13:26:46 athos openais[8569]: [SERV ] Initialising service handler 'openais cluster closed process group service v1.01' Apr 30 13:26:46 athos openais[8569]: [SERV ] Initialising service handler 'openais CMAN membership service 2.01' Apr 30 13:26:46 athos openais[8569]: [CMAN ] CMAN 2.0.98 (built Mar 5 2009 16:16:09) started Apr 30 13:26:46 athos openais[8569]: [SYNC ] Not using a virtual synchrony filter. Apr 30 13:26:46 athos openais[8569]: [TOTEM] Creating commit token because I am the rep. Apr 30 13:26:46 athos openais[8569]: [TOTEM] Saving state aru 0 high seq received 0 Apr 30 13:26:46 athos openais[8569]: [TOTEM] Storing new sequence id for ring 2704 Apr 30 13:26:46 athos openais[8569]: [TOTEM] entering COMMIT state. Apr 30 13:26:46 athos openais[8569]: [TOTEM] entering RECOVERY state. Apr 30 13:26:46 athos openais[8569]: [TOTEM] position [0] member 172.16.1.1: Apr 30 13:26:46 athos openais[8569]: [TOTEM] previous ring seq 9984 rep 172.16.1.1 Apr 30 13:26:46 athos openais[8569]: [TOTEM] aru 0 high delivered 0 received flag 1 Apr 30 13:26:46 athos openais[8569]: [TOTEM] Did not need to originate any messages in recovery. Apr 30 13:26:46 athos openais[8569]: [TOTEM] Sending initial ORF token Apr 30 13:26:46 athos openais[8569]: [CLM ] CLM CONFIGURATION CHANGE Apr 30 13:26:46 athos openais[8569]: [CLM ] New Configuration: Apr 30 13:26:46 athos openais[8569]: [CLM ] Members Left: Apr 30 13:26:46 athos openais[8569]: [CLM ] Members Joined: Apr 30 13:26:46 athos openais[8569]: [CLM ] CLM CONFIGURATION CHANGE Apr 30 13:26:46 athos openais[8569]: [CLM ] New Configuration: Apr 30 13:26:46 athos openais[8569]: [CLM ] r(0) ip(172.16.1.1) Apr 30 13:26:46 athos openais[8569]: [CLM ] Members Left: Apr 30 13:26:46 athos openais[8569]: [CLM ] Members Joined: Apr 30 13:26:46 athos openais[8569]: [CLM ] r(0) ip(172.16.1.1) Apr 30 13:26:46 athos openais[8569]: [SYNC ] This node is within the primary component and will provide service. Apr 30 13:26:46 athos openais[8569]: [TOTEM] entering OPERATIONAL state. Apr 30 13:26:46 athos openais[8569]: [CLM ] got nodejoin message 172.16.1.1 Apr 30 13:26:47 athos ccsd[8558]: Cluster is not quorate. Refusing connection. Apr 30 13:26:47 athos ccsd[8558]: Error while processing connect: Connection refused Apr 30 13:26:47 athos ccsd[8558]: Cluster is not quorate. Refusing connection. Apr 30 13:26:47 athos ccsd[8558]: Error while processing connect: Connection refused Apr 30 13:26:48 athos ccsd[8558]: Cluster is not quorate. Refusing connection. Apr 30 13:26:48 athos ccsd[8558]: Error while processing connect: Connection refused Apr 30 13:26:48 athos ccsd[8558]: Cluster is not quorate. Refusing connection. Apr 30 13:26:48 athos ccsd[8558]: Error while processing connect: Connection refused Apr 30 13:26:48 athos ccsd[8558]: Cluster is not quorate. Refusing connection. Apr 30 13:26:48 athos ccsd[8558]: Error while processing connect: Connection refused Apr 30 13:26:49 athos ccsd[8558]: Cluster is not quorate. Refusing connection. Apr 30 13:26:49 athos ccsd[8558]: Error while processing connect: Connection refused Apr 30 13:26:49 athos ccsd[8558]: Cluster is not quorate. Refusing connection. Apr 30 13:26:49 athos ccsd[8558]: Error while processing connect: Connection refused Apr 30 13:26:50 athos qdiskd[5311]: Quorum Daemon Initializing Apr 30 13:26:50 athos ccsd[8558]: Cluster is not quorate. Refusing connection. Apr 30 13:26:50 athos ccsd[8558]: Error while processing connect: Connection refused Apr 30 13:26:50 athos ccsd[8558]: Cluster is not quorate. Refusing connection. Apr 30 13:26:50 athos ccsd[8558]: Error while processing connect: Connection refused Apr 30 13:26:51 athos ccsd[8558]: Cluster is not quorate. Refusing connection. Apr 30 13:26:51 athos ccsd[8558]: Error while processing connect: Connection refused Apr 30 13:26:51 athos ccsd[8558]: Cluster is not quorate. Refusing connection. Apr 30 13:26:51 athos ccsd[8558]: Error while processing connect: Connection refused Apr 30 13:26:52 athos ccsd[8558]: Cluster is not quorate. Refusing connection. Apr 30 13:26:52 athos ccsd[8558]: Error while processing connect: Connection refused Apr 30 13:26:52 athos ccsd[8558]: Cluster is not quorate. Refusing connection. Apr 30 13:26:52 athos ccsd[8558]: Error while processing connect: Connection refused Apr 30 13:26:53 athos ccsd[8558]: Cluster is not quorate. Refusing connection. Apr 30 13:26:53 athos ccsd[8558]: Error while processing connect: Connection refused Apr 30 13:26:53 athos ccsd[8558]: Cluster is not quorate. Refusing connection. Apr 30 13:26:53 athos ccsd[8558]: Error while processing connect: Connection refused Apr 30 13:26:53 athos ccsd[8558]: Cluster is not quorate. Refusing connection. Apr 30 13:26:53 athos ccsd[8558]: Error while processing connect: Connection refused Apr 30 13:26:54 athos ccsd[8558]: Cluster is not quorate. Refusing connection. Apr 30 13:26:54 athos ccsd[8558]: Error while processing connect: Connection refused Apr 30 13:26:54 athos ccsd[8558]: Cluster is not quorate. Refusing connection. Apr 30 13:26:54 athos ccsd[8558]: Error while processing connect: Connection refused Apr 30 13:26:55 athos ccsd[8558]: Cluster is not quorate. Refusing connection. Apr 30 13:26:55 athos ccsd[8558]: Error while processing connect: Connection refused Apr 30 13:26:55 athos ccsd[8558]: Cluster is not quorate. Refusing connection. Apr 30 13:26:55 athos ccsd[8558]: Error while processing connect: Connection refused Apr 30 13:26:56 athos ccsd[8558]: Cluster is not quorate. Refusing connection. Apr 30 13:26:56 athos ccsd[8558]: Error while processing connect: Connection refused Apr 30 13:26:56 athos ccsd[8558]: Cluster is not quorate. Refusing connection. Apr 30 13:26:56 athos ccsd[8558]: Error while processing connect: Connection refused Apr 30 13:26:57 athos dlm_controld[8617]: connect to ccs error -111, check ccsd or cluster status Apr 30 13:26:57 athos ccsd[8558]: Cluster is not quorate. Refusing connection. Apr 30 13:26:57 athos ccsd[8558]: Error while processing connect: Connection refused Apr 30 13:26:57 athos gfs_controld[8623]: connect to ccs error -111, check ccsd or cluster status Apr 30 13:26:57 athos ccsd[8558]: Cluster is not quorate. Refusing connection. Apr 30 13:26:57 athos ccsd[8558]: Error while processing connect: Connection refused Apr 30 13:26:58 athos ccsd[8558]: Cluster is not quorate. Refusing connection. Apr 30 13:26:58 athos ccsd[8558]: Error while processing connect: Connection refused Apr 30 13:26:58 athos ccsd[8558]: Cluster is not quorate. Refusing connection. Apr 30 13:26:58 athos ccsd[8558]: Error while processing connect: Connection refused Apr 30 13:26:58 athos ccsd[8558]: Cluster is not quorate. Refusing connection. Apr 30 13:26:58 athos ccsd[8558]: Error while processing connect: Connection refused Apr 30 13:26:59 athos ccsd[8558]: Cluster is not quorate. Refusing connection. Apr 30 13:26:59 athos ccsd[8558]: Error while processing connect: Connection refused Apr 30 13:26:59 athos ccsd[8558]: Cluster is not quorate. Refusing connection. Apr 30 13:26:59 athos ccsd[8558]: Error while processing connect: Connection refused Apr 30 13:27:00 athos qdiskd[5311]: Initial score 1/1 Apr 30 13:27:00 athos qdiskd[5311]: Initialization complete Apr 30 13:27:00 athos openais[8569]: [CMAN ] quorum device registered Apr 30 13:27:00 athos qdiskd[5311]: Score sufficient for master operation (1/1; required=1); upgrading Apr 30 13:27:00 athos ccsd[8558]: Cluster is not quorate. Refusing connection. Apr 30 13:27:00 athos ccsd[8558]: Error while processing connect: Connection refused Apr 30 13:27:00 athos ccsd[8558]: Cluster is not quorate. Refusing connection. Apr 30 13:27:00 athos ccsd[8558]: Error while processing connect: Connection refused Apr 30 13:27:01 athos ccsd[8558]: Cluster is not quorate. Refusing connection. Apr 30 13:27:01 athos ccsd[8558]: Error while processing connect: Connection refused Apr 30 13:27:01 athos ccsd[8558]: Cluster is not quorate. Refusing connection. Apr 30 13:27:01 athos ccsd[8558]: Error while processing connect: Connection refused Apr 30 13:27:02 athos ccsd[8558]: Cluster is not quorate. Refusing connection. Apr 30 13:27:02 athos ccsd[8558]: Error while processing connect: Connection refused Apr 30 13:27:02 athos ccsd[8558]: Cluster is not quorate. Refusing connection. Apr 30 13:27:02 athos ccsd[8558]: Error while processing connect: Connection refused Apr 30 13:27:03 athos ccsd[8558]: Cluster is not quorate. Refusing connection. Apr 30 13:27:03 athos ccsd[8558]: Error while processing connect: Connection refused Apr 30 13:27:03 athos ccsd[8558]: Cluster is not quorate. Refusing connection. Apr 30 13:27:03 athos ccsd[8558]: Error while processing connect: Connection refused Apr 30 13:27:03 athos ccsd[8558]: Cluster is not quorate. Refusing connection. Apr 30 13:27:03 athos ccsd[8558]: Error while processing connect: Connection refused Apr 30 13:27:04 athos ccsd[8558]: Cluster is not quorate. Refusing connection. Apr 30 13:27:04 athos ccsd[8558]: Error while processing connect: Connection refused Apr 30 13:27:04 athos ccsd[8558]: Cluster is not quorate. Refusing connection. Apr 30 13:27:04 athos ccsd[8558]: Error while processing connect: Connection refused Apr 30 13:27:05 athos ccsd[8558]: Cluster is not quorate. Refusing connection. Apr 30 13:27:05 athos ccsd[8558]: Error while processing connect: Connection refused Apr 30 13:27:05 athos ccsd[8558]: Cluster is not quorate. Refusing connection. Apr 30 13:27:05 athos ccsd[8558]: Error while processing connect: Connection refused Apr 30 13:27:06 athos qdiskd[5311]: Assuming master role Apr 30 13:27:06 athos ccsd[8558]: Cluster is not quorate. Refusing connection. Apr 30 13:27:06 athos ccsd[8558]: Error while processing connect: Connection refused Apr 30 13:27:06 athos ccsd[8558]: Cluster is not quorate. Refusing connection. Apr 30 13:27:06 athos ccsd[8558]: Error while processing connect: Connection refused Apr 30 13:27:07 athos qdiskd[5311]: Writing eviction notice for node 3 Apr 30 13:27:07 athos openais[8569]: [CMAN ] quorum regained, resuming activity Apr 30 13:27:07 athos dlm_controld[8617]: connect to ccs error -111, check ccsd or cluster status Apr 30 13:27:07 athos gfs_controld[8623]: connect to ccs error -111, check ccsd or cluster status Apr 30 13:27:08 athos qdiskd[5311]: Node 3 evicted Apr 30 13:28:00 athos openais[8569]: [TOTEM] entering GATHER state from 11. Apr 30 13:28:00 athos openais[8569]: [TOTEM] Creating commit token because I am the rep. Apr 30 13:28:00 athos openais[8569]: [TOTEM] Saving state aru f high seq received f Apr 30 13:28:00 athos openais[8569]: [TOTEM] Storing new sequence id for ring 2708 Apr 30 13:28:00 athos openais[8569]: [TOTEM] entering COMMIT state. Apr 30 13:28:00 athos openais[8569]: [TOTEM] entering RECOVERY state. Apr 30 13:28:00 athos openais[8569]: [TOTEM] position [0] member 172.16.1.1: Apr 30 13:28:00 athos openais[8569]: [TOTEM] previous ring seq 9988 rep 172.16.1.1 Apr 30 13:28:00 athos openais[8569]: [TOTEM] aru f high delivered f received flag 1 Apr 30 13:28:00 athos openais[8569]: [TOTEM] position [1] member 172.16.1.2: Apr 30 13:28:00 athos openais[8569]: [TOTEM] previous ring seq 9968 rep 172.16.1.2 Apr 30 13:28:00 athos openais[8569]: [TOTEM] aru a high delivered a received flag 1 Apr 30 13:28:00 athos openais[8569]: [TOTEM] Did not need to originate any messages in recovery. Apr 30 13:28:00 athos openais[8569]: [TOTEM] Sending initial ORF token Apr 30 13:28:00 athos openais[8569]: [CLM ] CLM CONFIGURATION CHANGE Apr 30 13:28:00 athos openais[8569]: [CLM ] New Configuration: Apr 30 13:28:00 athos openais[8569]: [CLM ] r(0) ip(172.16.1.1) Apr 30 13:28:00 athos openais[8569]: [CLM ] Members Left: Apr 30 13:28:00 athos openais[8569]: [CLM ] Members Joined: Apr 30 13:28:00 athos openais[8569]: [CLM ] CLM CONFIGURATION CHANGE Apr 30 13:28:00 athos openais[8569]: [CLM ] New Configuration: Apr 30 13:28:00 athos openais[8569]: [CLM ] r(0) ip(172.16.1.1) Apr 30 13:28:00 athos openais[8569]: [CLM ] r(0) ip(172.16.1.2) Apr 30 13:28:00 athos openais[8569]: [CLM ] Members Left: Apr 30 13:28:00 athos openais[8569]: [CLM ] Members Joined: Apr 30 13:28:00 athos openais[8569]: [CLM ] r(0) ip(172.16.1.2) Apr 30 13:28:00 athos openais[8569]: [SYNC ] This node is within the primary component and will provide service. Apr 30 13:28:00 athos openais[8569]: [TOTEM] entering OPERATIONAL state. Apr 30 13:28:00 athos openais[8569]: [CLM ] got nodejoin message 172.16.1.1 Apr 30 13:28:00 athos openais[8569]: [CLM ] got nodejoin message 172.16.1.2 Apr 30 13:28:00 athos openais[8569]: [CPG ] got joinlist message from node 1 Apr 30 13:28:52 athos openais[8569]: [TOTEM] entering GATHER state from 11. Apr 30 13:28:52 athos openais[8569]: [TOTEM] Creating commit token because I am the rep. Apr 30 13:28:52 athos openais[8569]: [TOTEM] Saving state aru 1f high seq received 1f Apr 30 13:28:52 athos openais[8569]: [TOTEM] Storing new sequence id for ring 270c Apr 30 13:28:52 athos openais[8569]: [TOTEM] entering COMMIT state. Apr 30 13:28:52 athos openais[8569]: [TOTEM] entering RECOVERY state. Apr 30 13:28:52 athos openais[8569]: [TOTEM] position [0] member 172.16.1.1: Apr 30 13:28:52 athos openais[8569]: [TOTEM] previous ring seq 9992 rep 172.16.1.1 Apr 30 13:28:52 athos openais[8569]: [TOTEM] aru 1f high delivered 1f received flag 1 Apr 30 13:28:52 athos openais[8569]: [TOTEM] position [1] member 172.16.1.2: Apr 30 13:28:52 athos openais[8569]: [TOTEM] previous ring seq 9992 rep 172.16.1.1 Apr 30 13:28:52 athos openais[8569]: [TOTEM] aru 1f high delivered 1f received flag 1 Apr 30 13:28:52 athos openais[8569]: [TOTEM] position [2] member 172.16.1.3: Apr 30 13:28:52 athos openais[8569]: [TOTEM] previous ring seq 9992 rep 172.16.1.3 Apr 30 13:28:52 athos openais[8569]: [TOTEM] aru 0 high delivered 0 received flag 1 Apr 30 13:28:52 athos openais[8569]: [TOTEM] Did not need to originate any messages in recovery. Apr 30 13:28:52 athos openais[8569]: [TOTEM] Sending initial ORF token Apr 30 13:28:52 athos openais[8569]: [CLM ] CLM CONFIGURATION CHANGE Apr 30 13:28:52 athos openais[8569]: [CLM ] New Configuration: Apr 30 13:28:52 athos openais[8569]: [CLM ] r(0) ip(172.16.1.1) Apr 30 13:28:52 athos openais[8569]: [CLM ] r(0) ip(172.16.1.2) Apr 30 13:28:52 athos openais[8569]: [CLM ] Members Left: Apr 30 13:28:52 athos openais[8569]: [CLM ] Members Joined: Apr 30 13:28:52 athos openais[8569]: [CLM ] CLM CONFIGURATION CHANGE Apr 30 13:28:52 athos openais[8569]: [CLM ] New Configuration: Apr 30 13:28:52 athos openais[8569]: [CLM ] r(0) ip(172.16.1.1) Apr 30 13:28:52 athos openais[8569]: [CLM ] r(0) ip(172.16.1.2) Apr 30 13:28:52 athos openais[8569]: [CLM ] r(0) ip(172.16.1.3) Apr 30 13:28:52 athos openais[8569]: [CLM ] Members Left: Apr 30 13:28:52 athos openais[8569]: [CLM ] Members Joined: Apr 30 13:28:52 athos openais[8569]: [CLM ] r(0) ip(172.16.1.3) Apr 30 13:28:52 athos openais[8569]: [SYNC ] This node is within the primary component and will provide service. Apr 30 13:28:52 athos openais[8569]: [TOTEM] entering OPERATIONAL state. Apr 30 13:28:52 athos openais[8569]: [CLM ] got nodejoin message 172.16.1.1 Apr 30 13:28:52 athos openais[8569]: [CLM ] got nodejoin message 172.16.1.2 Apr 30 13:28:52 athos openais[8569]: [CLM ] got nodejoin message 172.16.1.3 Apr 30 13:28:52 athos openais[8569]: [TOTEM] Retransmit List: 15 Apr 30 13:28:52 athos last message repeated 26 times Apr 30 13:28:52 athos openais[8569]: [TOTEM] Retransmit List: 17 18 Apr 30 13:28:52 athos openais[8569]: [CPG ] got joinlist message from node 1 Apr 30 13:28:52 athos openais[8569]: [CPG ] got joinlist message from node 2 Apr 30 13:29:00 athos openais[8569]: [TOTEM] entering GATHER state from 12. Apr 30 13:29:00 athos openais[8569]: [TOTEM] Creating commit token because I am the rep. Apr 30 13:29:00 athos openais[8569]: [TOTEM] Saving state aru 21 high seq received 21 Apr 30 13:29:00 athos openais[8569]: [TOTEM] Storing new sequence id for ring 2710 Apr 30 13:29:00 athos openais[8569]: [TOTEM] entering COMMIT state. Apr 30 13:29:00 athos openais[8569]: [TOTEM] entering RECOVERY state. Apr 30 13:29:00 athos openais[8569]: [TOTEM] position [0] member 172.16.1.1: Apr 30 13:29:00 athos openais[8569]: [TOTEM] previous ring seq 9996 rep 172.16.1.1 Apr 30 13:29:00 athos openais[8569]: [TOTEM] aru 21 high delivered 21 received flag 1 Apr 30 13:29:00 athos openais[8569]: [TOTEM] position [1] member 172.16.1.2: Apr 30 13:29:00 athos openais[8569]: [TOTEM] previous ring seq 9996 rep 172.16.1.1 Apr 30 13:29:00 athos openais[8569]: [TOTEM] aru 21 high delivered 21 received flag 1 Apr 30 13:29:00 athos openais[8569]: [TOTEM] position [2] member 172.16.1.3: Apr 30 13:29:00 athos openais[8569]: [TOTEM] previous ring seq 9996 rep 172.16.1.1 Apr 30 13:29:00 athos openais[8569]: [TOTEM] aru 20 high delivered 20 received flag 0 Apr 30 13:29:00 athos openais[8569]: [TOTEM] copying all old ring messages from 21-21. Apr 30 13:29:00 athos openais[8569]: [TOTEM] Originated 1 messages in RECOVERY. Apr 30 13:29:00 athos openais[8569]: [TOTEM] Originated for recovery: 21 Apr 30 13:29:00 athos openais[8569]: [TOTEM] Not Originated for recovery: Apr 30 13:29:00 athos openais[8569]: [TOTEM] Sending initial ORF token Apr 30 13:29:00 athos openais[8569]: [CLM ] CLM CONFIGURATION CHANGE Apr 30 13:29:00 athos openais[8569]: [CLM ] New Configuration: Apr 30 13:29:00 athos openais[8569]: [CLM ] r(0) ip(172.16.1.1) Apr 30 13:29:00 athos openais[8569]: [CLM ] r(0) ip(172.16.1.2) Apr 30 13:29:00 athos openais[8569]: [CLM ] r(0) ip(172.16.1.3) Apr 30 13:29:00 athos openais[8569]: [CLM ] Members Left: Apr 30 13:29:00 athos openais[8569]: [CLM ] Members Joined: Apr 30 13:29:00 athos openais[8569]: [CLM ] CLM CONFIGURATION CHANGE Apr 30 13:29:00 athos openais[8569]: [CLM ] New Configuration: Apr 30 13:29:00 athos openais[8569]: [CLM ] r(0) ip(172.16.1.1) Apr 30 13:29:00 athos openais[8569]: [CLM ] r(0) ip(172.16.1.2) Apr 30 13:29:00 athos openais[8569]: [CLM ] r(0) ip(172.16.1.3) Apr 30 13:29:00 athos openais[8569]: [CLM ] Members Left: Apr 30 13:29:00 athos openais[8569]: [CLM ] Members Joined: Apr 30 13:29:00 athos openais[8569]: [SYNC ] This node is within the primary component and will provide service. Apr 30 13:29:00 athos openais[8569]: [TOTEM] entering OPERATIONAL state. Apr 30 13:29:00 athos openais[8569]: [CLM ] got nodejoin message 172.16.1.3 Apr 30 13:29:00 athos openais[8569]: [CLM ] got nodejoin message 172.16.1.1 Apr 30 13:29:00 athos openais[8569]: [CLM ] got nodejoin message 172.16.1.2 Apr 30 13:29:00 athos openais[8569]: [TOTEM] Retransmit List: b Apr 30 13:29:00 athos openais[8569]: [TOTEM] Retransmit List: 16 Apr 30 13:29:00 athos last message repeated 24 times Apr 30 13:29:17 athos openais[8569]: [TOTEM] FAILED TO RECEIVE Apr 30 13:29:17 athos openais[8569]: [TOTEM] entering GATHER state from 6. Apr 30 13:29:17 athos openais[8569]: [TOTEM] Creating commit token because I am the rep. Apr 30 13:29:17 athos openais[8569]: [TOTEM] Saving state aru 1a high seq received 1a Apr 30 13:29:17 athos openais[8569]: [TOTEM] Storing new sequence id for ring 2714 Apr 30 13:29:17 athos openais[8569]: [TOTEM] entering COMMIT state. Apr 30 13:29:17 athos openais[8569]: [TOTEM] entering RECOVERY state. Apr 30 13:29:17 athos openais[8569]: [TOTEM] position [0] member 172.16.1.1: Apr 30 13:29:17 athos openais[8569]: [TOTEM] previous ring seq 10000 rep 172.16.1.1 Apr 30 13:29:17 athos openais[8569]: [TOTEM] aru 1a high delivered 1a received flag 1 Apr 30 13:29:17 athos openais[8569]: [TOTEM] position [1] member 172.16.1.2: Apr 30 13:29:17 athos openais[8569]: [TOTEM] previous ring seq 10000 rep 172.16.1.1 Apr 30 13:29:17 athos openais[8569]: [TOTEM] aru 1a high delivered 1a received flag 1 Apr 30 13:29:17 athos openais[8569]: [TOTEM] position [2] member 172.16.1.3: Apr 30 13:29:17 athos openais[8569]: [TOTEM] previous ring seq 10000 rep 172.16.1.1 Apr 30 13:29:17 athos openais[8569]: [TOTEM] aru 17 high delivered 17 received flag 0 Apr 30 13:29:17 athos openais[8569]: [TOTEM] copying all old ring messages from 18-1a. Apr 30 13:29:17 athos openais[8569]: [TOTEM] Originated 3 messages in RECOVERY. Apr 30 13:29:17 athos openais[8569]: [TOTEM] Originated for recovery: 18 19 1a Apr 30 13:29:17 athos openais[8569]: [TOTEM] Not Originated for recovery: Apr 30 13:29:17 athos openais[8569]: [TOTEM] Sending initial ORF token Apr 30 13:29:25 athos openais[8569]: [TOTEM] FAILED TO RECEIVE Apr 30 13:29:25 athos openais[8569]: [TOTEM] Restoring instance->my_aru 1a my high seq received 1a Apr 30 13:29:25 athos openais[8569]: [TOTEM] entering GATHER state from 6. Apr 30 13:29:25 athos openais[8569]: [TOTEM] Creating commit token because I am the rep. Apr 30 13:29:25 athos openais[8569]: [TOTEM] Storing new sequence id for ring 2718 Apr 30 13:29:25 athos openais[8569]: [TOTEM] entering COMMIT state. Apr 30 13:29:25 athos openais[8569]: [TOTEM] entering RECOVERY state. Apr 30 13:29:25 athos openais[8569]: [TOTEM] position [0] member 172.16.1.1: Apr 30 13:29:25 athos openais[8569]: [TOTEM] previous ring seq 10000 rep 172.16.1.1 Apr 30 13:29:25 athos openais[8569]: [TOTEM] aru 1a high delivered 1a received flag 1 Apr 30 13:29:25 athos openais[8569]: [TOTEM] position [1] member 172.16.1.2: Apr 30 13:29:25 athos openais[8569]: [TOTEM] previous ring seq 10000 rep 172.16.1.1 Apr 30 13:29:25 athos openais[8569]: [TOTEM] aru 1a high delivered 1a received flag 1 Apr 30 13:29:25 athos openais[8569]: [TOTEM] position [2] member 172.16.1.3: Apr 30 13:29:25 athos openais[8569]: [TOTEM] previous ring seq 10000 rep 172.16.1.1 Apr 30 13:29:25 athos openais[8569]: [TOTEM] aru 17 high delivered 17 received flag 0 Apr 30 13:29:25 athos openais[8569]: [TOTEM] copying all old ring messages from 18-1a. Apr 30 13:29:25 athos openais[8569]: [TOTEM] Originated 3 messages in RECOVERY. Apr 30 13:29:25 athos openais[8569]: [TOTEM] Originated for recovery: 18 19 1a Apr 30 13:29:25 athos openais[8569]: [TOTEM] Not Originated for recovery: Apr 30 13:29:25 athos openais[8569]: [TOTEM] Sending initial ORF token Apr 30 13:29:25 athos openais[8569]: [CLM ] CLM CONFIGURATION CHANGE Apr 30 13:29:25 athos openais[8569]: [CLM ] New Configuration: Apr 30 13:29:25 athos openais[8569]: [CLM ] r(0) ip(172.16.1.1) Apr 30 13:29:25 athos openais[8569]: [CLM ] r(0) ip(172.16.1.2) Apr 30 13:29:25 athos openais[8569]: [CLM ] r(0) ip(172.16.1.3) Apr 30 13:29:25 athos openais[8569]: [CLM ] Members Left: Apr 30 13:29:25 athos openais[8569]: [CLM ] Members Joined: Apr 30 13:29:25 athos openais[8569]: [CLM ] CLM CONFIGURATION CHANGE Apr 30 13:29:25 athos openais[8569]: [CLM ] New Configuration: Apr 30 13:29:25 athos openais[8569]: [CLM ] r(0) ip(172.16.1.1) Apr 30 13:29:25 athos openais[8569]: [CLM ] r(0) ip(172.16.1.2) Apr 30 13:29:25 athos openais[8569]: [CLM ] r(0) ip(172.16.1.3) Apr 30 13:29:25 athos openais[8569]: [CLM ] Members Left: Apr 30 13:29:25 athos openais[8569]: [CLM ] Members Joined: Apr 30 13:29:25 athos openais[8569]: [SYNC ] This node is within the primary component and will provide service. Apr 30 13:29:25 athos openais[8569]: [TOTEM] entering OPERATIONAL state. Apr 30 13:29:32 athos openais[8569]: [TOTEM] entering GATHER state from 12. Apr 30 13:29:33 athos openais[8569]: [TOTEM] Creating commit token because I am the rep. Apr 30 13:29:33 athos openais[8569]: [TOTEM] Saving state aru 7 high seq received 7 Apr 30 13:29:33 athos openais[8569]: [TOTEM] Storing new sequence id for ring 271c Apr 30 13:29:33 athos openais[8569]: [TOTEM] entering COMMIT state. Apr 30 13:29:33 athos openais[8569]: [TOTEM] entering RECOVERY state. Apr 30 13:29:33 athos openais[8569]: [TOTEM] position [0] member 172.16.1.1: Apr 30 13:29:33 athos openais[8569]: [TOTEM] previous ring seq 10008 rep 172.16.1.1 Apr 30 13:29:33 athos openais[8569]: [TOTEM] aru 7 high delivered 7 received flag 1 Apr 30 13:29:33 athos openais[8569]: [TOTEM] position [1] member 172.16.1.2: Apr 30 13:29:33 athos openais[8569]: [TOTEM] previous ring seq 10000 rep 172.16.1.1 Apr 30 13:29:33 athos openais[8569]: [TOTEM] aru 1a high delivered 1a received flag 1 Apr 30 13:29:33 athos openais[8569]: [TOTEM] position [2] member 172.16.1.3: Apr 30 13:29:33 athos openais[8569]: [TOTEM] previous ring seq 10000 rep 172.16.1.1 Apr 30 13:29:33 athos openais[8569]: [TOTEM] aru 17 high delivered 17 received flag 0 Apr 30 13:29:33 athos openais[8569]: [TOTEM] Did not need to originate any messages in recovery. Apr 30 13:29:33 athos openais[8569]: [TOTEM] Sending initial ORF token Apr 30 13:29:33 athos openais[8569]: [TOTEM] -not adding msg with seq no 18 Apr 30 13:29:33 athos openais[8569]: [TOTEM] -not adding msg with seq no 19 Apr 30 13:29:33 athos openais[8569]: [TOTEM] -not adding msg with seq no 1a Apr 30 13:29:33 athos openais[8569]: [CLM ] CLM CONFIGURATION CHANGE Apr 30 13:29:33 athos openais[8569]: [CLM ] New Configuration: Apr 30 13:29:33 athos openais[8569]: [CLM ] r(0) ip(172.16.1.1) Apr 30 13:29:33 athos openais[8569]: [CLM ] r(0) ip(172.16.1.2) Apr 30 13:29:33 athos openais[8569]: [CLM ] r(0) ip(172.16.1.3) Apr 30 13:29:33 athos openais[8569]: [CLM ] Members Left: Apr 30 13:29:33 athos openais[8569]: [CLM ] Members Joined: Apr 30 13:29:33 athos openais[8569]: [CLM ] CLM CONFIGURATION CHANGE Apr 30 13:29:33 athos openais[8569]: [CLM ] New Configuration: Apr 30 13:29:33 athos openais[8569]: [CLM ] r(0) ip(172.16.1.1) Apr 30 13:29:33 athos openais[8569]: [CLM ] r(0) ip(172.16.1.2) Apr 30 13:29:33 athos openais[8569]: [CLM ] r(0) ip(172.16.1.3) Apr 30 13:29:33 athos openais[8569]: [CLM ] Members Left: Apr 30 13:29:33 athos openais[8569]: [CLM ] Members Joined: Apr 30 13:29:33 athos openais[8569]: [SYNC ] This node is within the primary component and will provide service. Apr 30 13:29:33 athos openais[8569]: [TOTEM] entering OPERATIONAL state. Apr 30 13:29:33 athos openais[8569]: [CLM ] got nodejoin message 172.16.1.2 Apr 30 13:29:33 athos openais[8569]: [CLM ] got nodejoin message 172.16.1.3 Apr 30 13:29:33 athos openais[8569]: [TOTEM] Retransmit List: 7 Apr 30 13:29:33 athos openais[8569]: [CLM ] got nodejoin message 172.16.1.1 Apr 30 13:29:33 athos openais[8569]: [TOTEM] Retransmit List: 7 9 Apr 30 13:29:33 athos last message repeated 6 times Apr 30 13:29:42 athos openais[8569]: [TOTEM] FAILED TO RECEIVE Apr 30 13:29:42 athos openais[8569]: [TOTEM] entering GATHER state from 6. Apr 30 13:29:42 athos openais[8569]: [TOTEM] Creating commit token because I am the rep. Apr 30 13:29:42 athos openais[8569]: [TOTEM] Saving state aru e high seq received e Apr 30 13:29:42 athos openais[8569]: [TOTEM] Storing new sequence id for ring 2720 Apr 30 13:29:42 athos openais[8569]: [TOTEM] entering COMMIT state. Apr 30 13:29:42 athos openais[8569]: [TOTEM] entering RECOVERY state. Apr 30 13:29:42 athos openais[8569]: [TOTEM] position [0] member 172.16.1.1: Apr 30 13:29:42 athos openais[8569]: [TOTEM] previous ring seq 10012 rep 172.16.1.1 Apr 30 13:29:42 athos openais[8569]: [TOTEM] aru e high delivered e received flag 1 Apr 30 13:29:42 athos openais[8569]: [TOTEM] position [1] member 172.16.1.2: Apr 30 13:29:42 athos openais[8569]: [TOTEM] previous ring seq 10012 rep 172.16.1.1 Apr 30 13:29:42 athos openais[8569]: [TOTEM] aru e high delivered e received flag 1 Apr 30 13:29:42 athos openais[8569]: [TOTEM] position [2] member 172.16.1.3: Apr 30 13:29:42 athos openais[8569]: [TOTEM] previous ring seq 10012 rep 172.16.1.1 Apr 30 13:29:42 athos openais[8569]: [TOTEM] aru a high delivered a received flag 0 Apr 30 13:29:42 athos openais[8569]: [TOTEM] copying all old ring messages from b-e. Apr 30 13:29:42 athos openais[8569]: [TOTEM] Originated 4 messages in RECOVERY. Apr 30 13:29:42 athos openais[8569]: [TOTEM] Originated for recovery: b c d e Apr 30 13:29:42 athos openais[8569]: [TOTEM] Not Originated for recovery: Apr 30 13:29:42 athos openais[8569]: [TOTEM] Sending initial ORF token Apr 30 13:29:50 athos openais[8569]: [TOTEM] FAILED TO RECEIVE Apr 30 13:29:50 athos openais[8569]: [TOTEM] Restoring instance->my_aru e my high seq received e Apr 30 13:29:50 athos openais[8569]: [TOTEM] entering GATHER state from 6. Apr 30 13:29:50 athos openais[8569]: [TOTEM] Creating commit token because I am the rep. Apr 30 13:29:50 athos openais[8569]: [TOTEM] Storing new sequence id for ring 2724 Apr 30 13:29:50 athos openais[8569]: [TOTEM] entering COMMIT state. Apr 30 13:29:50 athos openais[8569]: [TOTEM] entering RECOVERY state. Apr 30 13:29:50 athos openais[8569]: [TOTEM] position [0] member 172.16.1.1: Apr 30 13:29:50 athos openais[8569]: [TOTEM] previous ring seq 10012 rep 172.16.1.1 Apr 30 13:29:50 athos openais[8569]: [TOTEM] aru e high delivered e received flag 1 Apr 30 13:29:50 athos openais[8569]: [TOTEM] position [1] member 172.16.1.2: Apr 30 13:29:50 athos openais[8569]: [TOTEM] previous ring seq 10012 rep 172.16.1.1 Apr 30 13:29:50 athos openais[8569]: [TOTEM] aru e high delivered e received flag 1 Apr 30 13:29:50 athos openais[8569]: [TOTEM] position [2] member 172.16.1.3: Apr 30 13:29:50 athos openais[8569]: [TOTEM] previous ring seq 10012 rep 172.16.1.1 Apr 30 13:29:50 athos openais[8569]: [TOTEM] aru a high delivered a received flag 0 Apr 30 13:29:50 athos openais[8569]: [TOTEM] copying all old ring messages from b-e. Apr 30 13:29:50 athos openais[8569]: [TOTEM] Originated 4 messages in RECOVERY. Apr 30 13:29:50 athos openais[8569]: [TOTEM] Originated for recovery: b c d e Apr 30 13:29:50 athos openais[8569]: [TOTEM] Not Originated for recovery: Apr 30 13:29:50 athos openais[8569]: [TOTEM] Sending initial ORF token Apr 30 13:29:50 athos openais[8569]: [CLM ] CLM CONFIGURATION CHANGE Apr 30 13:29:50 athos openais[8569]: [CLM ] New Configuration: Apr 30 13:29:50 athos openais[8569]: [CLM ] r(0) ip(172.16.1.1) Apr 30 13:29:50 athos openais[8569]: [CLM ] r(0) ip(172.16.1.2) Apr 30 13:29:50 athos openais[8569]: [CLM ] r(0) ip(172.16.1.3) Apr 30 13:29:50 athos openais[8569]: [CLM ] Members Left: Apr 30 13:29:50 athos openais[8569]: [CLM ] Members Joined: Apr 30 13:29:50 athos openais[8569]: [CLM ] CLM CONFIGURATION CHANGE Apr 30 13:29:50 athos openais[8569]: [CLM ] New Configuration: Apr 30 13:29:50 athos openais[8569]: [CLM ] r(0) ip(172.16.1.1) Apr 30 13:29:50 athos openais[8569]: [CLM ] r(0) ip(172.16.1.2) Apr 30 13:29:50 athos openais[8569]: [CLM ] r(0) ip(172.16.1.3) Apr 30 13:29:50 athos openais[8569]: [CLM ] Members Left: Apr 30 13:29:50 athos openais[8569]: [CLM ] Members Joined: Apr 30 13:29:50 athos openais[8569]: [SYNC ] This node is within the primary component and will provide service. Apr 30 13:29:50 athos openais[8569]: [TOTEM] entering OPERATIONAL state. Apr 30 13:29:57 athos openais[8569]: [TOTEM] entering GATHER state from 12. Apr 30 13:29:58 athos openais[8569]: [TOTEM] Creating commit token because I am the rep. Apr 30 13:29:58 athos openais[8569]: [TOTEM] Saving state aru 9 high seq received 9 Apr 30 13:29:58 athos openais[8569]: [TOTEM] Storing new sequence id for ring 2728 Apr 30 13:29:58 athos openais[8569]: [TOTEM] entering COMMIT state. Apr 30 13:29:58 athos openais[8569]: [TOTEM] entering RECOVERY state. Apr 30 13:29:58 athos openais[8569]: [TOTEM] position [0] member 172.16.1.1: Apr 30 13:29:58 athos openais[8569]: [TOTEM] previous ring seq 10020 rep 172.16.1.1 Apr 30 13:29:58 athos openais[8569]: [TOTEM] aru 9 high delivered 9 received flag 1 Apr 30 13:29:58 athos openais[8569]: [TOTEM] position [1] member 172.16.1.2: Apr 30 13:29:58 athos openais[8569]: [TOTEM] previous ring seq 10012 rep 172.16.1.1 Apr 30 13:29:58 athos openais[8569]: [TOTEM] aru e high delivered e received flag 1 Apr 30 13:29:58 athos openais[8569]: [TOTEM] position [2] member 172.16.1.3: Apr 30 13:29:58 athos openais[8569]: [TOTEM] previous ring seq 10012 rep 172.16.1.1 Apr 30 13:29:58 athos openais[8569]: [TOTEM] aru a high delivered a received flag 0 Apr 30 13:29:58 athos openais[8569]: [TOTEM] Did not need to originate any messages in recovery. Apr 30 13:29:58 athos openais[8569]: [TOTEM] Sending initial ORF token Apr 30 13:29:58 athos openais[8569]: [TOTEM] -not adding msg with seq no b Apr 30 13:29:58 athos openais[8569]: [TOTEM] -not adding msg with seq no c Apr 30 13:29:58 athos openais[8569]: [TOTEM] -not adding msg with seq no d Apr 30 13:29:58 athos openais[8569]: [TOTEM] -not adding msg with seq no e Apr 30 13:29:58 athos openais[8569]: [CLM ] CLM CONFIGURATION CHANGE Apr 30 13:29:58 athos openais[8569]: [CLM ] New Configuration: Apr 30 13:29:58 athos openais[8569]: [CLM ] r(0) ip(172.16.1.1) Apr 30 13:29:58 athos openais[8569]: [CLM ] r(0) ip(172.16.1.2) Apr 30 13:29:58 athos openais[8569]: [CLM ] r(0) ip(172.16.1.3) Apr 30 13:29:58 athos openais[8569]: [CLM ] Members Left: Apr 30 13:29:58 athos openais[8569]: [CLM ] Members Joined: Apr 30 13:29:58 athos openais[8569]: [CLM ] CLM CONFIGURATION CHANGE Apr 30 13:29:58 athos openais[8569]: [CLM ] New Configuration: Apr 30 13:29:58 athos openais[8569]: [CLM ] r(0) ip(172.16.1.1) Apr 30 13:29:58 athos openais[8569]: [CLM ] r(0) ip(172.16.1.2) Apr 30 13:29:58 athos openais[8569]: [CLM ] r(0) ip(172.16.1.3) Apr 30 13:29:58 athos openais[8569]: [CLM ] Members Left: Apr 30 13:29:58 athos openais[8569]: [CLM ] Members Joined: Apr 30 13:29:58 athos openais[8569]: [SYNC ] This node is within the primary component and will provide service. Apr 30 13:29:58 athos openais[8569]: [TOTEM] entering OPERATIONAL state. Apr 30 13:29:58 athos openais[8569]: [CLM ] got nodejoin message 172.16.1.2 Apr 30 13:29:58 athos openais[8569]: [CLM ] got nodejoin message 172.16.1.3 Apr 30 13:29:58 athos openais[8569]: [TOTEM] Retransmit List: 8 Apr 30 13:29:58 athos openais[8569]: [CLM ] got nodejoin message 172.16.1.1 Apr 30 13:29:58 athos openais[8569]: [TOTEM] Retransmit List: 8 a Apr 30 13:29:58 athos last message repeated 7 times Apr 30 13:30:07 athos openais[8569]: [TOTEM] FAILED TO RECEIVE Apr 30 13:30:07 athos openais[8569]: [TOTEM] entering GATHER state from 6. Apr 30 13:30:08 athos openais[8569]: [TOTEM] Creating commit token because I am the rep. Apr 30 13:30:08 athos openais[8569]: [TOTEM] Saving state aru f high seq received f Apr 30 13:30:08 athos openais[8569]: [TOTEM] Storing new sequence id for ring 272c Apr 30 13:30:08 athos openais[8569]: [TOTEM] entering COMMIT state. Apr 30 13:30:08 athos openais[8569]: [TOTEM] entering RECOVERY state. Apr 30 13:30:08 athos openais[8569]: [TOTEM] position [0] member 172.16.1.1: Apr 30 13:30:08 athos openais[8569]: [TOTEM] previous ring seq 10024 rep 172.16.1.1 Apr 30 13:30:08 athos openais[8569]: [TOTEM] aru f high delivered f received flag 1 Apr 30 13:30:08 athos openais[8569]: [TOTEM] position [1] member 172.16.1.2: Apr 30 13:30:08 athos openais[8569]: [TOTEM] previous ring seq 10024 rep 172.16.1.1 Apr 30 13:30:08 athos openais[8569]: [TOTEM] aru f high delivered f received flag 1 Apr 30 13:30:08 athos openais[8569]: [TOTEM] position [2] member 172.16.1.3: Apr 30 13:30:08 athos openais[8569]: [TOTEM] previous ring seq 10024 rep 172.16.1.1 Apr 30 13:30:08 athos openais[8569]: [TOTEM] aru b high delivered b received flag 0 Apr 30 13:30:08 athos openais[8569]: [TOTEM] copying all old ring messages from c-f. Apr 30 13:30:08 athos openais[8569]: [TOTEM] Originated 4 messages in RECOVERY. Apr 30 13:30:08 athos openais[8569]: [TOTEM] Originated for recovery: c d e f Apr 30 13:30:08 athos openais[8569]: [TOTEM] Not Originated for recovery: Apr 30 13:30:08 athos openais[8569]: [TOTEM] Sending initial ORF token Apr 30 13:30:15 athos openais[8569]: [TOTEM] FAILED TO RECEIVE Apr 30 13:30:15 athos openais[8569]: [TOTEM] Restoring instance->my_aru f my high seq received f Apr 30 13:30:15 athos openais[8569]: [TOTEM] entering GATHER state from 6. Apr 30 13:30:15 athos openais[8569]: [TOTEM] Creating commit token because I am the rep. Apr 30 13:30:15 athos openais[8569]: [TOTEM] Storing new sequence id for ring 2730 Apr 30 13:30:15 athos openais[8569]: [TOTEM] entering COMMIT state. Apr 30 13:30:15 athos openais[8569]: [TOTEM] entering RECOVERY state. Apr 30 13:30:15 athos openais[8569]: [TOTEM] position [0] member 172.16.1.1: Apr 30 13:30:15 athos openais[8569]: [TOTEM] previous ring seq 10024 rep 172.16.1.1 Apr 30 13:30:15 athos openais[8569]: [TOTEM] aru f high delivered f received flag 1 Apr 30 13:30:15 athos openais[8569]: [TOTEM] position [1] member 172.16.1.2: Apr 30 13:30:15 athos openais[8569]: [TOTEM] previous ring seq 10024 rep 172.16.1.1 Apr 30 13:30:15 athos openais[8569]: [TOTEM] aru f high delivered f received flag 1 Apr 30 13:30:15 athos openais[8569]: [TOTEM] position [2] member 172.16.1.3: Apr 30 13:30:15 athos openais[8569]: [TOTEM] previous ring seq 10024 rep 172.16.1.1 Apr 30 13:30:15 athos openais[8569]: [TOTEM] aru b high delivered b received flag 0 Apr 30 13:30:15 athos openais[8569]: [TOTEM] copying all old ring messages from c-f. Apr 30 13:30:15 athos openais[8569]: [TOTEM] Originated 4 messages in RECOVERY. Apr 30 13:30:15 athos openais[8569]: [TOTEM] Originated for recovery: c d e f Apr 30 13:30:15 athos openais[8569]: [TOTEM] Not Originated for recovery: Apr 30 13:30:15 athos openais[8569]: [TOTEM] Sending initial ORF token Apr 30 13:30:15 athos openais[8569]: [CLM ] CLM CONFIGURATION CHANGE Apr 30 13:30:15 athos openais[8569]: [CLM ] New Configuration: Apr 30 13:30:15 athos openais[8569]: [CLM ] r(0) ip(172.16.1.1) Apr 30 13:30:15 athos openais[8569]: [CLM ] r(0) ip(172.16.1.2) Apr 30 13:30:16 athos openais[8569]: [CLM ] r(0) ip(172.16.1.3) Apr 30 13:30:16 athos openais[8569]: [CLM ] Members Left: Apr 30 13:30:16 athos openais[8569]: [CLM ] Members Joined: Apr 30 13:30:16 athos openais[8569]: [CLM ] CLM CONFIGURATION CHANGE Apr 30 13:30:16 athos openais[8569]: [CLM ] New Configuration: Apr 30 13:30:16 athos openais[8569]: [CLM ] r(0) ip(172.16.1.1) Apr 30 13:30:16 athos openais[8569]: [CLM ] r(0) ip(172.16.1.2) Apr 30 13:30:16 athos openais[8569]: [CLM ] r(0) ip(172.16.1.3) Apr 30 13:30:16 athos openais[8569]: [CLM ] Members Left: Apr 30 13:30:16 athos openais[8569]: [CLM ] Members Joined: Apr 30 13:30:16 athos openais[8569]: [SYNC ] This node is within the primary component and will provide service. Apr 30 13:30:16 athos openais[8569]: [TOTEM] entering OPERATIONAL state. Apr 30 13:30:23 athos openais[8569]: [TOTEM] entering GATHER state from 12. Apr 30 13:30:23 athos openais[8569]: [TOTEM] Creating commit token because I am the rep. Apr 30 13:30:23 athos openais[8569]: [TOTEM] Saving state aru 9 high seq received 9 Apr 30 13:30:23 athos openais[8569]: [TOTEM] Storing new sequence id for ring 2734 Apr 30 13:30:23 athos openais[8569]: [TOTEM] entering COMMIT state. Apr 30 13:30:23 athos openais[8569]: [TOTEM] entering RECOVERY state. Apr 30 13:30:23 athos openais[8569]: [TOTEM] position [0] member 172.16.1.1: Apr 30 13:30:23 athos openais[8569]: [TOTEM] previous ring seq 10032 rep 172.16.1.1 Apr 30 13:30:23 athos openais[8569]: [TOTEM] aru 9 high delivered 9 received flag 1 Apr 30 13:30:23 athos openais[8569]: [TOTEM] position [1] member 172.16.1.2: Apr 30 13:30:23 athos openais[8569]: [TOTEM] previous ring seq 10024 rep 172.16.1.1 Apr 30 13:30:23 athos openais[8569]: [TOTEM] aru f high delivered f received flag 1 Apr 30 13:30:23 athos openais[8569]: [TOTEM] position [2] member 172.16.1.3: Apr 30 13:30:23 athos openais[8569]: [TOTEM] previous ring seq 10024 rep 172.16.1.1 Apr 30 13:30:23 athos openais[8569]: [TOTEM] aru b high delivered b received flag 0 Apr 30 13:30:23 athos openais[8569]: [TOTEM] Did not need to originate any messages in recovery. Apr 30 13:30:23 athos openais[8569]: [TOTEM] Sending initial ORF token Apr 30 13:30:23 athos openais[8569]: [TOTEM] -not adding msg with seq no c Apr 30 13:30:23 athos openais[8569]: [TOTEM] -not adding msg with seq no d Apr 30 13:30:23 athos openais[8569]: [TOTEM] -not adding msg with seq no e Apr 30 13:30:23 athos openais[8569]: [TOTEM] -not adding msg with seq no f Apr 30 13:30:23 athos openais[8569]: [CLM ] CLM CONFIGURATION CHANGE Apr 30 13:30:23 athos openais[8569]: [CLM ] New Configuration: Apr 30 13:30:23 athos openais[8569]: [CLM ] r(0) ip(172.16.1.1) Apr 30 13:30:23 athos openais[8569]: [CLM ] r(0) ip(172.16.1.2) Apr 30 13:30:23 athos openais[8569]: [CLM ] r(0) ip(172.16.1.3) Apr 30 13:30:23 athos openais[8569]: [CLM ] Members Left: Apr 30 13:30:23 athos openais[8569]: [CLM ] Members Joined: Apr 30 13:30:23 athos openais[8569]: [CLM ] CLM CONFIGURATION CHANGE Apr 30 13:30:23 athos openais[8569]: [CLM ] New Configuration: Apr 30 13:30:23 athos openais[8569]: [CLM ] r(0) ip(172.16.1.1) Apr 30 13:30:23 athos openais[8569]: [CLM ] r(0) ip(172.16.1.2) Apr 30 13:30:23 athos openais[8569]: [CLM ] r(0) ip(172.16.1.3) Apr 30 13:30:23 athos openais[8569]: [CLM ] Members Left: Apr 30 13:30:23 athos openais[8569]: [CLM ] Members Joined: Apr 30 13:30:23 athos openais[8569]: [SYNC ] This node is within the primary component and will provide service. Apr 30 13:30:23 athos openais[8569]: [TOTEM] entering OPERATIONAL state. Apr 30 13:30:23 athos openais[8569]: [CLM ] got nodejoin message 172.16.1.2 Apr 30 13:30:23 athos openais[8569]: [CLM ] got nodejoin message 172.16.1.3 Apr 30 13:30:23 athos openais[8569]: [CLM ] got nodejoin message 172.16.1.1 Apr 30 13:30:23 athos openais[8569]: [TOTEM] Retransmit List: a Apr 30 13:30:23 athos last message repeated 27 times Apr 30 13:30:41 athos openais[8569]: [TOTEM] FAILED TO RECEIVE Apr 30 13:30:41 athos openais[8569]: [TOTEM] entering GATHER state from 6. Apr 30 13:30:41 athos openais[8569]: [TOTEM] Creating commit token because I am the rep. Apr 30 13:30:41 athos openais[8569]: [TOTEM] Saving state aru f high seq received f Apr 30 13:30:41 athos openais[8569]: [TOTEM] Storing new sequence id for ring 2738 Apr 30 13:30:41 athos openais[8569]: [TOTEM] entering COMMIT state. Apr 30 13:30:41 athos openais[8569]: [TOTEM] entering RECOVERY state. Apr 30 13:30:41 athos openais[8569]: [TOTEM] position [0] member 172.16.1.1: Apr 30 13:30:41 athos openais[8569]: [TOTEM] previous ring seq 10036 rep 172.16.1.1 Apr 30 13:30:41 athos openais[8569]: [TOTEM] aru f high delivered f received flag 1 Apr 30 13:30:41 athos openais[8569]: [TOTEM] position [1] member 172.16.1.2: Apr 30 13:30:41 athos openais[8569]: [TOTEM] previous ring seq 10036 rep 172.16.1.1 Apr 30 13:30:41 athos openais[8569]: [TOTEM] aru f high delivered f received flag 1 Apr 30 13:30:41 athos openais[8569]: [TOTEM] position [2] member 172.16.1.3: Apr 30 13:30:41 athos openais[8569]: [TOTEM] previous ring seq 10036 rep 172.16.1.1 Apr 30 13:30:41 athos openais[8569]: [TOTEM] aru b high delivered b received flag 0 Apr 30 13:30:41 athos openais[8569]: [TOTEM] copying all old ring messages from c-f. Apr 30 13:30:41 athos openais[8569]: [TOTEM] Originated 4 messages in RECOVERY. Apr 30 13:30:41 athos openais[8569]: [TOTEM] Originated for recovery: c d e f Apr 30 13:30:41 athos openais[8569]: [TOTEM] Not Originated for recovery: Apr 30 13:30:41 athos openais[8569]: [TOTEM] Sending initial ORF token Apr 30 13:30:49 athos openais[8569]: [TOTEM] FAILED TO RECEIVE Apr 30 13:30:49 athos openais[8569]: [TOTEM] Restoring instance->my_aru f my high seq received f Apr 30 13:30:49 athos openais[8569]: [TOTEM] entering GATHER state from 6. Apr 30 13:30:49 athos openais[8569]: [TOTEM] Creating commit token because I am the rep. Apr 30 13:30:49 athos openais[8569]: [TOTEM] Storing new sequence id for ring 273c Apr 30 13:30:49 athos openais[8569]: [TOTEM] entering COMMIT state. Apr 30 13:30:49 athos openais[8569]: [TOTEM] entering RECOVERY state. Apr 30 13:30:49 athos openais[8569]: [TOTEM] position [0] member 172.16.1.1: Apr 30 13:30:49 athos openais[8569]: [TOTEM] previous ring seq 10036 rep 172.16.1.1 Apr 30 13:30:49 athos openais[8569]: [TOTEM] aru f high delivered f received flag 1 Apr 30 13:30:49 athos openais[8569]: [TOTEM] position [1] member 172.16.1.2: Apr 30 13:30:49 athos openais[8569]: [TOTEM] previous ring seq 10036 rep 172.16.1.1 Apr 30 13:30:49 athos openais[8569]: [TOTEM] aru f high delivered f received flag 1 Apr 30 13:30:49 athos openais[8569]: [TOTEM] position [2] member 172.16.1.3: Apr 30 13:30:49 athos openais[8569]: [TOTEM] previous ring seq 10036 rep 172.16.1.1 Apr 30 13:30:49 athos openais[8569]: [TOTEM] aru b high delivered b received flag 0 Apr 30 13:30:49 athos openais[8569]: [TOTEM] copying all old ring messages from c-f. Apr 30 13:30:49 athos openais[8569]: [TOTEM] Originated 4 messages in RECOVERY. Apr 30 13:30:49 athos openais[8569]: [TOTEM] Originated for recovery: c d e f Apr 30 13:30:49 athos openais[8569]: [TOTEM] Not Originated for recovery: Apr 30 13:30:49 athos openais[8569]: [TOTEM] Sending initial ORF token Apr 30 13:30:49 athos openais[8569]: [CLM ] CLM CONFIGURATION CHANGE Apr 30 13:30:49 athos openais[8569]: [CLM ] New Configuration: Apr 30 13:30:49 athos openais[8569]: [CLM ] r(0) ip(172.16.1.1) Apr 30 13:30:49 athos openais[8569]: [CLM ] r(0) ip(172.16.1.2) Apr 30 13:30:49 athos openais[8569]: [CLM ] r(0) ip(172.16.1.3) Apr 30 13:30:49 athos openais[8569]: [CLM ] Members Left: Apr 30 13:30:49 athos openais[8569]: [CLM ] Members Joined: Apr 30 13:30:49 athos openais[8569]: [CLM ] CLM CONFIGURATION CHANGE Apr 30 13:30:49 athos openais[8569]: [CLM ] New Configuration: Apr 30 13:30:49 athos openais[8569]: [CLM ] r(0) ip(172.16.1.1) Apr 30 13:30:49 athos openais[8569]: [CLM ] r(0) ip(172.16.1.2) Apr 30 13:30:49 athos openais[8569]: [CLM ] r(0) ip(172.16.1.3) Apr 30 13:30:49 athos openais[8569]: [CLM ] Members Left: Apr 30 13:30:49 athos openais[8569]: [CLM ] Members Joined: Apr 30 13:30:49 athos openais[8569]: [SYNC ] This node is within the primary component and will provide service. Apr 30 13:30:49 athos openais[8569]: [TOTEM] entering OPERATIONAL state. Apr 30 13:30:56 athos openais[8569]: [TOTEM] entering GATHER state from 12. Apr 30 13:30:56 athos openais[8569]: [TOTEM] Creating commit token because I am the rep. Apr 30 13:30:56 athos openais[8569]: [TOTEM] Saving state aru 9 high seq received 9 Apr 30 13:30:56 athos openais[8569]: [TOTEM] Storing new sequence id for ring 2740 Apr 30 13:30:56 athos openais[8569]: [TOTEM] entering COMMIT state. Apr 30 13:30:56 athos openais[8569]: [TOTEM] entering RECOVERY state. Apr 30 13:30:56 athos openais[8569]: [TOTEM] position [0] member 172.16.1.1: Apr 30 13:30:56 athos openais[8569]: [TOTEM] previous ring seq 10044 rep 172.16.1.1 Apr 30 13:30:56 athos openais[8569]: [TOTEM] aru 9 high delivered 9 received flag 1 Apr 30 13:30:56 athos openais[8569]: [TOTEM] position [1] member 172.16.1.2: Apr 30 13:30:56 athos openais[8569]: [TOTEM] previous ring seq 10036 rep 172.16.1.1 Apr 30 13:30:56 athos openais[8569]: [TOTEM] aru f high delivered f received flag 1 Apr 30 13:30:56 athos openais[8569]: [TOTEM] position [2] member 172.16.1.3: Apr 30 13:30:56 athos openais[8569]: [TOTEM] previous ring seq 10036 rep 172.16.1.1 Apr 30 13:30:56 athos openais[8569]: [TOTEM] aru b high delivered b received flag 0 Apr 30 13:30:56 athos openais[8569]: [TOTEM] Did not need to originate any messages in recovery. Apr 30 13:30:56 athos openais[8569]: [TOTEM] Sending initial ORF token Apr 30 13:30:56 athos openais[8569]: [TOTEM] -not adding msg with seq no c Apr 30 13:30:56 athos openais[8569]: [TOTEM] -not adding msg with seq no d Apr 30 13:30:56 athos openais[8569]: [TOTEM] -not adding msg with seq no e Apr 30 13:30:56 athos openais[8569]: [TOTEM] -not adding msg with seq no f Apr 30 13:30:56 athos openais[8569]: [CLM ] CLM CONFIGURATION CHANGE Apr 30 13:30:56 athos openais[8569]: [CLM ] New Configuration: Apr 30 13:30:56 athos openais[8569]: [CLM ] r(0) ip(172.16.1.1) Apr 30 13:30:56 athos openais[8569]: [CLM ] r(0) ip(172.16.1.2) Apr 30 13:30:56 athos openais[8569]: [CLM ] r(0) ip(172.16.1.3) Apr 30 13:30:56 athos openais[8569]: [CLM ] Members Left: Apr 30 13:30:56 athos openais[8569]: [CLM ] Members Joined: Apr 30 13:30:56 athos openais[8569]: [CLM ] CLM CONFIGURATION CHANGE Apr 30 13:30:56 athos openais[8569]: [CLM ] New Configuration: Apr 30 13:30:56 athos openais[8569]: [CLM ] r(0) ip(172.16.1.1) Apr 30 13:30:56 athos openais[8569]: [CLM ] r(0) ip(172.16.1.2) Apr 30 13:30:56 athos openais[8569]: [CLM ] r(0) ip(172.16.1.3) Apr 30 13:30:56 athos openais[8569]: [CLM ] Members Left: Apr 30 13:30:56 athos openais[8569]: [CLM ] Members Joined: Apr 30 13:30:56 athos openais[8569]: [SYNC ] This node is within the primary component and will provide service. Apr 30 13:30:56 athos openais[8569]: [TOTEM] entering OPERATIONAL state. Apr 30 13:30:56 athos openais[8569]: [CLM ] got nodejoin message 172.16.1.2 Apr 30 13:30:56 athos openais[8569]: [CLM ] got nodejoin message 172.16.1.3 Apr 30 13:30:56 athos openais[8569]: [TOTEM] Retransmit List: 8 Apr 30 13:30:56 athos openais[8569]: [CLM ] got nodejoin message 172.16.1.1 Apr 30 13:30:56 athos openais[8569]: [TOTEM] Retransmit List: 8 a Apr 30 13:30:57 athos last message repeated 14 times Apr 30 13:31:09 athos openais[8569]: [TOTEM] FAILED TO RECEIVE Apr 30 13:31:09 athos openais[8569]: [TOTEM] entering GATHER state from 6. Apr 30 13:31:09 athos openais[8569]: [TOTEM] Creating commit token because I am the rep. Apr 30 13:31:09 athos openais[8569]: [TOTEM] Saving state aru f high seq received f Apr 30 13:31:09 athos openais[8569]: [TOTEM] Storing new sequence id for ring 2744 Apr 30 13:31:09 athos openais[8569]: [TOTEM] entering COMMIT state. Apr 30 13:31:09 athos openais[8569]: [TOTEM] entering RECOVERY state. Apr 30 13:31:09 athos openais[8569]: [TOTEM] position [0] member 172.16.1.1: Apr 30 13:31:09 athos openais[8569]: [TOTEM] previous ring seq 10048 rep 172.16.1.1 Apr 30 13:31:09 athos openais[8569]: [TOTEM] aru f high delivered f received flag 1 Apr 30 13:31:09 athos openais[8569]: [TOTEM] position [1] member 172.16.1.2: Apr 30 13:31:09 athos openais[8569]: [TOTEM] previous ring seq 10048 rep 172.16.1.1 Apr 30 13:31:09 athos openais[8569]: [TOTEM] aru f high delivered f received flag 1 Apr 30 13:31:09 athos openais[8569]: [TOTEM] position [2] member 172.16.1.3: Apr 30 13:31:09 athos openais[8569]: [TOTEM] previous ring seq 10048 rep 172.16.1.1 Apr 30 13:31:09 athos openais[8569]: [TOTEM] aru b high delivered b received flag 0 Apr 30 13:31:09 athos openais[8569]: [TOTEM] copying all old ring messages from c-f. Apr 30 13:31:09 athos openais[8569]: [TOTEM] Originated 4 messages in RECOVERY. Apr 30 13:31:09 athos openais[8569]: [TOTEM] Originated for recovery: c d e f Apr 30 13:31:09 athos openais[8569]: [TOTEM] Not Originated for recovery: Apr 30 13:31:09 athos openais[8569]: [TOTEM] Sending initial ORF token Apr 30 13:31:17 athos openais[8569]: [TOTEM] FAILED TO RECEIVE Apr 30 13:31:17 athos openais[8569]: [TOTEM] Restoring instance->my_aru f my high seq received f Apr 30 13:31:17 athos openais[8569]: [TOTEM] entering GATHER state from 6. Apr 30 13:31:17 athos openais[8569]: [TOTEM] Creating commit token because I am the rep. Apr 30 13:31:17 athos openais[8569]: [TOTEM] Storing new sequence id for ring 2748 Apr 30 13:31:17 athos openais[8569]: [TOTEM] entering COMMIT state. Apr 30 13:31:17 athos openais[8569]: [TOTEM] entering RECOVERY state. Apr 30 13:31:17 athos openais[8569]: [TOTEM] position [0] member 172.16.1.1: Apr 30 13:31:17 athos openais[8569]: [TOTEM] previous ring seq 10048 rep 172.16.1.1 Apr 30 13:31:17 athos openais[8569]: [TOTEM] aru f high delivered f received flag 1 Apr 30 13:31:17 athos openais[8569]: [TOTEM] position [1] member 172.16.1.2: Apr 30 13:31:17 athos openais[8569]: [TOTEM] previous ring seq 10048 rep 172.16.1.1 Apr 30 13:31:17 athos openais[8569]: [TOTEM] aru f high delivered f received flag 1 Apr 30 13:31:17 athos openais[8569]: [TOTEM] position [2] member 172.16.1.3: Apr 30 13:31:17 athos openais[8569]: [TOTEM] previous ring seq 10048 rep 172.16.1.1 Apr 30 13:31:17 athos openais[8569]: [TOTEM] aru b high delivered b received flag 0 Apr 30 13:31:17 athos openais[8569]: [TOTEM] copying all old ring messages from c-f. Apr 30 13:31:17 athos openais[8569]: [TOTEM] Originated 4 messages in RECOVERY. Apr 30 13:31:17 athos openais[8569]: [TOTEM] Originated for recovery: c d e f Apr 30 13:31:17 athos openais[8569]: [TOTEM] Not Originated for recovery: Apr 30 13:31:17 athos openais[8569]: [TOTEM] Sending initial ORF token Apr 30 13:31:17 athos openais[8569]: [CLM ] CLM CONFIGURATION CHANGE Apr 30 13:31:17 athos openais[8569]: [CLM ] New Configuration: Apr 30 13:31:17 athos openais[8569]: [CLM ] r(0) ip(172.16.1.1) Apr 30 13:31:17 athos openais[8569]: [CLM ] r(0) ip(172.16.1.2) Apr 30 13:31:17 athos openais[8569]: [CLM ] r(0) ip(172.16.1.3) Apr 30 13:31:17 athos openais[8569]: [CLM ] Members Left: Apr 30 13:31:17 athos openais[8569]: [CLM ] Members Joined: Apr 30 13:31:17 athos openais[8569]: [CLM ] CLM CONFIGURATION CHANGE Apr 30 13:31:17 athos openais[8569]: [CLM ] New Configuration: Apr 30 13:31:17 athos openais[8569]: [CLM ] r(0) ip(172.16.1.1) Apr 30 13:31:17 athos openais[8569]: [CLM ] r(0) ip(172.16.1.2) Apr 30 13:31:17 athos openais[8569]: [CLM ] r(0) ip(172.16.1.3) Apr 30 13:31:17 athos openais[8569]: [CLM ] Members Left: Apr 30 13:31:17 athos openais[8569]: [CLM ] Members Joined: Apr 30 13:31:17 athos openais[8569]: [SYNC ] This node is within the primary component and will provide service. Apr 30 13:31:17 athos openais[8569]: [TOTEM] entering OPERATIONAL state. Apr 30 13:31:25 athos openais[8569]: [TOTEM] entering GATHER state from 12. Apr 30 13:31:25 athos openais[8569]: [TOTEM] Creating commit token because I am the rep. Apr 30 13:31:25 athos openais[8569]: [TOTEM] Saving state aru 9 high seq received 9 Apr 30 13:31:25 athos openais[8569]: [TOTEM] Storing new sequence id for ring 274c Apr 30 13:31:25 athos openais[8569]: [TOTEM] entering COMMIT state. Apr 30 13:31:25 athos openais[8569]: [TOTEM] entering RECOVERY state. Apr 30 13:31:25 athos openais[8569]: [TOTEM] position [0] member 172.16.1.1: Apr 30 13:31:25 athos openais[8569]: [TOTEM] previous ring seq 10056 rep 172.16.1.1 Apr 30 13:31:25 athos openais[8569]: [TOTEM] aru 9 high delivered 9 received flag 1 Apr 30 13:31:25 athos openais[8569]: [TOTEM] position [1] member 172.16.1.2: Apr 30 13:31:25 athos openais[8569]: [TOTEM] previous ring seq 10048 rep 172.16.1.1 Apr 30 13:31:25 athos openais[8569]: [TOTEM] aru f high delivered f received flag 1 Apr 30 13:31:25 athos openais[8569]: [TOTEM] position [2] member 172.16.1.3: Apr 30 13:31:25 athos openais[8569]: [TOTEM] previous ring seq 10048 rep 172.16.1.1 Apr 30 13:31:25 athos openais[8569]: [TOTEM] aru b high delivered b received flag 0 Apr 30 13:31:25 athos openais[8569]: [TOTEM] Did not need to originate any messages in recovery. Apr 30 13:31:25 athos openais[8569]: [TOTEM] Sending initial ORF token Apr 30 13:31:25 athos openais[8569]: [TOTEM] -not adding msg with seq no c Apr 30 13:31:25 athos openais[8569]: [TOTEM] -not adding msg with seq no d Apr 30 13:31:25 athos openais[8569]: [TOTEM] -not adding msg with seq no e Apr 30 13:31:25 athos openais[8569]: [TOTEM] -not adding msg with seq no f Apr 30 13:31:25 athos openais[8569]: [CLM ] CLM CONFIGURATION CHANGE Apr 30 13:31:25 athos openais[8569]: [CLM ] New Configuration: Apr 30 13:31:25 athos openais[8569]: [CLM ] r(0) ip(172.16.1.1) Apr 30 13:31:25 athos openais[8569]: [CLM ] r(0) ip(172.16.1.2) Apr 30 13:31:25 athos openais[8569]: [CLM ] r(0) ip(172.16.1.3) Apr 30 13:31:25 athos openais[8569]: [CLM ] Members Left: Apr 30 13:31:25 athos openais[8569]: [CLM ] Members Joined: Apr 30 13:31:25 athos openais[8569]: [CLM ] CLM CONFIGURATION CHANGE Apr 30 13:31:25 athos openais[8569]: [CLM ] New Configuration: Apr 30 13:31:25 athos openais[8569]: [CLM ] r(0) ip(172.16.1.1) Apr 30 13:31:25 athos openais[8569]: [CLM ] r(0) ip(172.16.1.2) Apr 30 13:31:25 athos openais[8569]: [CLM ] r(0) ip(172.16.1.3) Apr 30 13:31:25 athos openais[8569]: [CLM ] Members Left: Apr 30 13:31:25 athos openais[8569]: [CLM ] Members Joined: Apr 30 13:31:25 athos openais[8569]: [SYNC ] This node is within the primary component and will provide service. Apr 30 13:31:25 athos openais[8569]: [TOTEM] entering OPERATIONAL state. Apr 30 13:31:25 athos openais[8569]: [CLM ] got nodejoin message 172.16.1.2 Apr 30 13:31:25 athos openais[8569]: [CLM ] got nodejoin message 172.16.1.3 Apr 30 13:31:25 athos openais[8569]: [CLM ] got nodejoin message 172.16.1.1 Apr 30 13:31:25 athos openais[8569]: [TOTEM] Retransmit List: f Apr 30 13:31:25 athos last message repeated 26 times Apr 30 13:31:25 athos openais[8569]: [TOTEM] Retransmit List: 11 12 13 Apr 30 13:31:25 athos openais[8569]: [TOTEM] Retransmit List: 17 18 Apr 30 13:31:25 athos last message repeated 24 times Apr 30 13:31:25 athos openais[8569]: [TOTEM] Retransmit List: 1b 1c Apr 30 13:31:25 athos openais[8569]: [CPG ] got joinlist message from node 1 Apr 30 13:31:25 athos openais[8569]: [CPG ] got joinlist message from node 2 Apr 30 13:31:32 athos openais[8569]: [TOTEM] FAILED TO RECEIVE Apr 30 13:31:32 athos openais[8569]: [TOTEM] entering GATHER state from 6. Apr 30 13:31:32 athos openais[8569]: [TOTEM] Creating commit token because I am the rep. Apr 30 13:31:32 athos openais[8569]: [TOTEM] Saving state aru 25 high seq received 25 Apr 30 13:31:32 athos openais[8569]: [TOTEM] Storing new sequence id for ring 2750 Apr 30 13:31:32 athos openais[8569]: [TOTEM] entering COMMIT state. Apr 30 13:31:32 athos openais[8569]: [TOTEM] entering RECOVERY state. Apr 30 13:31:32 athos openais[8569]: [TOTEM] position [0] member 172.16.1.1: Apr 30 13:31:32 athos openais[8569]: [TOTEM] previous ring seq 10060 rep 172.16.1.1 Apr 30 13:31:32 athos openais[8569]: [TOTEM] aru 25 high delivered 25 received flag 1 Apr 30 13:31:32 athos openais[8569]: [TOTEM] position [1] member 172.16.1.2: Apr 30 13:31:32 athos openais[8569]: [TOTEM] previous ring seq 10060 rep 172.16.1.1 Apr 30 13:31:32 athos openais[8569]: [TOTEM] aru 25 high delivered 25 received flag 1 Apr 30 13:31:32 athos openais[8569]: [TOTEM] position [2] member 172.16.1.3: Apr 30 13:31:32 athos openais[8569]: [TOTEM] previous ring seq 10060 rep 172.16.1.1 Apr 30 13:31:32 athos openais[8569]: [TOTEM] aru 23 high delivered 23 received flag 0 Apr 30 13:31:32 athos openais[8569]: [TOTEM] copying all old ring messages from 24-25. Apr 30 13:31:32 athos openais[8569]: [TOTEM] Originated 2 messages in RECOVERY. Apr 30 13:31:32 athos openais[8569]: [TOTEM] Originated for recovery: 24 25 Apr 30 13:31:32 athos openais[8569]: [TOTEM] Not Originated for recovery: Apr 30 13:31:32 athos openais[8569]: [TOTEM] Sending initial ORF token Apr 30 13:31:32 athos openais[8569]: [CLM ] CLM CONFIGURATION CHANGE Apr 30 13:31:32 athos openais[8569]: [CLM ] New Configuration: Apr 30 13:31:32 athos openais[8569]: [CLM ] r(0) ip(172.16.1.1) Apr 30 13:31:32 athos openais[8569]: [CLM ] r(0) ip(172.16.1.2) Apr 30 13:31:32 athos openais[8569]: [CLM ] r(0) ip(172.16.1.3) Apr 30 13:31:32 athos openais[8569]: [CLM ] Members Left: Apr 30 13:31:32 athos openais[8569]: [CLM ] Members Joined: Apr 30 13:31:32 athos openais[8569]: [CLM ] CLM CONFIGURATION CHANGE Apr 30 13:31:32 athos openais[8569]: [CLM ] New Configuration: Apr 30 13:31:32 athos openais[8569]: [CLM ] r(0) ip(172.16.1.1) Apr 30 13:31:32 athos openais[8569]: [CLM ] r(0) ip(172.16.1.2) Apr 30 13:31:32 athos openais[8569]: [CLM ] r(0) ip(172.16.1.3) Apr 30 13:31:32 athos openais[8569]: [CLM ] Members Left: Apr 30 13:31:32 athos openais[8569]: [CLM ] Members Joined: Apr 30 13:31:32 athos openais[8569]: [SYNC ] This node is within the primary component and will provide service. Apr 30 13:31:32 athos openais[8569]: [TOTEM] entering OPERATIONAL state. Apr 30 13:31:32 athos openais[8569]: [CLM ] got nodejoin message 172.16.1.3 Apr 30 13:31:32 athos openais[8569]: [CLM ] got nodejoin message 172.16.1.1 Apr 30 13:31:32 athos openais[8569]: [CLM ] got nodejoin message 172.16.1.2 Apr 30 13:31:39 athos openais[8569]: [TOTEM] FAILED TO RECEIVE Apr 30 13:31:39 athos openais[8569]: [TOTEM] entering GATHER state from 6. Apr 30 13:31:39 athos openais[8569]: [TOTEM] Creating commit token because I am the rep. Apr 30 13:31:39 athos openais[8569]: [TOTEM] Saving state aru f high seq received f Apr 30 13:31:39 athos openais[8569]: [TOTEM] Storing new sequence id for ring 2754 Apr 30 13:31:39 athos openais[8569]: [TOTEM] entering COMMIT state. Apr 30 13:31:39 athos openais[8569]: [TOTEM] entering RECOVERY state. Apr 30 13:31:39 athos openais[8569]: [TOTEM] position [0] member 172.16.1.1: Apr 30 13:31:39 athos openais[8569]: [TOTEM] previous ring seq 10064 rep 172.16.1.1 Apr 30 13:31:39 athos openais[8569]: [TOTEM] aru f high delivered f received flag 1 Apr 30 13:31:39 athos openais[8569]: [TOTEM] position [1] member 172.16.1.2: Apr 30 13:31:39 athos openais[8569]: [TOTEM] previous ring seq 10064 rep 172.16.1.1 Apr 30 13:31:39 athos openais[8569]: [TOTEM] aru f high delivered f received flag 1 Apr 30 13:31:39 athos openais[8569]: [TOTEM] position [2] member 172.16.1.3: Apr 30 13:31:39 athos openais[8569]: [TOTEM] previous ring seq 10064 rep 172.16.1.1 Apr 30 13:31:39 athos openais[8569]: [TOTEM] aru b high delivered b received flag 0 Apr 30 13:31:39 athos openais[8569]: [TOTEM] copying all old ring messages from c-f. Apr 30 13:31:39 athos openais[8569]: [TOTEM] Originated 4 messages in RECOVERY. Apr 30 13:31:39 athos openais[8569]: [TOTEM] Originated for recovery: c d e f Apr 30 13:31:39 athos openais[8569]: [TOTEM] Not Originated for recovery: Apr 30 13:31:39 athos openais[8569]: [TOTEM] Sending initial ORF token Apr 30 13:31:39 athos openais[8569]: [CLM ] CLM CONFIGURATION CHANGE Apr 30 13:31:39 athos openais[8569]: [CLM ] New Configuration: Apr 30 13:31:39 athos openais[8569]: [CLM ] r(0) ip(172.16.1.1) Apr 30 13:31:39 athos openais[8569]: [CLM ] r(0) ip(172.16.1.2) Apr 30 13:31:39 athos openais[8569]: [CLM ] r(0) ip(172.16.1.3) Apr 30 13:31:39 athos openais[8569]: [CLM ] Members Left: Apr 30 13:31:39 athos openais[8569]: [CLM ] Members Joined: Apr 30 13:31:39 athos openais[8569]: [CLM ] CLM CONFIGURATION CHANGE Apr 30 13:31:39 athos openais[8569]: [CLM ] New Configuration: Apr 30 13:31:39 athos openais[8569]: [CLM ] r(0) ip(172.16.1.1) Apr 30 13:31:39 athos openais[8569]: [CLM ] r(0) ip(172.16.1.2) Apr 30 13:31:39 athos openais[8569]: [CLM ] r(0) ip(172.16.1.3) Apr 30 13:31:39 athos openais[8569]: [CLM ] Members Left: Apr 30 13:31:39 athos openais[8569]: [CLM ] Members Joined: Apr 30 13:31:39 athos openais[8569]: [SYNC ] This node is within the primary component and will provide service. Apr 30 13:31:39 athos openais[8569]: [TOTEM] entering OPERATIONAL state. Apr 30 13:31:39 athos openais[8569]: [TOTEM] Retransmit List: 9 a Apr 30 13:31:39 athos openais[8569]: [CLM ] got nodejoin message 172.16.1.1 Apr 30 13:31:39 athos openais[8569]: [CLM ] got nodejoin message 172.16.1.2 Apr 30 13:31:39 athos openais[8569]: [TOTEM] Retransmit List: 9 a Apr 30 13:31:39 athos last message repeated 13 times Apr 30 13:31:39 athos openais[8569]: [CLM ] got nodejoin message 172.16.1.3 Apr 30 13:31:39 athos openais[8569]: [TOTEM] Retransmit List: c d e f Apr 30 13:31:45 athos openais[8569]: [TOTEM] FAILED TO RECEIVE Apr 30 13:31:45 athos openais[8569]: [TOTEM] entering GATHER state from 6. Apr 30 13:31:45 athos openais[8569]: [TOTEM] Creating commit token because I am the rep. Apr 30 13:31:45 athos openais[8569]: [TOTEM] Saving state aru 1c high seq received 1c Apr 30 13:31:45 athos openais[8569]: [TOTEM] Storing new sequence id for ring 2758 Apr 30 13:31:45 athos openais[8569]: [TOTEM] entering COMMIT state. Apr 30 13:31:45 athos openais[8569]: [TOTEM] entering RECOVERY state. Apr 30 13:31:45 athos openais[8569]: [TOTEM] position [0] member 172.16.1.1: Apr 30 13:31:45 athos openais[8569]: [TOTEM] previous ring seq 10068 rep 172.16.1.1 Apr 30 13:31:45 athos openais[8569]: [TOTEM] aru 1c high delivered 1c received flag 1 Apr 30 13:31:45 athos openais[8569]: [TOTEM] position [1] member 172.16.1.2: Apr 30 13:31:45 athos openais[8569]: [TOTEM] previous ring seq 10068 rep 172.16.1.1 Apr 30 13:31:45 athos openais[8569]: [TOTEM] aru 1c high delivered 1c received flag 1 Apr 30 13:31:45 athos openais[8569]: [TOTEM] position [2] member 172.16.1.3: Apr 30 13:31:45 athos openais[8569]: [TOTEM] previous ring seq 10068 rep 172.16.1.1 Apr 30 13:31:45 athos openais[8569]: [TOTEM] aru 16 high delivered 16 received flag 0 Apr 30 13:31:45 athos openais[8569]: [TOTEM] copying all old ring messages from 17-1c. Apr 30 13:31:45 athos openais[8569]: [TOTEM] Originated 6 messages in RECOVERY. Apr 30 13:31:45 athos openais[8569]: [TOTEM] Originated for recovery: 17 18 19 1a 1b 1c Apr 30 13:31:45 athos openais[8569]: [TOTEM] Not Originated for recovery: Apr 30 13:31:45 athos openais[8569]: [TOTEM] Sending initial ORF token Apr 30 13:31:53 athos openais[8569]: [TOTEM] FAILED TO RECEIVE Apr 30 13:31:53 athos openais[8569]: [TOTEM] Restoring instance->my_aru 1c my high seq received 1c Apr 30 13:31:53 athos openais[8569]: [TOTEM] entering GATHER state from 6. Apr 30 13:31:53 athos openais[8569]: [TOTEM] Creating commit token because I am the rep. Apr 30 13:31:53 athos openais[8569]: [TOTEM] Storing new sequence id for ring 275c Apr 30 13:31:53 athos openais[8569]: [TOTEM] entering COMMIT state. Apr 30 13:31:53 athos openais[8569]: [TOTEM] entering RECOVERY state. Apr 30 13:31:53 athos openais[8569]: [TOTEM] position [0] member 172.16.1.1: Apr 30 13:31:53 athos openais[8569]: [TOTEM] previous ring seq 10068 rep 172.16.1.1 Apr 30 13:31:53 athos openais[8569]: [TOTEM] aru 1c high delivered 1c received flag 1 Apr 30 13:31:53 athos openais[8569]: [TOTEM] position [1] member 172.16.1.2: Apr 30 13:31:53 athos openais[8569]: [TOTEM] previous ring seq 10068 rep 172.16.1.1 Apr 30 13:31:53 athos openais[8569]: [TOTEM] aru 1c high delivered 1c received flag 1 Apr 30 13:31:53 athos openais[8569]: [TOTEM] position [2] member 172.16.1.3: Apr 30 13:31:53 athos openais[8569]: [TOTEM] previous ring seq 10068 rep 172.16.1.1 Apr 30 13:31:53 athos openais[8569]: [TOTEM] aru 16 high delivered 16 received flag 0 Apr 30 13:31:53 athos openais[8569]: [TOTEM] copying all old ring messages from 17-1c. Apr 30 13:31:53 athos openais[8569]: [TOTEM] Originated 6 messages in RECOVERY. Apr 30 13:31:53 athos openais[8569]: [TOTEM] Originated for recovery: 17 18 19 1a 1b 1c Apr 30 13:31:53 athos openais[8569]: [TOTEM] Not Originated for recovery: Apr 30 13:31:53 athos openais[8569]: [TOTEM] Sending initial ORF token Apr 30 13:31:53 athos openais[8569]: [CLM ] CLM CONFIGURATION CHANGE Apr 30 13:31:53 athos openais[8569]: [CLM ] New Configuration: Apr 30 13:31:53 athos openais[8569]: [CLM ] r(0) ip(172.16.1.1) Apr 30 13:31:53 athos openais[8569]: [CLM ] r(0) ip(172.16.1.2) Apr 30 13:31:53 athos openais[8569]: [CLM ] r(0) ip(172.16.1.3) Apr 30 13:31:53 athos openais[8569]: [CLM ] Members Left: Apr 30 13:31:53 athos openais[8569]: [CLM ] Members Joined: Apr 30 13:31:53 athos openais[8569]: [CLM ] CLM CONFIGURATION CHANGE Apr 30 13:31:53 athos openais[8569]: [CLM ] New Configuration: Apr 30 13:31:53 athos openais[8569]: [CLM ] r(0) ip(172.16.1.1) Apr 30 13:31:53 athos openais[8569]: [CLM ] r(0) ip(172.16.1.2) Apr 30 13:31:53 athos openais[8569]: [CLM ] r(0) ip(172.16.1.3) Apr 30 13:31:53 athos openais[8569]: [CLM ] Members Left: Apr 30 13:31:53 athos openais[8569]: [CLM ] Members Joined: Apr 30 13:31:53 athos openais[8569]: [SYNC ] This node is within the primary component and will provide service. Apr 30 13:31:53 athos openais[8569]: [TOTEM] entering OPERATIONAL state. Apr 30 13:32:00 athos openais[8569]: [TOTEM] entering GATHER state from 12. Apr 30 13:32:00 athos openais[8569]: [TOTEM] Creating commit token because I am the rep. Apr 30 13:32:00 athos openais[8569]: [TOTEM] Saving state aru d high seq received d Apr 30 13:32:00 athos openais[8569]: [TOTEM] Storing new sequence id for ring 2760 Apr 30 13:32:00 athos openais[8569]: [TOTEM] entering COMMIT state. Apr 30 13:32:00 athos openais[8569]: [TOTEM] entering RECOVERY state. Apr 30 13:32:00 athos openais[8569]: [TOTEM] position [0] member 172.16.1.1: Apr 30 13:32:00 athos openais[8569]: [TOTEM] previous ring seq 10076 rep 172.16.1.1 Apr 30 13:32:00 athos openais[8569]: [TOTEM] aru d high delivered d received flag 1 Apr 30 13:32:00 athos openais[8569]: [TOTEM] position [1] member 172.16.1.2: Apr 30 13:32:00 athos openais[8569]: [TOTEM] previous ring seq 10068 rep 172.16.1.1 Apr 30 13:32:00 athos openais[8569]: [TOTEM] aru 1c high delivered 1c received flag 1 Apr 30 13:32:00 athos openais[8569]: [TOTEM] position [2] member 172.16.1.3: Apr 30 13:32:00 athos openais[8569]: [TOTEM] previous ring seq 10068 rep 172.16.1.1 Apr 30 13:32:00 athos openais[8569]: [TOTEM] aru 16 high delivered 16 received flag 0 Apr 30 13:32:00 athos openais[8569]: [TOTEM] Did not need to originate any messages in recovery. Apr 30 13:32:00 athos openais[8569]: [TOTEM] Sending initial ORF token Apr 30 13:32:00 athos openais[8569]: [TOTEM] Retransmit List: 5 6 7 Apr 30 13:32:00 athos last message repeated 8 times Apr 30 13:32:00 athos openais[8569]: [TOTEM] Retransmit List: 5 6 Apr 30 13:32:08 athos last message repeated 39 times Apr 30 13:32:08 athos openais[8569]: [TOTEM] FAILED TO RECEIVE Apr 30 13:32:08 athos openais[8569]: [TOTEM] Restoring instance->my_aru d my high seq received d Apr 30 13:32:08 athos openais[8569]: [TOTEM] entering GATHER state from 6. Apr 30 13:32:08 athos openais[8569]: [TOTEM] Creating commit token because I am the rep. Apr 30 13:32:08 athos openais[8569]: [TOTEM] Storing new sequence id for ring 2764 Apr 30 13:32:08 athos openais[8569]: [TOTEM] entering COMMIT state. Apr 30 13:32:08 athos openais[8569]: [TOTEM] entering RECOVERY state. Apr 30 13:32:08 athos openais[8569]: [TOTEM] position [0] member 172.16.1.1: Apr 30 13:32:08 athos openais[8569]: [TOTEM] previous ring seq 10076 rep 172.16.1.1 Apr 30 13:32:08 athos openais[8569]: [TOTEM] aru d high delivered d received flag 1 Apr 30 13:32:08 athos openais[8569]: [TOTEM] position [1] member 172.16.1.2: Apr 30 13:32:08 athos openais[8569]: [TOTEM] previous ring seq 10080 rep 172.16.1.1 Apr 30 13:32:08 athos openais[8569]: [TOTEM] aru 8 high delivered 8 received flag 1 Apr 30 13:32:08 athos openais[8569]: [TOTEM] position [2] member 172.16.1.3: Apr 30 13:32:08 athos openais[8569]: [TOTEM] previous ring seq 10080 rep 172.16.1.1 Apr 30 13:32:08 athos openais[8569]: [TOTEM] aru 4 high delivered 4 received flag 0 Apr 30 13:32:08 athos openais[8569]: [TOTEM] Did not need to originate any messages in recovery. Apr 30 13:32:08 athos openais[8569]: [TOTEM] Sending initial ORF token Apr 30 13:32:09 athos last message repeated 2 times Apr 30 13:32:09 athos qdiskd[5311]: Node 2 shutdown Apr 30 13:32:09 athos openais[8569]: [TOTEM] Sending initial ORF token Apr 30 13:32:17 athos last message repeated 16 times Apr 30 13:32:18 athos openais[8569]: [TOTEM] Restoring instance->my_aru d my high seq received d Apr 30 13:32:18 athos openais[8569]: [TOTEM] Creating commit token because I am the rep. Apr 30 13:32:18 athos openais[8569]: [TOTEM] Storing new sequence id for ring 2768 Apr 30 13:32:18 athos openais[8569]: [TOTEM] entering COMMIT state. Apr 30 13:32:18 athos openais[8569]: [TOTEM] entering GATHER state from 14. Apr 30 13:32:22 athos openais[8569]: [TOTEM] entering GATHER state from 11. Apr 30 13:32:22 athos openais[8569]: [TOTEM] Creating commit token because I am the rep. Apr 30 13:32:22 athos openais[8569]: [TOTEM] Storing new sequence id for ring 276c Apr 30 13:32:22 athos openais[8569]: [TOTEM] entering COMMIT state. Apr 30 13:32:22 athos openais[8569]: [TOTEM] entering RECOVERY state. Apr 30 13:32:22 athos openais[8569]: [TOTEM] position [0] member 172.16.1.1: Apr 30 13:32:22 athos openais[8569]: [TOTEM] previous ring seq 10076 rep 172.16.1.1 Apr 30 13:32:22 athos openais[8569]: [TOTEM] aru d high delivered d received flag 1 Apr 30 13:32:22 athos openais[8569]: [TOTEM] position [1] member 172.16.1.3: Apr 30 13:32:22 athos openais[8569]: [TOTEM] previous ring seq 10080 rep 172.16.1.1 Apr 30 13:32:22 athos openais[8569]: [TOTEM] aru 4 high delivered 4 received flag 0 Apr 30 13:32:22 athos openais[8569]: [TOTEM] Did not need to originate any messages in recovery. Apr 30 13:32:22 athos openais[8569]: [TOTEM] Sending initial ORF token Apr 30 13:32:22 athos openais[8569]: [TOTEM] -not adding msg with seq no 7 Apr 30 13:32:22 athos kernel: dlm: closing connection to node 2 Apr 30 13:32:22 athos openais[8569]: [TOTEM] -not adding msg with seq no 8 Apr 30 13:32:22 athos openais[8569]: [CLM ] CLM CONFIGURATION CHANGE Apr 30 13:32:22 athos openais[8569]: [CLM ] New Configuration: Apr 30 13:32:22 athos openais[8569]: [CLM ] r(0) ip(172.16.1.1) Apr 30 13:32:22 athos openais[8569]: [CLM ] r(0) ip(172.16.1.3) Apr 30 13:32:22 athos openais[8569]: [CLM ] Members Left: Apr 30 13:32:22 athos openais[8569]: [CLM ] r(0) ip(172.16.1.2) Apr 30 13:32:22 athos openais[8569]: [CLM ] Members Joined: Apr 30 13:32:22 athos openais[8569]: [CLM ] CLM CONFIGURATION CHANGE Apr 30 13:32:22 athos openais[8569]: [CLM ] New Configuration: Apr 30 13:32:22 athos openais[8569]: [CLM ] r(0) ip(172.16.1.1) Apr 30 13:32:22 athos openais[8569]: [CLM ] r(0) ip(172.16.1.3) Apr 30 13:32:22 athos openais[8569]: [CLM ] Members Left: Apr 30 13:32:22 athos openais[8569]: [CLM ] Members Joined: Apr 30 13:32:22 athos openais[8569]: [SYNC ] This node is within the primary component and will provide service. Apr 30 13:32:22 athos openais[8569]: [TOTEM] entering OPERATIONAL state. Apr 30 13:32:22 athos openais[8569]: [TOTEM] Retransmit List: 3 4 Apr 30 13:32:22 athos openais[8569]: [CLM ] got nodejoin message 172.16.1.1 Apr 30 13:32:22 athos openais[8569]: [TOTEM] Retransmit List: 4 Apr 30 13:32:22 athos last message repeated 6 times Apr 30 13:32:23 athos openais[8569]: [CLM ] got nodejoin message 172.16.1.3 Apr 30 13:32:23 athos openais[8569]: [TOTEM] Retransmit List: 6 7 Apr 30 13:32:23 athos groupd[8604]: cpg_mcast_joined retry 100 Apr 30 13:32:23 athos groupd[8604]: cpg_mcast_joined retry 200 Apr 30 13:32:23 athos groupd[8604]: cpg_mcast_joined retry 300 Apr 30 13:32:23 athos groupd[8604]: cpg_mcast_joined retry 400 Apr 30 13:32:23 athos groupd[8604]: cpg_mcast_joined retry 500 Apr 30 13:32:24 athos groupd[8604]: cpg_mcast_joined retry 600 Apr 30 13:32:24 athos groupd[8604]: cpg_mcast_joined retry 700 Apr 30 13:32:24 athos groupd[8604]: cpg_mcast_joined retry 800 Apr 30 13:32:24 athos groupd[8604]: cpg_mcast_joined retry 900 Apr 30 13:32:24 athos groupd[8604]: cpg_mcast_joined retry 1000 Apr 30 13:32:25 athos groupd[8604]: cpg_mcast_joined retry 1100 Apr 30 13:32:25 athos groupd[8604]: cpg_mcast_joined retry 1200 Apr 30 13:32:25 athos groupd[8604]: cpg_mcast_joined retry 1300 Apr 30 13:32:25 athos groupd[8604]: cpg_mcast_joined retry 1400 Apr 30 13:32:25 athos groupd[8604]: cpg_mcast_joined retry 1500 Apr 30 13:32:26 athos groupd[8604]: cpg_mcast_joined retry 1600 Apr 30 13:32:26 athos groupd[8604]: cpg_mcast_joined retry 1700 Apr 30 13:32:26 athos groupd[8604]: cpg_mcast_joined retry 1800 Apr 30 13:32:26 athos groupd[8604]: cpg_mcast_joined retry 1900 Apr 30 13:32:26 athos groupd[8604]: cpg_mcast_joined retry 2000 Apr 30 13:32:27 athos groupd[8604]: cpg_mcast_joined retry 2100 Apr 30 13:32:27 athos groupd[8604]: cpg_mcast_joined retry 2200 Apr 30 13:32:27 athos groupd[8604]: cpg_mcast_joined retry 2300 Apr 30 13:32:27 athos groupd[8604]: cpg_mcast_joined retry 2400 Apr 30 13:32:27 athos groupd[8604]: cpg_mcast_joined retry 2500 Apr 30 13:32:28 athos groupd[8604]: cpg_mcast_joined retry 2600 Apr 30 13:32:28 athos groupd[8604]: cpg_mcast_joined retry 2700 Apr 30 13:32:28 athos groupd[8604]: cpg_mcast_joined retry 2800 Apr 30 13:32:28 athos groupd[8604]: cpg_mcast_joined retry 2900 Apr 30 13:32:28 athos groupd[8604]: cpg_mcast_joined retry 3000 Apr 30 13:32:29 athos groupd[8604]: cpg_mcast_joined retry 3100 Apr 30 13:32:29 athos groupd[8604]: cpg_mcast_joined retry 3200 Apr 30 13:32:29 athos groupd[8604]: cpg_mcast_joined retry 3300 Apr 30 13:32:29 athos openais[8569]: [TOTEM] FAILED TO RECEIVE Apr 30 13:32:29 athos openais[8569]: [TOTEM] entering GATHER state from 6. Apr 30 13:32:29 athos openais[8569]: [TOTEM] Creating commit token because I am the rep. Apr 30 13:32:29 athos openais[8569]: [TOTEM] Saving state aru 15 high seq received 15 Apr 30 13:32:29 athos openais[8569]: [TOTEM] Storing new sequence id for ring 2770 Apr 30 13:32:29 athos openais[8569]: [TOTEM] entering COMMIT state. Apr 30 13:32:29 athos openais[8569]: [TOTEM] entering RECOVERY state. Apr 30 13:32:29 athos openais[8569]: [TOTEM] position [0] member 172.16.1.1: Apr 30 13:32:29 athos openais[8569]: [TOTEM] previous ring seq 10092 rep 172.16.1.1 Apr 30 13:32:29 athos openais[8569]: [TOTEM] aru 15 high delivered 15 received flag 1 Apr 30 13:32:29 athos openais[8569]: [TOTEM] position [1] member 172.16.1.3: Apr 30 13:32:29 athos openais[8569]: [TOTEM] previous ring seq 10092 rep 172.16.1.1 Apr 30 13:32:29 athos openais[8569]: [TOTEM] aru 13 high delivered 13 received flag 0 Apr 30 13:32:29 athos openais[8569]: [TOTEM] copying all old ring messages from 14-15. Apr 30 13:32:29 athos openais[8569]: [TOTEM] Originated 2 messages in RECOVERY. Apr 30 13:32:29 athos openais[8569]: [TOTEM] Originated for recovery: 14 15 Apr 30 13:32:29 athos openais[8569]: [TOTEM] Not Originated for recovery: Apr 30 13:32:29 athos openais[8569]: [TOTEM] Sending initial ORF token Apr 30 13:32:29 athos openais[8569]: [CLM ] CLM CONFIGURATION CHANGE Apr 30 13:32:29 athos openais[8569]: [CLM ] New Configuration: Apr 30 13:32:29 athos openais[8569]: [CLM ] r(0) ip(172.16.1.1) Apr 30 13:32:29 athos openais[8569]: [CLM ] r(0) ip(172.16.1.3) Apr 30 13:32:29 athos openais[8569]: [CLM ] Members Left: Apr 30 13:32:29 athos openais[8569]: [CLM ] Members Joined: Apr 30 13:32:29 athos openais[8569]: [CLM ] CLM CONFIGURATION CHANGE Apr 30 13:32:29 athos openais[8569]: [CLM ] New Configuration: Apr 30 13:32:29 athos openais[8569]: [CLM ] r(0) ip(172.16.1.1) Apr 30 13:32:29 athos openais[8569]: [CLM ] r(0) ip(172.16.1.3) Apr 30 13:32:29 athos openais[8569]: [CLM ] Members Left: Apr 30 13:32:29 athos openais[8569]: [CLM ] Members Joined: Apr 30 13:32:29 athos openais[8569]: [SYNC ] This node is within the primary component and will provide service. Apr 30 13:32:29 athos openais[8569]: [TOTEM] entering OPERATIONAL state. Apr 30 13:32:29 athos openais[8569]: [CLM ] got nodejoin message 172.16.1.3 Apr 30 13:32:29 athos openais[8569]: [CLM ] got nodejoin message 172.16.1.1 Apr 30 13:32:29 athos openais[8569]: [CPG ] got joinlist message from node 1 Apr 30 13:32:29 athos openais[8569]: [TOTEM] Retransmit List: 16 Apr 30 13:32:29 athos last message repeated 36 times Apr 30 13:32:34 athos fenced[8611]: porthos-priv not a cluster member after 5 sec post_fail_delay Apr 30 13:32:34 athos fenced[8611]: fencing node "porthos-priv" Apr 30 13:32:40 athos ntpd[5785]: time reset -0.465908 s Apr 30 13:32:49 athos openais[8569]: [TOTEM] FAILED TO RECEIVE Apr 30 13:32:49 athos openais[8569]: [TOTEM] entering GATHER state from 6. Apr 30 13:32:49 athos openais[8569]: [TOTEM] Creating commit token because I am the rep. Apr 30 13:32:49 athos openais[8569]: [TOTEM] Saving state aru 19 high seq received 19 Apr 30 13:32:49 athos openais[8569]: [TOTEM] Storing new sequence id for ring 2774 Apr 30 13:32:49 athos openais[8569]: [TOTEM] entering COMMIT state. Apr 30 13:32:49 athos openais[8569]: [TOTEM] entering RECOVERY state. Apr 30 13:32:49 athos openais[8569]: [TOTEM] position [0] member 172.16.1.1: Apr 30 13:32:49 athos openais[8569]: [TOTEM] previous ring seq 10096 rep 172.16.1.1 Apr 30 13:32:49 athos openais[8569]: [TOTEM] aru 19 high delivered 19 received flag 1 Apr 30 13:32:49 athos openais[8569]: [TOTEM] position [1] member 172.16.1.3: Apr 30 13:32:49 athos openais[8569]: [TOTEM] previous ring seq 10096 rep 172.16.1.1 Apr 30 13:32:49 athos openais[8569]: [TOTEM] aru 17 high delivered 17 received flag 0 Apr 30 13:32:49 athos openais[8569]: [TOTEM] copying all old ring messages from 18-19. Apr 30 13:32:49 athos openais[8569]: [TOTEM] Originated 2 messages in RECOVERY. Apr 30 13:32:49 athos openais[8569]: [TOTEM] Originated for recovery: 18 19 Apr 30 13:32:49 athos openais[8569]: [TOTEM] Not Originated for recovery: Apr 30 13:32:49 athos openais[8569]: [TOTEM] Sending initial ORF token Apr 30 13:32:49 athos openais[8569]: [CLM ] CLM CONFIGURATION CHANGE Apr 30 13:32:49 athos openais[8569]: [CLM ] New Configuration: Apr 30 13:32:49 athos openais[8569]: [CLM ] r(0) ip(172.16.1.1) Apr 30 13:32:49 athos openais[8569]: [CLM ] r(0) ip(172.16.1.3) Apr 30 13:32:49 athos openais[8569]: [CLM ] Members Left: Apr 30 13:32:49 athos openais[8569]: [CLM ] Members Joined: Apr 30 13:32:49 athos openais[8569]: [CLM ] CLM CONFIGURATION CHANGE Apr 30 13:32:49 athos openais[8569]: [CLM ] New Configuration: Apr 30 13:32:49 athos openais[8569]: [CLM ] r(0) ip(172.16.1.1) Apr 30 13:32:49 athos openais[8569]: [CLM ] r(0) ip(172.16.1.3) Apr 30 13:32:49 athos openais[8569]: [CLM ] Members Left: Apr 30 13:32:49 athos openais[8569]: [CLM ] Members Joined: Apr 30 13:32:49 athos openais[8569]: [SYNC ] This node is within the primary component and will provide service. Apr 30 13:32:49 athos openais[8569]: [TOTEM] entering OPERATIONAL state. Apr 30 13:32:49 athos openais[8569]: [CLM ] got nodejoin message 172.16.1.3 Apr 30 13:32:49 athos openais[8569]: [CLM ] got nodejoin message 172.16.1.1 Apr 30 13:32:49 athos openais[8569]: [TOTEM] Retransmit List: e Apr 30 13:32:49 athos openais[8569]: [TOTEM] Retransmit List: e 10 Apr 30 13:32:49 athos last message repeated 25 times Apr 30 13:32:49 athos openais[8569]: [TOTEM] Retransmit List: 12 Apr 30 13:32:49 athos openais[8569]: [CPG ] got joinlist message from node 1 Apr 30 13:34:00 athos ccsd[8558]: Attempt to close an unopened CCS descriptor (2250). Apr 30 13:34:00 athos ccsd[8558]: Error while processing disconnect: Invalid request descriptor Apr 30 13:34:00 athos fenced[8611]: fence "porthos-priv" success