Jul 11 20:13:55 pve02 qm[3851801]: end task UPID:pve02:003AC61E:0424DE4A:5D276E52:qmstart:104:root@pam: OK Jul 11 20:13:55 pve02 systemd[1]: Started Session 18487 of user root. Jul 11 20:14:00 pve02 systemd[1]: Starting Proxmox VE replication runner... Jul 11 20:14:08 pve02 corosync[4638]: notice [TOTEM ] A new membership (172.16.25.10:13760) was formed. Members Jul 11 20:14:08 pve02 corosync[4638]: [TOTEM ] A new membership (172.16.25.10:13760) was formed. Members Jul 11 20:14:10 pve02 pmxcfs[4479]: [status] notice: cpg_send_message retry 10 Jul 11 20:14:11 pve02 pmxcfs[4479]: [status] notice: cpg_send_message retry 20 Jul 11 20:14:12 pve02 pmxcfs[4479]: [status] notice: cpg_send_message retry 30 Jul 11 20:14:13 pve02 pmxcfs[4479]: [status] notice: cpg_send_message retry 40 Jul 11 20:14:14 pve02 pmxcfs[4479]: [status] notice: cpg_send_message retry 50 Jul 11 20:14:15 pve02 pmxcfs[4479]: [status] notice: cpg_send_message retry 60 Jul 11 20:14:16 pve02 pmxcfs[4479]: [status] notice: cpg_send_message retry 70 Jul 11 20:14:17 pve02 pmxcfs[4479]: [status] notice: cpg_send_message retry 80 Jul 11 20:14:18 pve02 pmxcfs[4479]: [status] notice: cpg_send_message retry 90 Jul 11 20:14:19 pve02 pmxcfs[4479]: [status] notice: cpg_send_message retry 100 Jul 11 20:14:19 pve02 pmxcfs[4479]: [status] notice: cpg_send_message retried 100 times Jul 11 20:14:19 pve02 pmxcfs[4479]: [status] crit: cpg_send_message failed: 6 Jul 11 20:14:19 pve02 pve-firewall[4941]: firewall update time (6.042 seconds) Jul 11 20:14:20 pve02 pmxcfs[4479]: [status] notice: cpg_send_message retry 10 Jul 11 20:14:21 pve02 pmxcfs[4479]: [status] notice: cpg_send_message retry 20 Jul 11 20:14:22 pve02 pmxcfs[4479]: [status] notice: cpg_send_message retry 30 Jul 11 20:14:23 pve02 pmxcfs[4479]: [status] notice: cpg_send_message retry 40 Jul 11 20:14:24 pve02 pmxcfs[4479]: [status] notice: cpg_send_message retry 50 Jul 11 20:14:25 pve02 pmxcfs[4479]: [status] notice: cpg_send_message retry 60 Jul 11 20:14:26 pve02 pmxcfs[4479]: [status] notice: cpg_send_message retry 70 Jul 11 20:14:27 pve02 pmxcfs[4479]: [status] notice: cpg_send_message retry 80 Jul 11 20:14:28 pve02 pmxcfs[4479]: [status] notice: cpg_send_message retry 90 Jul 11 20:14:29 pve02 pmxcfs[4479]: [status] notice: cpg_send_message retry 100 Jul 11 20:14:29 pve02 pmxcfs[4479]: [status] notice: cpg_send_message retried 100 times Jul 11 20:14:29 pve02 pmxcfs[4479]: [status] crit: cpg_send_message failed: 6 Jul 11 20:14:29 pve02 pve-firewall[4941]: firewall update time (6.007 seconds) Jul 11 20:14:30 pve02 pmxcfs[4479]: [status] notice: cpg_send_message retry 10 Jul 11 20:14:30 pve02 corosync[4638]: notice [TOTEM ] A new membership (172.16.25.10:13788) was formed. Members joined: 3 left: 3 Jul 11 20:14:30 pve02 corosync[4638]: notice [TOTEM ] Failed to receive the leave message. failed: 3 Jul 11 20:14:30 pve02 corosync[4638]: [TOTEM ] A new membership (172.16.25.10:13788) was formed. Members joined: 3 left: 3 Jul 11 20:14:30 pve02 corosync[4638]: [TOTEM ] Failed to receive the leave message. failed: 3 Jul 11 20:14:31 pve02 pmxcfs[4479]: [status] notice: cpg_send_message retry 20 Jul 11 20:14:32 pve02 pmxcfs[4479]: [status] notice: cpg_send_message retry 30 Jul 11 20:14:33 pve02 pmxcfs[4479]: [status] notice: cpg_send_message retry 40 Jul 11 20:14:34 pve02 pmxcfs[4479]: [status] notice: cpg_send_message retry 50 Jul 11 20:14:35 pve02 pmxcfs[4479]: [status] notice: cpg_send_message retry 60 Jul 11 20:14:36 pve02 pmxcfs[4479]: [status] notice: cpg_send_message retry 70 Jul 11 20:14:37 pve02 pmxcfs[4479]: [status] notice: cpg_send_message retry 80 Jul 11 20:14:38 pve02 pmxcfs[4479]: [status] notice: cpg_send_message retry 90 Jul 11 20:14:39 pve02 pmxcfs[4479]: [status] notice: cpg_send_message retry 100 Jul 11 20:14:39 pve02 pmxcfs[4479]: [status] notice: cpg_send_message retried 100 times Jul 11 20:14:39 pve02 pmxcfs[4479]: [status] crit: cpg_send_message failed: 6 Jul 11 20:14:40 pve02 pmxcfs[4479]: [status] notice: cpg_send_message retry 10 Jul 11 20:14:41 pve02 pmxcfs[4479]: [status] notice: cpg_send_message retry 20 Jul 11 20:14:42 pve02 pmxcfs[4479]: [status] notice: cpg_send_message retry 30 Jul 11 20:14:43 pve02 pmxcfs[4479]: [status] notice: cpg_send_message retry 40 Jul 11 20:14:44 pve02 pmxcfs[4479]: [status] notice: cpg_send_message retry 50 Jul 11 20:14:45 pve02 pmxcfs[4479]: [status] notice: cpg_send_message retry 60 Jul 11 20:14:46 pve02 pmxcfs[4479]: [status] notice: cpg_send_message retry 70 Jul 11 20:14:47 pve02 pmxcfs[4479]: [status] notice: cpg_send_message retry 80 Jul 11 20:14:48 pve02 pmxcfs[4479]: [status] notice: cpg_send_message retry 90 Jul 11 20:14:49 pve02 pmxcfs[4479]: [status] notice: cpg_send_message retry 100 Jul 11 20:14:49 pve02 pmxcfs[4479]: [status] notice: cpg_send_message retried 100 times Jul 11 20:14:49 pve02 pmxcfs[4479]: [status] crit: cpg_send_message failed: 6 Jul 11 20:14:49 pve02 corosync[4638]: notice [TOTEM ] A new membership (172.16.25.10:13808) was formed. Members joined: 3 left: 3 Jul 11 20:14:49 pve02 corosync[4638]: notice [TOTEM ] Failed to receive the leave message. failed: 3 Jul 11 20:14:49 pve02 corosync[4638]: [TOTEM ] A new membership (172.16.25.10:13808) was formed. Members joined: 3 left: 3 Jul 11 20:14:49 pve02 corosync[4638]: [TOTEM ] Failed to receive the leave message. failed: 3 Jul 11 20:14:49 pve02 corosync[4638]: notice [TOTEM ] Retransmit List: 1 2 Jul 11 20:14:49 pve02 corosync[4638]: [TOTEM ] Retransmit List: 1 2 Jul 11 20:14:49 pve02 corosync[4638]: warning [CPG ] downlist left_list: 2 received Jul 11 20:14:49 pve02 corosync[4638]: warning [CPG ] downlist left_list: 1 received Jul 11 20:14:49 pve02 corosync[4638]: warning [CPG ] downlist left_list: 1 received Jul 11 20:14:49 pve02 corosync[4638]: [CPG ] downlist left_list: 2 received Jul 11 20:14:49 pve02 corosync[4638]: [CPG ] downlist left_list: 1 received Jul 11 20:14:49 pve02 corosync[4638]: notice [QUORUM] Members[3]: 3 2 1 Jul 11 20:14:49 pve02 corosync[4638]: notice [MAIN ] Completed service synchronization, ready to provide service. Jul 11 20:14:49 pve02 corosync[4638]: [CPG ] downlist left_list: 1 received Jul 11 20:14:49 pve02 pmxcfs[4479]: [dcdb] notice: members: 1/4892, 2/4479 Jul 11 20:14:49 pve02 pmxcfs[4479]: [dcdb] notice: starting data syncronisation Jul 11 20:14:49 pve02 pmxcfs[4479]: [dcdb] notice: members: 1/4892, 2/4479, 3/1880 Jul 11 20:14:49 pve02 pmxcfs[4479]: [status] notice: members: 1/4892, 2/4479 Jul 11 20:14:49 pve02 pmxcfs[4479]: [status] notice: starting data syncronisation Jul 11 20:14:49 pve02 pmxcfs[4479]: [status] notice: members: 1/4892, 2/4479, 3/1880 Jul 11 20:14:49 pve02 corosync[4638]: [QUORUM] Members[3]: 3 2 1 Jul 11 20:14:49 pve02 corosync[4638]: [MAIN ] Completed service synchronization, ready to provide service. Jul 11 20:14:49 pve02 pve-firewall[4941]: firewall update time (16.095 seconds) Jul 11 20:14:49 pve02 pmxcfs[4479]: [status] notice: cpg_send_message retried 1 times Jul 11 20:14:49 pve02 pmxcfs[4479]: [dcdb] notice: received sync request (epoch 1/4892/00000008) Jul 11 20:14:49 pve02 pmxcfs[4479]: [dcdb] notice: received sync request (epoch 1/4892/00000009) Jul 11 20:14:49 pve02 pmxcfs[4479]: [status] notice: received sync request (epoch 1/4892/00000008) Jul 11 20:14:49 pve02 pmxcfs[4479]: [status] notice: received sync request (epoch 1/4892/00000009) Jul 11 20:14:49 pve02 pmxcfs[4479]: [dcdb] notice: received all states Jul 11 20:14:49 pve02 pmxcfs[4479]: [dcdb] notice: leader is 1/4892 Jul 11 20:14:49 pve02 pmxcfs[4479]: [dcdb] notice: synced members: 1/4892, 2/4479, 3/1880 Jul 11 20:14:49 pve02 pmxcfs[4479]: [dcdb] notice: all data is up to date Jul 11 20:14:49 pve02 pmxcfs[4479]: [dcdb] notice: dfsm_deliver_queue: queue length 3 Jul 11 20:14:49 pve02 pvestatd[4957]: status update time (40.255 seconds) Jul 11 20:14:49 pve02 pmxcfs[4479]: [status] notice: received all states Jul 11 20:14:49 pve02 pmxcfs[4479]: [status] notice: all data is up to date Jul 11 20:14:49 pve02 pmxcfs[4479]: [status] notice: dfsm_deliver_queue: queue length 44 Jul 11 20:14:49 pve02 systemd[1]: Started Proxmox VE replication runner. Jul 11 20:14:49 pve02 pmxcfs[4479]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve03/BackUp: -1 Jul 11 20:14:49 pve02 pmxcfs[4479]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve03/local: -1 Jul 11 20:14:49 pve02 pmxcfs[4479]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve03/ssd-data: -1 Jul 11 20:14:49 pve02 pmxcfs[4479]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve03/zfs-data: -1 Jul 11 20:14:49 pve02 pmxcfs[4479]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve01/local: -1 Jul 11 20:14:49 pve02 pmxcfs[4479]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve01/BackUp: -1 Jul 11 20:14:49 pve02 pmxcfs[4479]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve01/pve-data: -1 Jul 11 20:14:51 pve02 kernel: [695319.153503] vmbr0: port 2(tap104i0) entered disabled state Jul 11 20:14:51 pve02 systemd[1]: Started Session 18488 of user root. Jul 11 20:14:52 pve02 zed: eid=47672 class=history_event pool_guid=0xC7636CE199EB7385 Jul 11 20:14:53 pve02 systemd[1]: Started Session 18489 of user root. Jul 11 20:14:53 pve02 zed: eid=47673 class=history_event pool_guid=0xC7636CE199EB7385 Jul 11 20:14:54 pve02 systemd[1]: Started Session 18490 of user root. Jul 11 20:14:54 pve02 pve-ha-lrm[5037]: loop take too long (56 seconds) Jul 11 20:14:55 pve02 qm[4009305]: starting task UPID:pve02:003D2D5F:0424F61D:5D276E8F:qmstop:104:root@pam: Jul 11 20:14:55 pve02 qm[4009311]: stop VM 104: UPID:pve02:003D2D5F:0424F61D:5D276E8F:qmstop:104:root@pam: Jul 11 20:14:55 pve02 qm[4009305]: end task UPID:pve02:003D2D5F:0424F61D:5D276E8F:qmstop:104:root@pam: OK