01/02/2021 19:36:34 INFO Start settings IPs 01/02/2021 19:41:58 INFO Start settings IPs 01/02/2021 19:53:26 INFO Start settings IPs 01/02/2021 07:55:09 ERROR Config file error. The PetaSAN os maybe just installed. Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/cluster/deploy.py", line 69, in get_node_status node_name = config.get_node_info().name File "/usr/lib/python3/dist-packages/PetaSAN/core/cluster/configuration.py", line 99, in get_node_info with open(config.get_node_info_file_path(), 'r') as f: FileNotFoundError: [Errno 2] No such file or directory: '/opt/petasan/config/node_info.json' 01/02/2021 07:55:35 INFO Created keys for cluster petasan 01/02/2021 07:55:35 INFO Created cluster file and set cluster name to petasan 01/02/2021 07:55:35 INFO password set successfully. 01/02/2021 07:55:44 INFO Updated cluster interface successfully. 01/02/2021 07:56:24 INFO 100 01/02/2021 07:56:24 INFO Updated cluster network successfully. 01/02/2021 07:57:44 INFO Updated cluster tuning successfully. 01/02/2021 07:57:44 INFO Current tuning configurations saved. 01/02/2021 07:58:46 ERROR 400 Bad Request: The browser (or proxy) sent a request that this server could not understand. 01/02/2021 07:58:46 ERROR 400 Bad Request: The browser (or proxy) sent a request that this server could not understand. 01/02/2021 07:58:46 ERROR 400 Bad Request: The browser (or proxy) sent a request that this server could not understand. 01/02/2021 07:58:46 ERROR 400 Bad Request: The browser (or proxy) sent a request that this server could not understand. 01/02/2021 07:58:46 INFO Set node role completed successfully. 01/02/2021 07:58:47 INFO Set node info completed successfully. 01/02/2021 07:59:00 INFO Node 1 added, cluster requires 2 other nodes to build. 01/02/2021 07:59:00 INFO Run post deploy script. 01/02/2021 08:43:59 ERROR Cluster is not completed, PetaSAN will check node join status. Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/cluster/deploy.py", line 78, in get_node_status raise Exception("Cluster is not completed, PetaSAN will check node join status.") Exception: Cluster is not completed, PetaSAN will check node join status. 01/02/2021 10:42:08 INFO Start settings IPs 01/02/2021 11:58:31 INFO files_sync.py process is 1095 01/02/2021 11:58:31 INFO files_sync.py process stopped 01/02/2021 11:58:31 INFO iscsi_service.py process is 1097 01/02/2021 11:58:31 INFO iscsi_service.py process stopped 01/02/2021 11:58:31 INFO admin.py process is 1099 01/02/2021 11:58:31 INFO admin.py process stopped 01/02/2021 11:58:32 INFO Stopping ceph services 01/02/2021 11:58:32 INFO Start cleaning config files 01/02/2021 11:58:33 INFO Starting ceph services 01/02/2021 11:58:34 INFO delete /opt/petasan/config/etc/consul.d 01/02/2021 11:58:34 INFO delete /opt/petasan/config/var/consul 01/02/2021 11:58:36 INFO str_start_command: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consul agent -raft-protocol 2 -config-dir /opt/petasan/config/etc/consul.d/server -bind 172.16.0.151 -retry-join 172.16.0.152 -retry-join 172.16.0.153 01/02/2021 12:01:19 ERROR Cluster is not completed, PetaSAN will check node join status. Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/cluster/deploy.py", line 78, in get_node_status raise Exception("Cluster is not completed, PetaSAN will check node join status.") Exception: Cluster is not completed, PetaSAN will check node join status. 01/02/2021 12:01:24 INFO create_mgr() fresh install 01/02/2021 12:01:24 INFO create_mgr() started 01/02/2021 12:01:24 INFO create_mgr() cmd : mkdir -p /var/lib/ceph/mgr/ceph-peta1 01/02/2021 12:01:24 INFO create_mgr() cmd : ceph --cluster ceph auth get-or-create mgr.peta1 mon 'allow profile mgr' osd 'allow *' mds 'allow *' -o /var/lib/ceph/mgr/ceph-peta1/keyring 01/02/2021 12:01:36 INFO create_mgr() ended successfully 01/02/2021 12:01:36 INFO create_mds() started 01/02/2021 12:01:36 INFO create_mds() cmd : mkdir -p /var/lib/ceph/mds/ceph-peta1 01/02/2021 12:01:37 INFO create_mds() ended successfully 01/02/2021 12:03:58 INFO Start cleaning disk : sdf 01/02/2021 12:04:02 INFO Executing : wipefs --all /dev/sdf1 01/02/2021 12:04:02 INFO Executing : dd if=/dev/zero of=/dev/sdf1 bs=1M count=20 oflag=direct,dsync >/dev/null 2>&1 01/02/2021 12:04:02 ERROR Error executing : dd if=/dev/zero of=/dev/sdf1 bs=1M count=20 oflag=direct,dsync >/dev/null 2>&1 01/02/2021 12:04:05 INFO Executing : wipefs --all /dev/sdf2 01/02/2021 12:04:05 INFO Executing : dd if=/dev/zero of=/dev/sdf2 bs=1M count=20 oflag=direct,dsync >/dev/null 2>&1 01/02/2021 12:04:05 INFO Executing : wipefs --all /dev/sdf 01/02/2021 12:04:05 INFO Executing : dd if=/dev/zero of=/dev/sdf bs=1M count=20 oflag=direct,dsync >/dev/null 2>&1 01/02/2021 12:04:05 INFO Executing : parted -s /dev/sdf mklabel gpt 01/02/2021 12:04:05 INFO Executing : partprobe /dev/sdf 01/02/2021 12:04:08 INFO Start cleaning disk : sdf 01/02/2021 12:04:08 INFO Executing : wipefs --all /dev/sdf 01/02/2021 12:04:08 INFO Executing : dd if=/dev/zero of=/dev/sdf bs=1M count=20 oflag=direct,dsync >/dev/null 2>&1 01/02/2021 12:04:08 INFO Executing : parted -s /dev/sdf mklabel gpt 01/02/2021 12:04:08 INFO Executing : partprobe /dev/sdf 01/02/2021 12:04:13 INFO Start cleaning disk : sda 01/02/2021 12:04:13 INFO Executing : wipefs --all /dev/sda 01/02/2021 12:04:13 INFO Executing : dd if=/dev/zero of=/dev/sda bs=1M count=20 oflag=direct,dsync >/dev/null 2>&1 01/02/2021 12:04:14 INFO Executing : parted -s /dev/sda mklabel gpt 01/02/2021 12:04:14 INFO Executing : partprobe /dev/sda 01/02/2021 12:04:19 INFO Start cleaning disk : sdf 01/02/2021 12:04:22 INFO Executing : wipefs --all /dev/sdf1 01/02/2021 12:04:22 INFO Executing : dd if=/dev/zero of=/dev/sdf1 bs=1M count=20 oflag=direct,dsync >/dev/null 2>&1 01/02/2021 12:04:22 INFO Executing : wipefs --all /dev/sdf 01/02/2021 12:04:22 INFO Executing : dd if=/dev/zero of=/dev/sdf bs=1M count=20 oflag=direct,dsync >/dev/null 2>&1 01/02/2021 12:04:23 INFO Executing : parted -s /dev/sdf mklabel gpt 01/02/2021 12:04:23 INFO Executing : partprobe /dev/sdf 01/02/2021 12:04:26 INFO Start prepare bluestore OSD : sda 01/02/2021 12:04:26 INFO Running command: /sbin/lvs --noheadings --readonly --separator=";" -a -o lv_tags,lv_path,lv_name,vg_name,lv_uuid,lv_size 01/02/2021 12:04:26 INFO Creating data partition num 1 size 0 on /dev/sda 01/02/2021 12:04:27 INFO Calling partprobe on sda device 01/02/2021 12:04:27 INFO Executing partprobe /dev/sda 01/02/2021 12:04:27 INFO Calling udevadm on sda device 01/02/2021 12:04:27 INFO Executing udevadm settle --timeout 30 01/02/2021 12:04:31 INFO Start remove osd.3 from crush map 01/02/2021 12:04:33 INFO osd.3 is removed from crush map 01/02/2021 12:04:33 INFO creating new journal partition. 01/02/2021 12:04:34 INFO Creating journal-db-3-c74c6e06 partition num 1 size 61440 on /dev/sdf 01/02/2021 12:04:35 INFO Calling partprobe on sdf device 01/02/2021 12:04:35 INFO Executing partprobe /dev/sdf 01/02/2021 12:04:35 INFO Calling udevadm on sdf device 01/02/2021 12:04:35 INFO Executing udevadm settle --timeout 30 01/02/2021 12:04:38 INFO Starting : ceph-volume --log-path /opt/petasan/log lvm prepare --bluestore --data /dev/sda1 --block.db /dev/sdf1 01/02/2021 12:04:44 INFO starting activate osd. 01/02/2021 12:04:45 INFO Starting : ceph-volume --log-path /opt/petasan/log lvm activate --bluestore 3 0db3b469-bcee-4c49-9d80-5b3eac37fda6 01/02/2021 12:04:45 INFO Successfully executed ceph-volume prepare for sda 01/02/2021 12:04:45 INFO Start cleaning disk : sdb 01/02/2021 12:04:46 INFO Executing : wipefs --all /dev/sdb 01/02/2021 12:04:46 INFO Executing : dd if=/dev/zero of=/dev/sdb bs=1M count=20 oflag=direct,dsync >/dev/null 2>&1 01/02/2021 12:04:47 INFO Executing : parted -s /dev/sdb mklabel gpt 01/02/2021 12:04:47 INFO Executing : partprobe /dev/sdb 01/02/2021 12:04:52 INFO Start prepare bluestore OSD : sdb 01/02/2021 12:04:52 INFO Running command: /sbin/lvs --noheadings --readonly --separator=";" -a -o lv_tags,lv_path,lv_name,vg_name,lv_uuid,lv_size 01/02/2021 12:04:52 INFO stdout ceph.block_device=/dev/ceph-e72ec456-4943-4572-8b6f-f20fa4b85417/osd-block-0db3b469-bcee-4c49-9d80-5b3eac37fda6,ceph.block_uuid=5f6FRj-Z1M0-t4Wo-gyex-cFvy-qvXR-ogG1ST,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=c74c6e06-673f-4d0c-8235-563559e08858,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/sdf1,ceph.db_uuid=2fbde886-866b-415c-a09f-bca4a37d32ae,ceph.encrypted=0,ceph.osd_fsid=0db3b469-bcee-4c49-9d80-5b3eac37fda6,ceph.osd_id=3,ceph.osdspec_affinity=,ceph.type=block,ceph.vdo=0";"/dev/ceph-e72ec456-4943-4572-8b6f-f20fa4b85417/osd-block-0db3b469-bcee-4c49-9d80-5b3eac37fda6";"osd-block-0db3b469-bcee-4c49-9d80-5b3eac37fda6";"ceph-e72ec456-4943-4572-8b6f-f20fa4b85417";"5f6FRj-Z1M0-t4Wo-gyex-cFvy-qvXR-ogG1ST";"<1.82t 01/02/2021 12:04:52 INFO Creating data partition num 1 size 0 on /dev/sdb 01/02/2021 12:04:53 INFO Calling partprobe on sdb device 01/02/2021 12:04:53 INFO Executing partprobe /dev/sdb 01/02/2021 12:04:53 INFO Calling udevadm on sdb device 01/02/2021 12:04:53 INFO Executing udevadm settle --timeout 30 01/02/2021 12:04:58 INFO Start remove osd.4 from crush map 01/02/2021 12:05:00 INFO osd.4 is removed from crush map 01/02/2021 12:05:00 INFO creating new journal partition. 01/02/2021 12:05:01 INFO Creating journal-db-4-c74c6e06 partition num 2 size 61440 on /dev/sdf 01/02/2021 12:05:02 INFO Calling partprobe on sdf device 01/02/2021 12:05:02 INFO Executing partprobe /dev/sdf 01/02/2021 12:05:02 INFO Calling udevadm on sdf device 01/02/2021 12:05:02 INFO Executing udevadm settle --timeout 30 01/02/2021 12:05:05 INFO Starting : ceph-volume --log-path /opt/petasan/log lvm prepare --bluestore --data /dev/sdb1 --block.db /dev/sdf2 01/02/2021 12:05:11 INFO starting activate osd. 01/02/2021 12:05:12 INFO Starting : ceph-volume --log-path /opt/petasan/log lvm activate --bluestore 4 12bc918a-066d-4340-a0d4-9b61f11bf3c5 01/02/2021 12:05:13 INFO Successfully executed ceph-volume prepare for sdb 01/02/2021 12:05:13 INFO Start cleaning disk : sdc 01/02/2021 12:05:13 INFO Executing : wipefs --all /dev/sdc 01/02/2021 12:05:13 INFO Executing : dd if=/dev/zero of=/dev/sdc bs=1M count=20 oflag=direct,dsync >/dev/null 2>&1 01/02/2021 12:05:14 INFO Executing : parted -s /dev/sdc mklabel gpt 01/02/2021 12:05:14 INFO Executing : partprobe /dev/sdc 01/02/2021 12:05:19 INFO Start prepare bluestore OSD : sdc 01/02/2021 12:05:19 INFO Running command: /sbin/lvs --noheadings --readonly --separator=";" -a -o lv_tags,lv_path,lv_name,vg_name,lv_uuid,lv_size 01/02/2021 12:05:19 INFO stdout ceph.block_device=/dev/ceph-847b13a2-67ce-4374-b26b-f42d81ceb154/osd-block-12bc918a-066d-4340-a0d4-9b61f11bf3c5,ceph.block_uuid=ai28RZ-QoHE-N46c-OCSe-0dNf-VpAI-rMBDPM,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=c74c6e06-673f-4d0c-8235-563559e08858,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/sdf2,ceph.db_uuid=2bff43e1-8653-4404-a988-7314f0973333,ceph.encrypted=0,ceph.osd_fsid=12bc918a-066d-4340-a0d4-9b61f11bf3c5,ceph.osd_id=4,ceph.osdspec_affinity=,ceph.type=block,ceph.vdo=0";"/dev/ceph-847b13a2-67ce-4374-b26b-f42d81ceb154/osd-block-12bc918a-066d-4340-a0d4-9b61f11bf3c5";"osd-block-12bc918a-066d-4340-a0d4-9b61f11bf3c5";"ceph-847b13a2-67ce-4374-b26b-f42d81ceb154";"ai28RZ-QoHE-N46c-OCSe-0dNf-VpAI-rMBDPM";"<1.82t 01/02/2021 12:05:19 INFO stdout ceph.block_device=/dev/ceph-e72ec456-4943-4572-8b6f-f20fa4b85417/osd-block-0db3b469-bcee-4c49-9d80-5b3eac37fda6,ceph.block_uuid=5f6FRj-Z1M0-t4Wo-gyex-cFvy-qvXR-ogG1ST,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=c74c6e06-673f-4d0c-8235-563559e08858,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/sdf1,ceph.db_uuid=2fbde886-866b-415c-a09f-bca4a37d32ae,ceph.encrypted=0,ceph.osd_fsid=0db3b469-bcee-4c49-9d80-5b3eac37fda6,ceph.osd_id=3,ceph.osdspec_affinity=,ceph.type=block,ceph.vdo=0";"/dev/ceph-e72ec456-4943-4572-8b6f-f20fa4b85417/osd-block-0db3b469-bcee-4c49-9d80-5b3eac37fda6";"osd-block-0db3b469-bcee-4c49-9d80-5b3eac37fda6";"ceph-e72ec456-4943-4572-8b6f-f20fa4b85417";"5f6FRj-Z1M0-t4Wo-gyex-cFvy-qvXR-ogG1ST";"<1.82t 01/02/2021 12:05:19 INFO Creating data partition num 1 size 0 on /dev/sdc 01/02/2021 12:05:20 INFO Calling partprobe on sdc device 01/02/2021 12:05:20 INFO Executing partprobe /dev/sdc 01/02/2021 12:05:20 INFO Calling udevadm on sdc device 01/02/2021 12:05:20 INFO Executing udevadm settle --timeout 30 01/02/2021 12:05:24 INFO Start remove osd.5 from crush map 01/02/2021 12:05:26 INFO osd.5 is removed from crush map 01/02/2021 12:05:26 INFO creating new journal partition. 01/02/2021 12:05:27 INFO Creating journal-db-5-c74c6e06 partition num 3 size 61440 on /dev/sdf 01/02/2021 12:05:28 INFO Calling partprobe on sdf device 01/02/2021 12:05:28 INFO Executing partprobe /dev/sdf 01/02/2021 12:05:28 INFO Calling udevadm on sdf device 01/02/2021 12:05:28 INFO Executing udevadm settle --timeout 30 01/02/2021 12:05:31 INFO Starting : ceph-volume --log-path /opt/petasan/log lvm prepare --bluestore --data /dev/sdc1 --block.db /dev/sdf3 01/02/2021 12:05:37 INFO starting activate osd. 01/02/2021 12:05:38 INFO Starting : ceph-volume --log-path /opt/petasan/log lvm activate --bluestore 5 e97014c9-e0ec-45b1-b04f-7c40da3bed0b 01/02/2021 12:05:39 INFO Successfully executed ceph-volume prepare for sdc 01/02/2021 12:05:39 INFO Start cleaning disk : sdd 01/02/2021 12:05:39 INFO Executing : wipefs --all /dev/sdd 01/02/2021 12:05:39 INFO Executing : dd if=/dev/zero of=/dev/sdd bs=1M count=20 oflag=direct,dsync >/dev/null 2>&1 01/02/2021 12:05:39 INFO Executing : parted -s /dev/sdd mklabel gpt 01/02/2021 12:05:39 INFO Executing : partprobe /dev/sdd 01/02/2021 12:07:54 INFO Starting cluster file sync service 01/02/2021 12:07:54 INFO Starting Cluster Management application 01/02/2021 12:07:54 INFO LeaderElectionBase dropping old sessions 01/02/2021 12:07:55 INFO Starting activating PetaSAN lvs 01/02/2021 12:07:55 INFO Running command: /sbin/vgs --noheadings --readonly --units=b --nosuffix --separator=";" -o vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_count,vg_extent_size 01/02/2021 12:07:55 INFO stdout ceph-847b13a2-67ce-4374-b26b-f42d81ceb154";"1";"1";"wz--n-";"476931";"0";"4194304 01/02/2021 12:07:55 INFO stdout ceph-e72ec456-4943-4572-8b6f-f20fa4b85417";"1";"1";"wz--n-";"476931";"0";"4194304 01/02/2021 12:07:55 INFO stdout ceph-fe73888d-9945-41db-8d0d-5bb67360981c";"1";"1";"wz--n-";"476931";"0";"4194304 01/02/2021 12:07:55 INFO Starting activating PetaSAN lvs 01/02/2021 12:07:55 INFO Running command: /sbin/vgs --noheadings --readonly --units=b --nosuffix --separator=";" -o vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_count,vg_extent_size 01/02/2021 12:07:55 INFO stdout ceph-847b13a2-67ce-4374-b26b-f42d81ceb154";"1";"1";"wz--n-";"476931";"0";"4194304 01/02/2021 12:07:55 INFO stdout ceph-e72ec456-4943-4572-8b6f-f20fa4b85417";"1";"1";"wz--n-";"476931";"0";"4194304 01/02/2021 12:07:55 INFO stdout ceph-fe73888d-9945-41db-8d0d-5bb67360981c";"1";"1";"wz--n-";"476931";"0";"4194304 01/02/2021 12:07:56 INFO Starting Node Stats Service 01/02/2021 12:07:56 INFO Starting OSDs 01/02/2021 12:07:56 INFO Starting petasan tuning service 01/02/2021 12:07:56 INFO Starting petasan-qperf server 01/02/2021 12:08:04 INFO LeaderElectionBase successfully dropped old sessions 01/02/2021 12:08:34 ERROR Cluster is not completed, PetaSAN will check node join status. Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/cluster/deploy.py", line 78, in get_node_status raise Exception("Cluster is not completed, PetaSAN will check node join status.") Exception: Cluster is not completed, PetaSAN will check node join status. 01/02/2021 12:09:32 ERROR Cluster is not completed, PetaSAN will check node join status. Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/cluster/deploy.py", line 78, in get_node_status raise Exception("Cluster is not completed, PetaSAN will check node join status.") Exception: Cluster is not completed, PetaSAN will check node join status. 01/02/2021 12:09:55 INFO GlusterFS mount attempt 01/02/2021 12:10:06 INFO console.py process is 792 01/02/2021 12:10:06 INFO Trying to stop console.py process 01/02/2021 12:10:09 INFO console.py process stopped 01/02/2021 12:10:09 INFO console.py process is 797 01/02/2021 12:10:09 INFO console.py process stopped 01/02/2021 12:10:09 INFO console.py process is 798 01/02/2021 12:10:09 INFO console.py process stopped 01/02/2021 12:10:09 INFO console.py process is 7326 01/02/2021 12:12:02 INFO Searching for any unlinked journal to be reused. 01/02/2021 12:12:02 INFO OSD IDs list for this node are : [5, 4, 3] 01/02/2021 12:12:34 INFO script 01/02/2021 12:12:34 INFO /opt/petasan/scripts/admin/node_manage_disks.py add-osd 01/02/2021 12:12:34 INFO params 01/02/2021 12:12:34 INFO -disk_name sdd 01/02/2021 12:12:34 INFO Start add osd job for disk sdd. 01/02/2021 12:12:34 INFO Start add osd job 11118 01/02/2021 12:12:36 INFO Start cleaning disk : sdd 01/02/2021 12:12:37 INFO Executing : wipefs --all /dev/sdd 01/02/2021 12:12:37 INFO Executing : dd if=/dev/zero of=/dev/sdd bs=1M count=20 oflag=direct,dsync >/dev/null 2>&1 01/02/2021 12:12:37 INFO Executing : parted -s /dev/sdd mklabel gpt 01/02/2021 12:12:37 INFO Executing : partprobe /dev/sdd 01/02/2021 12:12:40 INFO User didn't select a journal for disk sdd, so the journal will be on the same disk. 01/02/2021 12:12:40 INFO User didn't select a cache for disk sdd. 01/02/2021 12:12:40 INFO User selected cache disk for disk sdd. 01/02/2021 12:12:40 INFO Start prepare bluestore OSD : sdd 01/02/2021 12:12:40 INFO Running command: /sbin/lvs --noheadings --readonly --separator=";" -a -o lv_tags,lv_path,lv_name,vg_name,lv_uuid,lv_size 01/02/2021 12:12:40 INFO stdout ceph.block_device=/dev/ceph-847b13a2-67ce-4374-b26b-f42d81ceb154/osd-block-12bc918a-066d-4340-a0d4-9b61f11bf3c5,ceph.block_uuid=ai28RZ-QoHE-N46c-OCSe-0dNf-VpAI-rMBDPM,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=c74c6e06-673f-4d0c-8235-563559e08858,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/sdf2,ceph.db_uuid=2bff43e1-8653-4404-a988-7314f0973333,ceph.encrypted=0,ceph.osd_fsid=12bc918a-066d-4340-a0d4-9b61f11bf3c5,ceph.osd_id=4,ceph.osdspec_affinity=,ceph.type=block,ceph.vdo=0";"/dev/ceph-847b13a2-67ce-4374-b26b-f42d81ceb154/osd-block-12bc918a-066d-4340-a0d4-9b61f11bf3c5";"osd-block-12bc918a-066d-4340-a0d4-9b61f11bf3c5";"ceph-847b13a2-67ce-4374-b26b-f42d81ceb154";"ai28RZ-QoHE-N46c-OCSe-0dNf-VpAI-rMBDPM";"<1.82t 01/02/2021 12:12:40 INFO stdout ceph.block_device=/dev/ceph-e72ec456-4943-4572-8b6f-f20fa4b85417/osd-block-0db3b469-bcee-4c49-9d80-5b3eac37fda6,ceph.block_uuid=5f6FRj-Z1M0-t4Wo-gyex-cFvy-qvXR-ogG1ST,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=c74c6e06-673f-4d0c-8235-563559e08858,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/sdf1,ceph.db_uuid=2fbde886-866b-415c-a09f-bca4a37d32ae,ceph.encrypted=0,ceph.osd_fsid=0db3b469-bcee-4c49-9d80-5b3eac37fda6,ceph.osd_id=3,ceph.osdspec_affinity=,ceph.type=block,ceph.vdo=0";"/dev/ceph-e72ec456-4943-4572-8b6f-f20fa4b85417/osd-block-0db3b469-bcee-4c49-9d80-5b3eac37fda6";"osd-block-0db3b469-bcee-4c49-9d80-5b3eac37fda6";"ceph-e72ec456-4943-4572-8b6f-f20fa4b85417";"5f6FRj-Z1M0-t4Wo-gyex-cFvy-qvXR-ogG1ST";"<1.82t 01/02/2021 12:12:40 INFO stdout ceph.block_device=/dev/ceph-fe73888d-9945-41db-8d0d-5bb67360981c/osd-block-e97014c9-e0ec-45b1-b04f-7c40da3bed0b,ceph.block_uuid=7x57lV-xGbZ-0BmG-pPeV-piMB-ltBA-YseShF,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=c74c6e06-673f-4d0c-8235-563559e08858,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/sdf3,ceph.db_uuid=4f0b9f29-b3b6-4be1-87e1-7ab5c1e6b655,ceph.encrypted=0,ceph.osd_fsid=e97014c9-e0ec-45b1-b04f-7c40da3bed0b,ceph.osd_id=5,ceph.osdspec_affinity=,ceph.type=block,ceph.vdo=0";"/dev/ceph-fe73888d-9945-41db-8d0d-5bb67360981c/osd-block-e97014c9-e0ec-45b1-b04f-7c40da3bed0b";"osd-block-e97014c9-e0ec-45b1-b04f-7c40da3bed0b";"ceph-fe73888d-9945-41db-8d0d-5bb67360981c";"7x57lV-xGbZ-0BmG-pPeV-piMB-ltBA-YseShF";"<1.82t 01/02/2021 12:12:40 INFO Creating data partition num 1 size 0 on /dev/sdd 01/02/2021 12:12:42 INFO Calling partprobe on sdd device 01/02/2021 12:12:42 INFO Executing partprobe /dev/sdd 01/02/2021 12:12:42 INFO Calling udevadm on sdd device 01/02/2021 12:12:42 INFO Executing udevadm settle --timeout 30 01/02/2021 12:12:45 INFO Starting : ceph-volume --log-path /opt/petasan/log lvm prepare --bluestore --data /dev/sdd1 01/02/2021 12:12:50 INFO starting activate osd. 01/02/2021 12:12:51 INFO Starting : ceph-volume --log-path /opt/petasan/log lvm activate --bluestore 9 cbcc32dc-972a-4351-8ec9-2b3e5f715daa 01/02/2021 12:13:37 INFO Start add osd job 11450 01/02/2021 12:14:21 INFO Start add osd job 12873 01/02/2021 12:21:38 INFO script 01/02/2021 12:21:38 INFO /opt/petasan/scripts/util/benchmark.py manager 01/02/2021 12:21:38 INFO params 01/02/2021 12:21:38 INFO -d 60 -t 16 -type 2 -c peta1,peta2 -p cephfs_data --cleanup 1 01/02/2021 12:21:38 INFO Benchmark Test Started 01/02/2021 12:21:38 INFO Benchmark manager cmd. 01/02/2021 12:21:38 INFO Benchmark start rados write. 01/02/2021 12:21:38 INFO Run rados write cmd on node peta1 : python /opt/petasan/scripts/jobs/benchmark/client_stress.py -d 30 -t 16 -b 4096 -m w -p cephfs_data 01/02/2021 12:21:38 INFO script 01/02/2021 12:21:38 INFO /opt/petasan/scripts/util/benchmark.py client 01/02/2021 12:21:38 INFO params 01/02/2021 12:21:38 INFO -d 30 -t 16 -b 4096 -m w -p cephfs_data 01/02/2021 12:21:38 INFO Start client stress job for rados 01/02/2021 12:21:39 INFO Run rados write cmd on node peta2 : python /opt/petasan/scripts/jobs/benchmark/client_stress.py -d 30 -t 16 -b 4096 -m w -p cephfs_data 01/02/2021 12:21:39 INFO Benchmark client cmd. 01/02/2021 12:21:39 INFO Wait time before collect storage state. 01/02/2021 12:21:47 INFO Run sar state cmd on node peta3 : python /opt/petasan/scripts/jobs/benchmark/storage_load.py -d 15 01/02/2021 12:22:12 INFO Benchmark start rados read. 01/02/2021 12:22:12 INFO Run rados read cmd on node peta1 : python /opt/petasan/scripts/jobs/benchmark/client_stress.py -d 30 -t 16 -m r -p cephfs_data 01/02/2021 12:22:12 INFO script 01/02/2021 12:22:12 INFO /opt/petasan/scripts/util/benchmark.py client 01/02/2021 12:22:12 INFO params 01/02/2021 12:22:12 INFO -d 30 -t 16 -b 4000 -m r -p cephfs_data 01/02/2021 12:22:12 INFO Start client stress job for rados 01/02/2021 12:22:12 INFO Run rados read cmd on node peta2 : python /opt/petasan/scripts/jobs/benchmark/client_stress.py -d 30 -t 16 -m r -p cephfs_data 01/02/2021 12:22:13 INFO Benchmark client cmd. 01/02/2021 12:22:13 INFO Wait time before collect node state. 01/02/2021 12:22:20 INFO Run sar state cmd on node peta3 : python /opt/petasan/scripts/jobs/benchmark/storage_load.py -d 15 01/02/2021 12:22:46 INFO Benchmark finished. 01/02/2021 12:22:46 INFO Running benchmark clean command on node:peta1 cmd:/opt/petasan/scripts/util/benchmark.py clean -p cephfs_data 01/02/2021 12:22:46 INFO Benchmark clean cmd. 01/02/2021 12:22:57 INFO Running benchmark clean command on node:peta2 cmd:/opt/petasan/scripts/util/benchmark.py clean -p cephfs_data 01/02/2021 12:23:13 INFO Benchmark Test Completed 01/02/2021 12:26:27 INFO Success saving application config 01/02/2021 12:28:09 INFO Success saving application config 01/02/2021 12:59:40 ERROR Node Stats exception. 01/02/2021 12:59:40 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.5 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 13.63 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 921.6 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 542.72 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.06 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 65771.52 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 69754.88 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.05 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 61112.32 `date +%s`" | nc -q0 10.10.0.153 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.5 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 13.63 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 921.6 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 542.72 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.06 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 65771.52 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 69754.88 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.05 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 61112.32 `date +%s`" | nc -q0 10.10.0.153 2003 01/02/2021 13:02:02 ERROR Node Stats exception. 01/02/2021 13:02:02 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 8.22 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 14.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1474.56 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 215.04 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.91 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 459151.36 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 1131479.04 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.87 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 430755.84 `date +%s`" | nc -q0 10.10.0.152 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 8.22 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 14.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1474.56 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 215.04 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.91 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 459151.36 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 1131479.04 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.87 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 430755.84 `date +%s`" | nc -q0 10.10.0.152 2003 01/02/2021 13:04:09 INFO Start settings IPs 01/02/2021 13:04:21 INFO GlusterFS mount attempt 01/02/2021 13:04:21 INFO str_start_command: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consul agent -raft-protocol 2 -config-dir /opt/petasan/config/etc/consul.d/server -bind 172.16.0.151 -retry-join 172.16.0.152 -retry-join 172.16.0.153 01/02/2021 13:04:24 INFO Starting cluster file sync service 01/02/2021 13:04:24 INFO Starting Cluster Management application 01/02/2021 13:04:24 INFO Starting Node Stats Service 01/02/2021 13:04:24 INFO LeaderElectionBase dropping old sessions 01/02/2021 13:04:24 INFO Starting activating PetaSAN lvs 01/02/2021 13:04:24 INFO Running command: /sbin/vgs --noheadings --readonly --units=b --nosuffix --separator=";" -o vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_count,vg_extent_size 01/02/2021 13:04:24 INFO stdout ceph-847b13a2-67ce-4374-b26b-f42d81ceb154";"1";"1";"wz--n-";"476931";"0";"4194304 01/02/2021 13:04:24 INFO stdout ceph-d0153ed6-0b75-470c-b1c7-906759dee371";"1";"1";"wz--n-";"476931";"0";"4194304 01/02/2021 13:04:24 INFO stdout ceph-e72ec456-4943-4572-8b6f-f20fa4b85417";"1";"1";"wz--n-";"476931";"0";"4194304 01/02/2021 13:04:24 INFO stdout ceph-fe73888d-9945-41db-8d0d-5bb67360981c";"1";"1";"wz--n-";"476931";"0";"4194304 01/02/2021 13:04:24 INFO Starting activating PetaSAN lvs 01/02/2021 13:04:25 INFO Running command: /sbin/vgs --noheadings --readonly --units=b --nosuffix --separator=";" -o vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_count,vg_extent_size 01/02/2021 13:04:25 INFO stdout ceph-847b13a2-67ce-4374-b26b-f42d81ceb154";"1";"1";"wz--n-";"476931";"0";"4194304 01/02/2021 13:04:25 INFO stdout ceph-d0153ed6-0b75-470c-b1c7-906759dee371";"1";"1";"wz--n-";"476931";"0";"4194304 01/02/2021 13:04:25 INFO stdout ceph-e72ec456-4943-4572-8b6f-f20fa4b85417";"1";"1";"wz--n-";"476931";"0";"4194304 01/02/2021 13:04:25 INFO stdout ceph-fe73888d-9945-41db-8d0d-5bb67360981c";"1";"1";"wz--n-";"476931";"0";"4194304 01/02/2021 13:04:26 INFO Starting OSDs 01/02/2021 13:04:26 INFO Starting petasan tuning service 01/02/2021 13:04:26 INFO Starting Config Upload service 01/02/2021 13:04:26 INFO Starting petasan-qperf server 01/02/2021 13:04:40 INFO Checking backend latencies : 01/02/2021 13:04:40 INFO Network latency for backend 172.16.0.151 = 7.29 us 01/02/2021 13:04:40 INFO Network latency for backend 172.16.0.152 = 68.0 us 01/02/2021 13:04:40 INFO Network latency for backend 172.16.0.153 = 73.7 us 01/02/2021 13:04:42 INFO Checking backend latencies : 01/02/2021 13:04:42 INFO Network latency for backend 172.16.0.151 = 7.27 us 01/02/2021 13:04:42 INFO Network latency for backend 172.16.0.152 = 74.7 us 01/02/2021 13:04:42 INFO Network latency for backend 172.16.0.153 = 77.5 us 01/02/2021 13:04:47 INFO Checking backend latencies : 01/02/2021 13:04:47 INFO Network latency for backend 172.16.0.151 = 20.4 us 01/02/2021 13:04:47 INFO Network latency for backend 172.16.0.152 = 103.0 us 01/02/2021 13:04:47 INFO Network latency for backend 172.16.0.153 = 119.0 us 01/02/2021 13:04:49 INFO Checking backend latencies : 01/02/2021 13:04:49 INFO Network latency for backend 172.16.0.151 = 21.0 us 01/02/2021 13:04:49 INFO Network latency for backend 172.16.0.152 = 92.3 us 01/02/2021 13:04:49 INFO Network latency for backend 172.16.0.153 = 113.0 us 01/02/2021 13:04:51 INFO LeaderElectionBase successfully dropped old sessions 01/02/2021 13:04:57 INFO GlusterFS mount attempt 01/02/2021 16:21:38 INFO GlusterFS mount attempt 01/02/2021 16:23:29 INFO GlusterFS mount attempt 01/02/2021 16:26:03 INFO Checking backend latencies : 01/02/2021 16:26:03 INFO Network latency for backend 172.16.0.151 = 7.3 us 01/02/2021 16:26:03 INFO Network latency for backend 172.16.0.152 = 51.1 us 01/02/2021 16:26:03 INFO Network latency for backend 172.16.0.153 = 01/02/2021 16:26:05 INFO Checking backend latencies : 01/02/2021 16:26:05 INFO Network latency for backend 172.16.0.151 = 8.24 us 01/02/2021 16:26:05 INFO Network latency for backend 172.16.0.152 = 50.2 us 01/02/2021 16:26:05 INFO Network latency for backend 172.16.0.153 = 01/02/2021 16:26:34 INFO ClusterLeader start action 01/02/2021 16:26:34 INFO ClusterLeader starting services 01/02/2021 16:26:38 ERROR Node Stats exception. 01/02/2021 16:26:38 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 18.25 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 17.26 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 993.28 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 20.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.07 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 89815.04 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 87234.56 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.07 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 84367.36 `date +%s`" | nc -q0 10.10.0.153 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 18.25 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 17.26 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 993.28 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 20.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.07 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 89815.04 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 87234.56 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.07 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 84367.36 `date +%s`" | nc -q0 10.10.0.153 2003 01/02/2021 16:32:08 INFO Start settings IPs 01/02/2021 16:32:19 INFO GlusterFS mount attempt 01/02/2021 16:32:19 INFO str_start_command: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consul agent -raft-protocol 2 -config-dir /opt/petasan/config/etc/consul.d/server -bind 172.16.0.151 -retry-join 172.16.0.152 -retry-join 172.16.0.153 01/02/2021 16:32:22 INFO Starting cluster file sync service 01/02/2021 16:32:22 INFO Starting Cluster Management application 01/02/2021 16:32:22 INFO Starting Node Stats Service 01/02/2021 16:32:22 INFO LeaderElectionBase dropping old sessions 01/02/2021 16:32:22 INFO Starting activating PetaSAN lvs 01/02/2021 16:32:22 INFO Running command: /sbin/vgs --noheadings --readonly --units=b --nosuffix --separator=";" -o vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_count,vg_extent_size 01/02/2021 16:32:23 INFO stdout ceph-847b13a2-67ce-4374-b26b-f42d81ceb154";"1";"1";"wz--n-";"476931";"0";"4194304 01/02/2021 16:32:23 INFO stdout ceph-d0153ed6-0b75-470c-b1c7-906759dee371";"1";"1";"wz--n-";"476931";"0";"4194304 01/02/2021 16:32:23 INFO stdout ceph-e72ec456-4943-4572-8b6f-f20fa4b85417";"1";"1";"wz--n-";"476931";"0";"4194304 01/02/2021 16:32:23 INFO stdout ceph-fe73888d-9945-41db-8d0d-5bb67360981c";"1";"1";"wz--n-";"476931";"0";"4194304 01/02/2021 16:32:23 INFO Starting activating PetaSAN lvs 01/02/2021 16:32:23 INFO Running command: /sbin/vgs --noheadings --readonly --units=b --nosuffix --separator=";" -o vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_count,vg_extent_size 01/02/2021 16:32:23 INFO stdout ceph-847b13a2-67ce-4374-b26b-f42d81ceb154";"1";"1";"wz--n-";"476931";"0";"4194304 01/02/2021 16:32:23 INFO stdout ceph-d0153ed6-0b75-470c-b1c7-906759dee371";"1";"1";"wz--n-";"476931";"0";"4194304 01/02/2021 16:32:23 INFO stdout ceph-e72ec456-4943-4572-8b6f-f20fa4b85417";"1";"1";"wz--n-";"476931";"0";"4194304 01/02/2021 16:32:23 INFO stdout ceph-fe73888d-9945-41db-8d0d-5bb67360981c";"1";"1";"wz--n-";"476931";"0";"4194304 01/02/2021 16:32:24 INFO Starting OSDs 01/02/2021 16:32:24 INFO Starting petasan tuning service 01/02/2021 16:32:24 INFO Starting Config Upload service 01/02/2021 16:32:24 INFO Starting petasan-qperf server 01/02/2021 16:32:40 INFO Checking backend latencies : 01/02/2021 16:32:40 INFO Network latency for backend 172.16.0.151 = 7.45 us 01/02/2021 16:32:40 INFO Network latency for backend 172.16.0.152 = 80.1 us 01/02/2021 16:32:40 INFO Network latency for backend 172.16.0.153 = 71.4 us 01/02/2021 16:32:42 INFO Checking backend latencies : 01/02/2021 16:32:42 INFO Network latency for backend 172.16.0.151 = 7.28 us 01/02/2021 16:32:42 INFO Network latency for backend 172.16.0.152 = 71.2 us 01/02/2021 16:32:42 INFO Network latency for backend 172.16.0.153 = 63.2 us 01/02/2021 16:32:44 INFO Checking backend latencies : 01/02/2021 16:32:44 INFO Network latency for backend 172.16.0.151 = 15.7 us 01/02/2021 16:32:44 INFO Network latency for backend 172.16.0.152 = 60.3 us 01/02/2021 16:32:44 INFO Network latency for backend 172.16.0.153 = 70.1 us 01/02/2021 16:32:46 INFO Checking backend latencies : 01/02/2021 16:32:46 INFO Network latency for backend 172.16.0.151 = 12.1 us 01/02/2021 16:32:46 INFO Network latency for backend 172.16.0.152 = 66.4 us 01/02/2021 16:32:46 INFO Network latency for backend 172.16.0.153 = 64.1 us 01/02/2021 16:32:48 INFO LeaderElectionBase successfully dropped old sessions 01/02/2021 16:32:52 INFO Checking backend latencies : 01/02/2021 16:32:52 INFO Network latency for backend 172.16.0.151 = 9.7 us 01/02/2021 16:32:52 INFO Network latency for backend 172.16.0.152 = 52.9 us 01/02/2021 16:32:52 INFO Network latency for backend 172.16.0.153 = 57.8 us 01/02/2021 16:32:56 INFO GlusterFS mount attempt 01/02/2021 19:53:24 INFO GlusterFS mount attempt 02/02/2021 03:53:16 ERROR Node Stats exception. 02/02/2021 03:53:16 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 7.19 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 16.15 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 2201.6 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 327.68 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.05 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 56279.04 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 54763.52 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.04 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 51957.76 `date +%s`" | nc -q0 10.10.0.153 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 7.19 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 16.15 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 2201.6 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 327.68 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.05 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 56279.04 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 54763.52 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.04 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 51957.76 `date +%s`" | nc -q0 10.10.0.153 2003 02/02/2021 03:53:18 INFO ClusterLeader start action 02/02/2021 03:53:18 INFO ClusterLeader starting services 02/02/2021 03:58:23 INFO Checking backend latencies : 02/02/2021 03:58:23 INFO Network latency for backend 172.16.0.151 = 02/02/2021 03:58:23 INFO Network latency for backend 172.16.0.152 = 02/02/2021 03:58:23 INFO Network latency for backend 172.16.0.153 = 57.5 us 02/02/2021 03:58:23 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 02/02/2021 03:58:25 INFO Checking backend latencies : 02/02/2021 03:58:25 INFO Network latency for backend 172.16.0.151 = 7.17 us 02/02/2021 03:58:25 INFO Network latency for backend 172.16.0.152 = 02/02/2021 03:58:25 INFO Network latency for backend 172.16.0.153 = 58.0 us 02/02/2021 04:02:40 INFO Start settings IPs 02/02/2021 04:02:52 INFO str_start_command: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consul agent -raft-protocol 2 -config-dir /opt/petasan/config/etc/consul.d/server -bind 172.16.0.151 -retry-join 172.16.0.152 -retry-join 172.16.0.153 02/02/2021 04:02:52 INFO GlusterFS mount attempt 02/02/2021 04:02:55 INFO Starting cluster file sync service 02/02/2021 04:02:55 INFO Starting Cluster Management application 02/02/2021 04:02:55 INFO Starting Node Stats Service 02/02/2021 04:02:55 INFO LeaderElectionBase dropping old sessions 02/02/2021 04:02:55 INFO Starting activating PetaSAN lvs 02/02/2021 04:02:55 INFO Running command: /sbin/vgs --noheadings --readonly --units=b --nosuffix --separator=";" -o vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_count,vg_extent_size 02/02/2021 04:02:55 INFO stdout ceph-847b13a2-67ce-4374-b26b-f42d81ceb154";"1";"1";"wz--n-";"476931";"0";"4194304 02/02/2021 04:02:55 INFO stdout ceph-d0153ed6-0b75-470c-b1c7-906759dee371";"1";"1";"wz--n-";"476931";"0";"4194304 02/02/2021 04:02:55 INFO stdout ceph-e72ec456-4943-4572-8b6f-f20fa4b85417";"1";"1";"wz--n-";"476931";"0";"4194304 02/02/2021 04:02:55 INFO stdout ceph-fe73888d-9945-41db-8d0d-5bb67360981c";"1";"1";"wz--n-";"476931";"0";"4194304 02/02/2021 04:02:55 INFO Starting activating PetaSAN lvs 02/02/2021 04:02:56 INFO Running command: /sbin/vgs --noheadings --readonly --units=b --nosuffix --separator=";" -o vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_count,vg_extent_size 02/02/2021 04:02:56 INFO stdout ceph-847b13a2-67ce-4374-b26b-f42d81ceb154";"1";"1";"wz--n-";"476931";"0";"4194304 02/02/2021 04:02:56 INFO stdout ceph-d0153ed6-0b75-470c-b1c7-906759dee371";"1";"1";"wz--n-";"476931";"0";"4194304 02/02/2021 04:02:56 INFO stdout ceph-e72ec456-4943-4572-8b6f-f20fa4b85417";"1";"1";"wz--n-";"476931";"0";"4194304 02/02/2021 04:02:56 INFO stdout ceph-fe73888d-9945-41db-8d0d-5bb67360981c";"1";"1";"wz--n-";"476931";"0";"4194304 02/02/2021 04:02:57 INFO Starting OSDs 02/02/2021 04:02:57 INFO Starting petasan tuning service 02/02/2021 04:02:57 INFO Starting Config Upload service 02/02/2021 04:02:57 INFO Starting petasan-qperf server 02/02/2021 04:03:11 INFO Checking backend latencies : 02/02/2021 04:03:11 INFO Network latency for backend 172.16.0.151 = 6.64 us 02/02/2021 04:03:11 INFO Network latency for backend 172.16.0.152 = 70.4 us 02/02/2021 04:03:11 INFO Network latency for backend 172.16.0.153 = 87.2 us 02/02/2021 04:03:13 INFO Checking backend latencies : 02/02/2021 04:03:13 INFO Network latency for backend 172.16.0.151 = 7.12 us 02/02/2021 04:03:13 INFO Network latency for backend 172.16.0.152 = 85.9 us 02/02/2021 04:03:13 INFO Network latency for backend 172.16.0.153 = 79.2 us 02/02/2021 04:03:15 INFO Checking backend latencies : 02/02/2021 04:03:15 INFO Network latency for backend 172.16.0.151 = 12.2 us 02/02/2021 04:03:15 INFO Network latency for backend 172.16.0.152 = 73.8 us 02/02/2021 04:03:15 INFO Network latency for backend 172.16.0.153 = 84.7 us 02/02/2021 04:03:18 INFO Checking backend latencies : 02/02/2021 04:03:18 INFO Network latency for backend 172.16.0.151 = 17.8 us 02/02/2021 04:03:18 INFO Network latency for backend 172.16.0.152 = 88.2 us 02/02/2021 04:03:18 INFO Network latency for backend 172.16.0.153 = 75.8 us 02/02/2021 04:03:26 INFO Checking backend latencies : 02/02/2021 04:03:26 INFO Network latency for backend 172.16.0.151 = 7.67 us 02/02/2021 04:03:26 INFO Network latency for backend 172.16.0.152 = 60.2 us 02/02/2021 04:03:26 INFO Network latency for backend 172.16.0.153 = 52.7 us 02/02/2021 04:03:28 INFO Checking backend latencies : 02/02/2021 04:03:28 INFO Network latency for backend 172.16.0.151 = 8.29 us 02/02/2021 04:03:28 INFO Network latency for backend 172.16.0.152 = 52.3 us 02/02/2021 04:03:28 INFO Network latency for backend 172.16.0.153 = 51.1 us 02/02/2021 04:03:28 INFO GlusterFS mount attempt 02/02/2021 04:03:30 INFO Checking backend latencies : 02/02/2021 04:03:30 INFO Network latency for backend 172.16.0.151 = 8.44 us 02/02/2021 04:03:30 INFO Network latency for backend 172.16.0.152 = 54.2 us 02/02/2021 04:03:30 INFO Network latency for backend 172.16.0.153 = 51.5 us 02/02/2021 04:03:34 INFO Checking backend latencies : 02/02/2021 04:03:34 INFO Network latency for backend 172.16.0.151 = 8.3 us 02/02/2021 04:03:34 INFO Network latency for backend 172.16.0.152 = 50.3 us 02/02/2021 04:03:34 INFO Network latency for backend 172.16.0.153 = 55.5 us 02/02/2021 04:03:38 INFO LeaderElectionBase successfully dropped old sessions 02/02/2021 04:06:13 INFO Success saving application config 02/02/2021 04:06:46 INFO Success saving application config 02/02/2021 04:16:08 ERROR 400 Bad Request: The browser (or proxy) sent a request that this server could not understand. 02/02/2021 04:16:08 ERROR 400 Bad Request: The browser (or proxy) sent a request that this server could not understand. 02/02/2021 04:16:08 ERROR 400 Bad Request: The browser (or proxy) sent a request that this server could not understand. 02/02/2021 04:16:08 ERROR 400 Bad Request: The browser (or proxy) sent a request that this server could not understand. 02/02/2021 06:25:06 INFO GlusterFS mount attempt 02/02/2021 07:06:24 INFO PetaSAN certificate has been synced 02/02/2021 07:22:07 ERROR Node Stats exception. 02/02/2021 07:22:07 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 9.2 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 12.96 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1105.92 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 20.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.25 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 280586.24 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 312156.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.24 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 256983.04 `date +%s`" | nc -q0 10.10.0.152 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 9.2 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 12.96 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1105.92 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 20.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.25 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 280586.24 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 312156.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.24 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 256983.04 `date +%s`" | nc -q0 10.10.0.152 2003 02/02/2021 08:16:01 ERROR Node Stats exception. 02/02/2021 08:16:01 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.disks.smart.sda.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sda.power_on_hours 84473 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sda.temperature_celsius 34 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.power_on_hours 84473 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.temperature_celsius 33 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.power_on_hours 84473 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.temperature_celsius 33 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdd.reallocated_sector_ct 0 `date +%s`" | nc -q0 10.10.0.152 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.disks.smart.sda.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sda.power_on_hours 84473 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sda.temperature_celsius 34 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.power_on_hours 84473 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.temperature_celsius 33 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.power_on_hours 84473 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.temperature_celsius 33 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdd.reallocated_sector_ct 0 `date +%s`" | nc -q0 10.10.0.152 2003 02/02/2021 09:28:51 ERROR Node Stats exception. 02/02/2021 09:28:51 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 11.58 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 14.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1280.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 256.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.17 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 199249.92 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 211425.28 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 187187.2 `date +%s`" | nc -q0 10.10.0.152 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 11.58 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 14.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1280.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 256.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.17 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 199249.92 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 211425.28 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 187187.2 `date +%s`" | nc -q0 10.10.0.152 2003 02/02/2021 09:29:56 ERROR Node Stats exception. 02/02/2021 09:29:56 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.disks.smart.sda.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sda.power_on_hours 84474 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sda.temperature_celsius 35 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.power_on_hours 84474 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.temperature_celsius 34 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.power_on_hours 84474 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.temperature_celsius 34 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdd.reallocated_sector_ct 0 `date +%s`" | nc -q0 10.10.0.152 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.disks.smart.sda.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sda.power_on_hours 84474 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sda.temperature_celsius 35 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.power_on_hours 84474 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.temperature_celsius 34 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.power_on_hours 84474 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.temperature_celsius 34 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdd.reallocated_sector_ct 0 `date +%s`" | nc -q0 10.10.0.152 2003 02/02/2021 09:30:59 ERROR Node Stats exception. 02/02/2021 09:30:59 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 14.68 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 14.37 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.06 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 71127.04 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 3297.28 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.12 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 149739.52 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 134277.12 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.11 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 139960.32 `date +%s`" | nc -q0 10.10.0.152 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 14.68 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 14.37 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.06 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 71127.04 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 3297.28 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.12 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 149739.52 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 134277.12 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.11 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 139960.32 `date +%s`" | nc -q0 10.10.0.152 2003 02/02/2021 09:32:02 ERROR Node Stats exception. 02/02/2021 09:32:02 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.67 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 14.21 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1351.68 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 686.08 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.15 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 122060.8 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 192696.32 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.15 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 114298.88 `date +%s`" | nc -q0 10.10.0.152 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.67 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 14.21 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1351.68 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 686.08 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.15 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 122060.8 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 192696.32 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.15 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 114298.88 `date +%s`" | nc -q0 10.10.0.152 2003 02/02/2021 09:33:06 ERROR Node Stats exception. 02/02/2021 09:33:06 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 12.52 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 14.25 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1167.36 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 256.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.18 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 187688.96 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 230430.72 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.17 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 175390.72 `date +%s`" | nc -q0 10.10.0.152 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 12.52 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 14.25 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1167.36 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 256.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.18 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 187688.96 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 230430.72 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.17 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 175390.72 `date +%s`" | nc -q0 10.10.0.152 2003 02/02/2021 09:34:09 ERROR Node Stats exception. 02/02/2021 09:34:09 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.5 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 14.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1126.4 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 20.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.09 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 113295.36 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 115251.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.09 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 105635.84 `date +%s`" | nc -q0 10.10.0.152 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.5 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 14.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1126.4 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 20.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.09 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 113295.36 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 115251.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.09 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 105635.84 `date +%s`" | nc -q0 10.10.0.152 2003 02/02/2021 09:35:12 ERROR Node Stats exception. 02/02/2021 09:35:12 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 12.62 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 14.27 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1566.72 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 256.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 158597.12 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 204748.8 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 148039.68 `date +%s`" | nc -q0 10.10.0.152 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 12.62 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 14.27 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1566.72 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 256.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 158597.12 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 204748.8 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 148039.68 `date +%s`" | nc -q0 10.10.0.152 2003 02/02/2021 09:36:15 ERROR Node Stats exception. 02/02/2021 09:36:15 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 11.12 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 14.27 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1259.52 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 276.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.15 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 140410.88 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 188129.28 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.14 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 130754.56 `date +%s`" | nc -q0 10.10.0.152 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 11.12 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 14.27 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1259.52 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 276.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.15 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 140410.88 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 188129.28 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.14 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 130754.56 `date +%s`" | nc -q0 10.10.0.152 2003 02/02/2021 10:41:43 ERROR Node Stats exception. 02/02/2021 10:41:43 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 13.34 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 15.19 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1546.24 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 20.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.19 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 241182.72 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 147886.08 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.18 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 227184.64 `date +%s`" | nc -q0 10.10.0.152 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 13.34 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 15.19 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1546.24 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 20.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.19 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 241182.72 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 147886.08 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.18 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 227184.64 `date +%s`" | nc -q0 10.10.0.152 2003 02/02/2021 10:42:47 ERROR Node Stats exception. 02/02/2021 10:42:47 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 2.16 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 15.04 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1269.76 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.18 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 169308.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 230205.44 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.17 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 158197.76 `date +%s`" | nc -q0 10.10.0.152 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 2.16 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 15.04 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1269.76 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.18 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 169308.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 230205.44 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.17 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 158197.76 `date +%s`" | nc -q0 10.10.0.152 2003 02/02/2021 16:36:37 ERROR Node Stats exception. 02/02/2021 16:36:37 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.5 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 18.87 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 983.04 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 120832.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 203776.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.15 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 112353.28 `date +%s`" | nc -q0 10.10.0.152 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.5 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 18.87 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 983.04 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 120832.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 203776.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.15 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 112353.28 `date +%s`" | nc -q0 10.10.0.152 2003 03/02/2021 06:13:57 INFO Success saving application config 03/02/2021 06:14:00 INFO Success saving application config 03/02/2021 06:14:00 INFO Success saving application config 03/02/2021 06:25:21 INFO GlusterFS mount attempt 03/02/2021 06:38:23 INFO Success saving application config 03/02/2021 06:38:27 INFO Success saving application config 03/02/2021 06:38:27 INFO Success saving application config 03/02/2021 06:39:42 INFO Start settings IPs 03/02/2021 06:39:54 INFO GlusterFS mount attempt 03/02/2021 06:39:54 INFO str_start_command: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consul agent -raft-protocol 2 -config-dir /opt/petasan/config/etc/consul.d/server -bind 172.16.0.151 -retry-join 172.16.0.152 -retry-join 172.16.0.153 03/02/2021 06:39:57 INFO Starting cluster file sync service 03/02/2021 06:39:57 INFO Starting Cluster Management application 03/02/2021 06:39:57 INFO Starting Node Stats Service 03/02/2021 06:39:57 INFO LeaderElectionBase dropping old sessions 03/02/2021 06:39:57 INFO Starting activating PetaSAN lvs 03/02/2021 06:39:57 INFO Running command: /sbin/vgs --noheadings --readonly --units=b --nosuffix --separator=";" -o vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_count,vg_extent_size 03/02/2021 06:39:58 INFO stdout ceph-847b13a2-67ce-4374-b26b-f42d81ceb154";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 06:39:58 INFO stdout ceph-d0153ed6-0b75-470c-b1c7-906759dee371";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 06:39:58 INFO stdout ceph-e72ec456-4943-4572-8b6f-f20fa4b85417";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 06:39:58 INFO stdout ceph-fe73888d-9945-41db-8d0d-5bb67360981c";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 06:39:58 INFO Starting activating PetaSAN lvs 03/02/2021 06:39:58 INFO Running command: /sbin/vgs --noheadings --readonly --units=b --nosuffix --separator=";" -o vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_count,vg_extent_size 03/02/2021 06:39:58 INFO stdout ceph-847b13a2-67ce-4374-b26b-f42d81ceb154";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 06:39:58 INFO stdout ceph-d0153ed6-0b75-470c-b1c7-906759dee371";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 06:39:58 INFO stdout ceph-e72ec456-4943-4572-8b6f-f20fa4b85417";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 06:39:58 INFO stdout ceph-fe73888d-9945-41db-8d0d-5bb67360981c";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 06:39:59 INFO Starting OSDs 03/02/2021 06:39:59 INFO Starting petasan tuning service 03/02/2021 06:39:59 INFO Starting Config Upload service 03/02/2021 06:39:59 INFO Starting petasan-qperf server 03/02/2021 06:40:13 INFO Checking backend latencies : 03/02/2021 06:40:13 INFO Network latency for backend 172.16.0.151 = 6.94 us 03/02/2021 06:40:13 INFO Network latency for backend 172.16.0.152 = 73.6 us 03/02/2021 06:40:13 INFO Network latency for backend 172.16.0.153 = 84.7 us 03/02/2021 06:40:15 INFO Checking backend latencies : 03/02/2021 06:40:15 INFO Network latency for backend 172.16.0.151 = 7.35 us 03/02/2021 06:40:15 INFO Network latency for backend 172.16.0.152 = 83.2 us 03/02/2021 06:40:15 INFO Network latency for backend 172.16.0.153 = 91.6 us 03/02/2021 06:40:21 INFO Checking backend latencies : 03/02/2021 06:40:21 INFO Network latency for backend 172.16.0.151 = 13.9 us 03/02/2021 06:40:21 INFO Network latency for backend 172.16.0.152 = 80.5 us 03/02/2021 06:40:21 INFO Network latency for backend 172.16.0.153 = 76.6 us 03/02/2021 06:40:23 INFO LeaderElectionBase successfully dropped old sessions 03/02/2021 06:40:24 INFO Checking backend latencies : 03/02/2021 06:40:24 INFO Network latency for backend 172.16.0.151 = 9.8 us 03/02/2021 06:40:24 INFO Network latency for backend 172.16.0.152 = 62.7 us 03/02/2021 06:40:24 INFO Network latency for backend 172.16.0.153 = 59.1 us 03/02/2021 06:40:28 INFO Checking backend latencies : 03/02/2021 06:40:28 INFO Network latency for backend 172.16.0.151 = 8.16 us 03/02/2021 06:40:28 INFO Network latency for backend 172.16.0.152 = 52.3 us 03/02/2021 06:40:28 INFO Network latency for backend 172.16.0.153 = 58.9 us 03/02/2021 06:40:30 INFO GlusterFS mount attempt 03/02/2021 06:43:22 ERROR 400 Bad Request: The browser (or proxy) sent a request that this server could not understand. 03/02/2021 06:43:22 ERROR 400 Bad Request: The browser (or proxy) sent a request that this server could not understand. 03/02/2021 06:43:22 ERROR 400 Bad Request: The browser (or proxy) sent a request that this server could not understand. 03/02/2021 06:43:22 ERROR 400 Bad Request: The browser (or proxy) sent a request that this server could not understand. 03/02/2021 06:43:23 INFO Update roles. 03/02/2021 06:43:23 INFO Update node iscsi role to true 03/02/2021 06:43:23 INFO Starting PetaSAN-iscsi service 03/02/2021 06:43:24 INFO checking uptime 03/02/2021 06:43:24 INFO checking uptime of node: peta1 ip:10.10.0.151 03/02/2021 06:43:24 INFO checking uptime of node: peta2 ip:10.10.0.152 03/02/2021 06:43:24 INFO checking uptime of node: peta3 ip:10.10.0.153 03/02/2021 06:43:24 INFO Service is starting. 03/02/2021 06:43:24 INFO iSCSI Service dropping old sessions 03/02/2021 06:43:24 INFO sync_replication_node called on non-backup node 03/02/2021 06:43:39 INFO iSCSI Service successfully dropped old sessions 03/02/2021 06:43:39 INFO Cleaning unused configurations. 03/02/2021 06:43:39 INFO Cleaning all mapped disks 03/02/2021 06:43:39 INFO Cleaning unused rbd images. 03/02/2021 06:43:39 INFO Cleaning unused ips. 03/02/2021 06:43:50 ERROR 400 Bad Request: The browser (or proxy) sent a request that this server could not understand. 03/02/2021 06:43:50 ERROR 400 Bad Request: The browser (or proxy) sent a request that this server could not understand. 03/02/2021 06:43:50 ERROR 400 Bad Request: The browser (or proxy) sent a request that this server could not understand. 03/02/2021 06:43:50 ERROR 400 Bad Request: The browser (or proxy) sent a request that this server could not understand. 03/02/2021 06:44:11 ERROR 400 Bad Request: The browser (or proxy) sent a request that this server could not understand. 03/02/2021 06:44:11 ERROR 400 Bad Request: The browser (or proxy) sent a request that this server could not understand. 03/02/2021 06:44:11 ERROR 400 Bad Request: The browser (or proxy) sent a request that this server could not understand. 03/02/2021 06:44:11 ERROR 400 Bad Request: The browser (or proxy) sent a request that this server could not understand. 03/02/2021 06:45:17 INFO include_wwn_fsid_tag() is true 03/02/2021 06:45:17 INFO add disk wwn is c74c6e0600001 03/02/2021 06:45:22 INFO Disk iscsi1 created 03/02/2021 06:45:22 INFO Successfully created key 00001 for new disk. 03/02/2021 06:45:22 INFO Successfully created key /00001/1 for new disk. 03/02/2021 06:45:22 INFO Successfully created key /00001/2 for new disk. 03/02/2021 06:45:22 INFO Successfully created key /00001/3 for new disk. 03/02/2021 06:45:29 INFO Found pool:rbd for disk:00001 via consul 03/02/2021 06:45:29 INFO Image image-00001 mapped successfully. 03/02/2021 06:45:32 INFO LIO add_target() disk wwn is c74c6e0600001 03/02/2021 06:45:32 INFO Path 00001/1 acquired successfully 03/02/2021 06:47:09 INFO Successfully created key 00001 for new disk. 03/02/2021 06:47:11 INFO Cleaned disk path 00001/1. 03/02/2021 06:47:11 INFO PetaSAN cleaned local paths not locked by this node in consul. 03/02/2021 06:47:11 INFO Stopping disk 00001 03/02/2021 06:47:11 ERROR Could not find ips for image-00001 03/02/2021 06:47:11 INFO LIO deleted backstore image image-00001 03/02/2021 06:47:11 INFO LIO deleted Target iqn.2016-05.com.petasan:00001 03/02/2021 06:47:11 INFO Image image-00001 unmapped successfully. 03/02/2021 06:47:13 INFO Found pool:rbd for disk:00001 via consul 03/02/2021 06:47:13 INFO PetaSAN removed key of stopped disk 00001 from consul. 03/02/2021 06:50:28 INFO call get assignments stats function. 03/02/2021 06:50:41 INFO Successfully created key 00001 for new disk. 03/02/2021 06:50:41 INFO Successfully created key /00001/1 for new disk. 03/02/2021 06:50:41 INFO Successfully created key /00001/2 for new disk. 03/02/2021 06:50:41 INFO Successfully created key /00001/3 for new disk. 03/02/2021 06:50:48 INFO Found pool:rbd for disk:00001 via consul 03/02/2021 06:50:49 INFO Image image-00001 mapped successfully. 03/02/2021 06:50:52 INFO LIO add_target() disk wwn is c74c6e0600001 03/02/2021 06:50:52 INFO Path 00001/3 acquired successfully 03/02/2021 06:51:20 INFO call get assignments stats function. 03/02/2021 06:51:30 INFO call get assignments stats function. 03/02/2021 06:51:40 INFO call get assignments stats function. 03/02/2021 06:51:50 INFO call get assignments stats function. 03/02/2021 06:52:00 INFO call get assignments stats function. 03/02/2021 06:52:10 INFO call get assignments stats function. 03/02/2021 06:52:20 INFO call get assignments stats function. 03/02/2021 06:52:30 INFO call get assignments stats function. 03/02/2021 06:52:40 INFO call get assignments stats function. 03/02/2021 06:52:50 INFO call get assignments stats function. 03/02/2021 06:53:00 INFO call get assignments stats function. 03/02/2021 06:53:10 INFO call get assignments stats function. 03/02/2021 06:53:20 INFO call get assignments stats function. 03/02/2021 06:53:30 INFO call get assignments stats function. 03/02/2021 06:53:40 INFO call get assignments stats function. 03/02/2021 06:53:50 INFO call get assignments stats function. 03/02/2021 06:54:01 INFO call get assignments stats function. 03/02/2021 06:54:11 INFO call get assignments stats function. 03/02/2021 06:54:21 INFO call get assignments stats function. 03/02/2021 06:54:31 INFO call get assignments stats function. 03/02/2021 06:54:41 INFO call get assignments stats function. 03/02/2021 06:54:51 INFO call get assignments stats function. 03/02/2021 06:55:01 INFO call get assignments stats function. 03/02/2021 06:55:11 INFO call get assignments stats function. 03/02/2021 06:55:21 INFO call get assignments stats function. 03/02/2021 06:55:32 INFO call get assignments stats function. 03/02/2021 06:55:42 INFO call get assignments stats function. 03/02/2021 06:55:52 INFO call get assignments stats function. 03/02/2021 06:56:03 INFO call get assignments stats function. 03/02/2021 06:56:13 INFO call get assignments stats function. 03/02/2021 06:56:23 INFO call get assignments stats function. 03/02/2021 06:56:33 INFO call get assignments stats function. 03/02/2021 06:56:43 INFO call get assignments stats function. 03/02/2021 06:56:53 INFO call get assignments stats function. 03/02/2021 06:57:03 INFO call get assignments stats function. 03/02/2021 06:57:13 INFO call get assignments stats function. 03/02/2021 06:57:23 INFO call get assignments stats function. 03/02/2021 06:57:33 INFO call get assignments stats function. 03/02/2021 06:57:44 INFO call get assignments stats function. 03/02/2021 06:57:54 INFO call get assignments stats function. 03/02/2021 06:58:04 INFO call get assignments stats function. 03/02/2021 06:58:14 INFO call get assignments stats function. 03/02/2021 06:58:24 INFO call get assignments stats function. 03/02/2021 06:58:34 INFO call get assignments stats function. 03/02/2021 06:58:44 INFO call get assignments stats function. 03/02/2021 06:58:55 INFO call get assignments stats function. 03/02/2021 06:59:05 INFO call get assignments stats function. 03/02/2021 06:59:15 INFO call get assignments stats function. 03/02/2021 06:59:25 INFO call get assignments stats function. 03/02/2021 06:59:35 INFO call get assignments stats function. 03/02/2021 06:59:45 INFO call get assignments stats function. 03/02/2021 06:59:55 INFO call get assignments stats function. 03/02/2021 07:00:05 INFO call get assignments stats function. 03/02/2021 07:00:15 INFO call get assignments stats function. 03/02/2021 07:00:25 INFO call get assignments stats function. 03/02/2021 07:00:35 INFO call get assignments stats function. 03/02/2021 07:00:45 INFO call get assignments stats function. 03/02/2021 07:00:55 INFO call get assignments stats function. 03/02/2021 07:01:05 INFO call get assignments stats function. 03/02/2021 07:01:15 INFO call get assignments stats function. 03/02/2021 07:01:45 INFO Checking backend latencies : 03/02/2021 07:01:45 INFO Network latency for backend 172.16.0.151 = 6.81 us 03/02/2021 07:01:45 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:01:45 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:01:47 INFO Checking backend latencies : 03/02/2021 07:01:47 INFO Network latency for backend 172.16.0.151 = 7.34 us 03/02/2021 07:01:47 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:01:47 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:01:48 INFO Checking backend latencies : 03/02/2021 07:01:48 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:01:48 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:01:48 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:01:49 INFO Checking backend latencies : 03/02/2021 07:01:49 INFO Network latency for backend 172.16.0.151 = 6.72 us 03/02/2021 07:01:49 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:01:49 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:01:53 INFO Checking backend latencies : 03/02/2021 07:01:53 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:01:53 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:01:53 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:02:03 INFO Checking backend latencies : 03/02/2021 07:02:03 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:02:03 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:02:03 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:02:09 INFO Checking backend latencies : 03/02/2021 07:02:09 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:02:09 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:02:09 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:02:11 INFO Checking backend latencies : 03/02/2021 07:02:11 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:02:11 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:02:11 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:02:12 INFO Checking backend latencies : 03/02/2021 07:02:12 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:02:12 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:02:12 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:02:13 INFO Checking backend latencies : 03/02/2021 07:02:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:02:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:02:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:02:13 INFO Checking backend latencies : 03/02/2021 07:02:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:02:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:02:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:02:18 INFO Checking backend latencies : 03/02/2021 07:02:18 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:02:18 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:02:18 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:02:23 INFO Checking backend latencies : 03/02/2021 07:02:23 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:02:23 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:02:23 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:02:28 INFO Checking backend latencies : 03/02/2021 07:02:28 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:02:28 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:02:28 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:02:35 INFO Checking backend latencies : 03/02/2021 07:02:35 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:02:35 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:02:35 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:02:37 INFO Checking backend latencies : 03/02/2021 07:02:37 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:02:37 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:02:37 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:02:38 INFO Checking backend latencies : 03/02/2021 07:02:38 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:02:38 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:02:38 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:02:38 INFO Checking backend latencies : 03/02/2021 07:02:38 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:02:38 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:02:38 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:02:39 INFO Checking backend latencies : 03/02/2021 07:02:39 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:02:39 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:02:39 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:02:43 INFO Checking backend latencies : 03/02/2021 07:02:43 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:02:43 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:02:43 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:02:44 INFO Checking backend latencies : 03/02/2021 07:02:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:02:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:02:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:02:48 INFO Checking backend latencies : 03/02/2021 07:02:48 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:02:48 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:02:48 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:02:49 INFO Checking backend latencies : 03/02/2021 07:02:49 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:02:49 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:02:49 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:02:54 INFO Checking backend latencies : 03/02/2021 07:02:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:02:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:02:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:02:54 INFO Checking backend latencies : 03/02/2021 07:02:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:02:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:02:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:03:03 INFO Checking backend latencies : 03/02/2021 07:03:03 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:03:03 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:03:03 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:03:04 INFO Checking backend latencies : 03/02/2021 07:03:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:03:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:03:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:03:05 INFO Checking backend latencies : 03/02/2021 07:03:05 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:03:05 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:03:05 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:03:05 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:03:07 INFO Checking backend latencies : 03/02/2021 07:03:07 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:03:07 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:03:07 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:03:08 INFO Checking backend latencies : 03/02/2021 07:03:08 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:03:08 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:03:08 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:03:09 INFO Checking backend latencies : 03/02/2021 07:03:09 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:03:09 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:03:09 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:03:09 ERROR Could not find key resource. 03/02/2021 07:03:09 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 496, in get_key_blocking index, data = consul_obj.get(key, lock_index, False, timeout) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:03:09 INFO Checking backend latencies : 03/02/2021 07:03:09 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:03:09 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:03:09 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:03:13 INFO Checking backend latencies : 03/02/2021 07:03:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:03:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:03:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:03:13 INFO Checking backend latencies : 03/02/2021 07:03:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:03:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:03:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:03:14 INFO Checking backend latencies : 03/02/2021 07:03:14 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:03:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:03:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:03:14 INFO Checking backend latencies : 03/02/2021 07:03:14 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:03:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:03:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:03:18 INFO Checking backend latencies : 03/02/2021 07:03:18 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:03:18 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:03:18 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:03:24 INFO Checking backend latencies : 03/02/2021 07:03:24 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:03:24 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:03:24 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:03:24 INFO Checking backend latencies : 03/02/2021 07:03:24 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:03:24 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:03:24 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:03:27 INFO Checking backend latencies : 03/02/2021 07:03:27 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:03:27 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:03:27 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:03:30 INFO Checking backend latencies : 03/02/2021 07:03:30 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:03:30 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:03:30 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:03:31 INFO Checking backend latencies : 03/02/2021 07:03:31 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:03:31 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:03:31 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:03:34 INFO Checking backend latencies : 03/02/2021 07:03:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:03:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:03:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:03:34 INFO Checking backend latencies : 03/02/2021 07:03:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:03:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:03:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:03:37 INFO Checking backend latencies : 03/02/2021 07:03:37 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:03:37 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:03:37 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:03:39 INFO Checking backend latencies : 03/02/2021 07:03:39 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:03:39 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:03:39 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:03:42 ERROR Node Stats exception. 03/02/2021 07:03:42 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 2.17 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 12.32 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 972.8 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 317.44 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 122.88 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 0.0 `date +%s`" | nc -q0 10.10.0.152 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 2.17 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 12.32 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 972.8 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 317.44 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 122.88 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 0.0 `date +%s`" | nc -q0 10.10.0.152 2003 03/02/2021 07:03:43 INFO Checking backend latencies : 03/02/2021 07:03:43 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:03:43 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:03:43 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:03:44 INFO Checking backend latencies : 03/02/2021 07:03:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:03:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:03:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:03:44 INFO Checking backend latencies : 03/02/2021 07:03:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:03:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:03:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:03:46 INFO Checking backend latencies : 03/02/2021 07:03:46 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:03:46 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:03:46 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:03:46 ERROR Could not start watch. 03/02/2021 07:03:48 INFO Checking backend latencies : 03/02/2021 07:03:48 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:03:48 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:03:48 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:03:48 INFO Checking backend latencies : 03/02/2021 07:03:48 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:03:48 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:03:48 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:03:48 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:03:49 INFO Checking backend latencies : 03/02/2021 07:03:49 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:03:49 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:03:49 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:03:52 INFO Checking backend latencies : 03/02/2021 07:03:52 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:03:52 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:03:52 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:03:52 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:03:53 INFO Checking backend latencies : 03/02/2021 07:03:53 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:03:53 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:03:53 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:03:54 INFO Checking backend latencies : 03/02/2021 07:03:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:03:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:03:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:03:54 INFO Checking backend latencies : 03/02/2021 07:03:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:03:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:03:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:03:55 INFO Checking backend latencies : 03/02/2021 07:03:55 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:03:55 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:03:55 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:04:02 INFO Checking backend latencies : 03/02/2021 07:04:02 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:02 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:02 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:04:02 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:04:03 INFO Checking backend latencies : 03/02/2021 07:04:03 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:03 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:03 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:04:04 INFO Checking backend latencies : 03/02/2021 07:04:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:04:04 INFO Checking backend latencies : 03/02/2021 07:04:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:04:04 INFO Checking backend latencies : 03/02/2021 07:04:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:04:08 INFO Checking backend latencies : 03/02/2021 07:04:08 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:08 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:08 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:04:10 INFO Checking backend latencies : 03/02/2021 07:04:10 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:10 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:10 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:04:10 INFO Checking backend latencies : 03/02/2021 07:04:10 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:10 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:10 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:04:12 INFO Checking backend latencies : 03/02/2021 07:04:12 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:12 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:12 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:04:12 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:04:13 INFO Checking backend latencies : 03/02/2021 07:04:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:04:13 INFO Checking backend latencies : 03/02/2021 07:04:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:04:14 INFO Checking backend latencies : 03/02/2021 07:04:14 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:04:14 INFO Checking backend latencies : 03/02/2021 07:04:14 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:04:18 INFO Checking backend latencies : 03/02/2021 07:04:18 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:18 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:18 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:04:20 INFO Checking backend latencies : 03/02/2021 07:04:20 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:20 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:20 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:04:21 INFO Checking backend latencies : 03/02/2021 07:04:21 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:21 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:21 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:04:22 INFO Checking backend latencies : 03/02/2021 07:04:22 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:22 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:22 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:04:22 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:04:23 INFO Checking backend latencies : 03/02/2021 07:04:23 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:23 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:23 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:04:24 INFO Checking backend latencies : 03/02/2021 07:04:24 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:24 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:24 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:04:28 INFO Checking backend latencies : 03/02/2021 07:04:28 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:28 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:28 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:04:29 INFO Checking backend latencies : 03/02/2021 07:04:29 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:29 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:29 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:04:34 INFO Checking backend latencies : 03/02/2021 07:04:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:04:34 INFO Checking backend latencies : 03/02/2021 07:04:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:04:34 INFO Checking backend latencies : 03/02/2021 07:04:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:04:38 INFO Checking backend latencies : 03/02/2021 07:04:38 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:38 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:38 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:04:39 INFO Checking backend latencies : 03/02/2021 07:04:39 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:39 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:39 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:04:42 INFO Checking backend latencies : 03/02/2021 07:04:42 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:42 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:42 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:04:42 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:04:43 INFO Checking backend latencies : 03/02/2021 07:04:43 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:43 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:43 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:04:44 INFO Checking backend latencies : 03/02/2021 07:04:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:04:44 INFO Checking backend latencies : 03/02/2021 07:04:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:04:48 INFO Checking backend latencies : 03/02/2021 07:04:48 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:48 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:48 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:04:48 INFO Checking backend latencies : 03/02/2021 07:04:48 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:48 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:48 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:04:48 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:04:49 INFO Checking backend latencies : 03/02/2021 07:04:49 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:49 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:49 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:04:50 INFO Checking backend latencies : 03/02/2021 07:04:50 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:50 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:50 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:04:52 INFO Checking backend latencies : 03/02/2021 07:04:52 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:52 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:52 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:04:52 INFO Checking backend latencies : 03/02/2021 07:04:52 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:52 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:52 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:04:52 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:04:53 INFO Checking backend latencies : 03/02/2021 07:04:53 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:53 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:53 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:04:54 INFO Checking backend latencies : 03/02/2021 07:04:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:04:55 INFO Checking backend latencies : 03/02/2021 07:04:55 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:55 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:55 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:05:00 INFO Checking backend latencies : 03/02/2021 07:05:00 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:05:00 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:05:00 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:05:02 INFO Checking backend latencies : 03/02/2021 07:05:02 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:05:02 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:05:02 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:05:02 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:05:03 INFO Checking backend latencies : 03/02/2021 07:05:03 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:05:03 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:05:03 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:05:04 INFO Checking backend latencies : 03/02/2021 07:05:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:05:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:05:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:05:04 INFO Checking backend latencies : 03/02/2021 07:05:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:05:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:05:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:05:07 INFO Checking backend latencies : 03/02/2021 07:05:07 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:05:07 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:05:07 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:05:10 INFO Checking backend latencies : 03/02/2021 07:05:10 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:05:10 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:05:10 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:05:12 INFO Checking backend latencies : 03/02/2021 07:05:12 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:05:12 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:05:12 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:05:12 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:05:13 INFO Checking backend latencies : 03/02/2021 07:05:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:05:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:05:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:05:13 INFO Checking backend latencies : 03/02/2021 07:05:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:05:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:05:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:05:14 INFO Checking backend latencies : 03/02/2021 07:05:14 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:05:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:05:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:05:14 INFO Checking backend latencies : 03/02/2021 07:05:14 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:05:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:05:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:05:18 INFO Checking backend latencies : 03/02/2021 07:05:18 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:05:18 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:05:18 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:05:23 INFO Checking backend latencies : 03/02/2021 07:05:23 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:05:23 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:05:23 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:05:23 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:05:23 INFO Checking backend latencies : 03/02/2021 07:05:23 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:05:23 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:05:23 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:05:24 INFO Checking backend latencies : 03/02/2021 07:05:24 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:05:24 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:05:24 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:05:25 INFO Checking backend latencies : 03/02/2021 07:05:25 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:05:25 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:05:25 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:05:28 INFO Checking backend latencies : 03/02/2021 07:05:28 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:05:28 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:05:28 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:05:28 INFO Checking backend latencies : 03/02/2021 07:05:28 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:05:28 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:05:28 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:05:28 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:05:30 INFO Checking backend latencies : 03/02/2021 07:05:30 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:05:30 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:05:30 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:05:30 ERROR LeaderElectionBase exception 03/02/2021 07:05:30 ERROR 03/02/2021 07:05:30 INFO Checking backend latencies : 03/02/2021 07:05:30 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:05:30 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:05:30 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:05:34 INFO Checking backend latencies : 03/02/2021 07:05:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:05:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:05:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:05:34 INFO Checking backend latencies : 03/02/2021 07:05:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:05:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:05:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:05:37 INFO Checking backend latencies : 03/02/2021 07:05:37 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:05:37 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:05:37 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:05:39 INFO Checking backend latencies : 03/02/2021 07:05:39 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:05:39 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:05:39 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:05:42 INFO Checking backend latencies : 03/02/2021 07:05:42 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:05:42 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:05:42 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:05:42 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:05:43 INFO Checking backend latencies : 03/02/2021 07:05:43 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:05:43 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:05:43 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:05:44 INFO Checking backend latencies : 03/02/2021 07:05:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:05:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:05:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:05:44 INFO Checking backend latencies : 03/02/2021 07:05:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:05:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:05:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:05:48 INFO Checking backend latencies : 03/02/2021 07:05:48 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:05:48 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:05:48 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:05:48 INFO Checking backend latencies : 03/02/2021 07:05:48 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:05:48 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:05:48 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:05:48 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:05:49 INFO Checking backend latencies : 03/02/2021 07:05:49 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:05:49 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:05:49 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:05:52 INFO Checking backend latencies : 03/02/2021 07:05:52 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:05:52 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:05:52 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:05:52 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:05:53 INFO Checking backend latencies : 03/02/2021 07:05:53 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:05:53 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:05:53 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:05:53 INFO Checking backend latencies : 03/02/2021 07:05:53 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:05:53 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:05:53 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:05:54 INFO Checking backend latencies : 03/02/2021 07:05:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:05:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:05:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:06:02 INFO Checking backend latencies : 03/02/2021 07:06:02 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:02 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:02 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:06:02 INFO Checking backend latencies : 03/02/2021 07:06:02 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:02 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:02 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:06:02 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:06:03 INFO Checking backend latencies : 03/02/2021 07:06:03 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:03 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:03 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:06:03 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader 03/02/2021 07:06:03 ERROR Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 2.17 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 12.32 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 972.8 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 317.44 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 122.88 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 0.0 `date +%s`" | nc -q0 10.10.0.152 2003 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 516, in get_key index, data = cons.get(key) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:06:04 INFO Checking backend latencies : 03/02/2021 07:06:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:06:04 INFO Checking backend latencies : 03/02/2021 07:06:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:06:04 INFO Checking backend latencies : 03/02/2021 07:06:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:06:08 INFO Checking backend latencies : 03/02/2021 07:06:08 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:08 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:08 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:06:09 INFO Checking backend latencies : 03/02/2021 07:06:09 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:09 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:09 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:06:09 ERROR Error on consul connection. 03/02/2021 07:06:09 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/iscsi_service.py", line 158, in start self.__current_lock_index = consul_api.current_index() File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 348, in current_index raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 345, in current_index index = consul_obj.get(self.root)[0] File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:06:09 INFO Checking backend latencies : 03/02/2021 07:06:09 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:09 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:09 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:06:12 INFO Checking backend latencies : 03/02/2021 07:06:12 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:12 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:12 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:06:12 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:06:13 INFO Checking backend latencies : 03/02/2021 07:06:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:06:13 INFO Checking backend latencies : 03/02/2021 07:06:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:06:14 INFO Checking backend latencies : 03/02/2021 07:06:14 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:06:14 INFO Checking backend latencies : 03/02/2021 07:06:14 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:06:17 INFO Checking backend latencies : 03/02/2021 07:06:17 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:17 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:17 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:06:18 INFO Checking backend latencies : 03/02/2021 07:06:18 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:18 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:18 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:06:22 INFO Checking backend latencies : 03/02/2021 07:06:22 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:22 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:22 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:06:22 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:06:24 INFO Checking backend latencies : 03/02/2021 07:06:24 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:24 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:24 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:06:24 INFO Checking backend latencies : 03/02/2021 07:06:24 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:24 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:24 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:06:26 INFO Checking backend latencies : 03/02/2021 07:06:26 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:26 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:26 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:06:28 INFO Checking backend latencies : 03/02/2021 07:06:28 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:28 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:28 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:06:34 INFO Checking backend latencies : 03/02/2021 07:06:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:06:34 INFO Checking backend latencies : 03/02/2021 07:06:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:06:38 INFO Checking backend latencies : 03/02/2021 07:06:38 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:38 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:38 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:06:39 INFO Checking backend latencies : 03/02/2021 07:06:39 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:39 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:39 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:06:41 INFO Checking backend latencies : 03/02/2021 07:06:41 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:41 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:41 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:06:42 INFO Checking backend latencies : 03/02/2021 07:06:42 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:42 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:42 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:06:42 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:06:43 INFO Checking backend latencies : 03/02/2021 07:06:43 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:43 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:43 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:06:44 INFO Checking backend latencies : 03/02/2021 07:06:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:06:44 INFO Checking backend latencies : 03/02/2021 07:06:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:06:44 INFO Checking backend latencies : 03/02/2021 07:06:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:06:48 INFO Checking backend latencies : 03/02/2021 07:06:48 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:48 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:48 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:06:48 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:06:48 INFO Checking backend latencies : 03/02/2021 07:06:48 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:48 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:48 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:06:49 INFO Checking backend latencies : 03/02/2021 07:06:49 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:49 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:49 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:06:52 INFO Checking backend latencies : 03/02/2021 07:06:52 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:52 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:52 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:06:52 INFO Checking backend latencies : 03/02/2021 07:06:52 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:52 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:52 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:06:52 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:06:53 INFO Checking backend latencies : 03/02/2021 07:06:53 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:53 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:53 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:06:54 INFO Checking backend latencies : 03/02/2021 07:06:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:07:02 INFO Checking backend latencies : 03/02/2021 07:07:02 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:07:02 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:07:02 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:07:02 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:07:03 INFO Checking backend latencies : 03/02/2021 07:07:03 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:07:03 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:07:03 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:07:04 INFO Checking backend latencies : 03/02/2021 07:07:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:07:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:07:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:07:04 INFO Checking backend latencies : 03/02/2021 07:07:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:07:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:07:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:07:05 INFO Checking backend latencies : 03/02/2021 07:07:05 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:07:05 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:07:05 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:07:08 INFO Checking backend latencies : 03/02/2021 07:07:08 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:07:08 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:07:08 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:07:09 INFO Checking backend latencies : 03/02/2021 07:07:09 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:07:09 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:07:09 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:07:12 INFO Checking backend latencies : 03/02/2021 07:07:12 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:07:12 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:07:12 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:07:12 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:07:13 INFO Checking backend latencies : 03/02/2021 07:07:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:07:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:07:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:07:13 INFO Checking backend latencies : 03/02/2021 07:07:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:07:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:07:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:07:13 INFO Checking backend latencies : 03/02/2021 07:07:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:07:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:07:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:07:14 INFO Checking backend latencies : 03/02/2021 07:07:14 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:07:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:07:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:07:14 INFO Checking backend latencies : 03/02/2021 07:07:14 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:07:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:07:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:07:18 INFO Checking backend latencies : 03/02/2021 07:07:18 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:07:18 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:07:18 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:07:22 INFO Checking backend latencies : 03/02/2021 07:07:22 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:07:22 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:07:22 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:07:22 INFO Checking backend latencies : 03/02/2021 07:07:22 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:07:22 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:07:22 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:07:22 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:07:23 INFO Checking backend latencies : 03/02/2021 07:07:23 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:07:23 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:07:23 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:07:24 INFO Checking backend latencies : 03/02/2021 07:07:24 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:07:24 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:07:24 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:07:26 INFO Checking backend latencies : 03/02/2021 07:07:26 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:07:26 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:07:26 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:07:28 INFO Checking backend latencies : 03/02/2021 07:07:28 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:07:28 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:07:28 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:07:31 INFO Checking backend latencies : 03/02/2021 07:07:31 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:07:31 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:07:31 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:07:34 INFO Checking backend latencies : 03/02/2021 07:07:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:07:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:07:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:07:34 INFO Checking backend latencies : 03/02/2021 07:07:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:07:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:07:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:07:38 INFO Checking backend latencies : 03/02/2021 07:07:38 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:07:38 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:07:38 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:07:39 INFO Checking backend latencies : 03/02/2021 07:07:39 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:07:39 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:07:39 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:07:42 INFO Checking backend latencies : 03/02/2021 07:07:42 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:07:42 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:07:42 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:07:42 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:07:43 INFO Checking backend latencies : 03/02/2021 07:07:43 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:07:43 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:07:43 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:07:44 INFO Checking backend latencies : 03/02/2021 07:07:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:07:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:07:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:07:44 INFO Checking backend latencies : 03/02/2021 07:07:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:07:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:07:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:07:48 INFO Checking backend latencies : 03/02/2021 07:07:48 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:07:48 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:07:48 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:07:48 INFO Checking backend latencies : 03/02/2021 07:07:48 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:07:48 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:07:48 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:07:48 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:07:49 INFO Checking backend latencies : 03/02/2021 07:07:49 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:07:49 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:07:49 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:07:50 INFO Checking backend latencies : 03/02/2021 07:07:50 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:07:50 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:07:50 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:07:51 INFO Checking backend latencies : 03/02/2021 07:07:51 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:07:51 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:07:51 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:07:51 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:07:52 INFO Checking backend latencies : 03/02/2021 07:07:52 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:07:52 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:07:52 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:07:52 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:07:53 INFO Checking backend latencies : 03/02/2021 07:07:53 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:07:53 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:07:53 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:07:54 INFO Checking backend latencies : 03/02/2021 07:07:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:07:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:07:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:08:00 INFO Checking backend latencies : 03/02/2021 07:08:00 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:00 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:00 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:08:00 ERROR Could not find key resource. 03/02/2021 07:08:00 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 496, in get_key_blocking index, data = consul_obj.get(key, lock_index, False, timeout) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:08:01 INFO Checking backend latencies : 03/02/2021 07:08:01 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:01 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:01 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:08:02 INFO Checking backend latencies : 03/02/2021 07:08:02 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:02 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:02 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:08:02 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:08:03 INFO Checking backend latencies : 03/02/2021 07:08:03 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:03 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:03 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:08:04 INFO Checking backend latencies : 03/02/2021 07:08:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:08:04 INFO Checking backend latencies : 03/02/2021 07:08:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:08:08 INFO Checking backend latencies : 03/02/2021 07:08:08 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:08 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:08 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:08:09 INFO Checking backend latencies : 03/02/2021 07:08:09 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:09 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:09 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:08:12 INFO Checking backend latencies : 03/02/2021 07:08:12 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:12 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:12 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:08:12 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:08:13 INFO Checking backend latencies : 03/02/2021 07:08:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:08:14 INFO Checking backend latencies : 03/02/2021 07:08:14 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:08:14 INFO Checking backend latencies : 03/02/2021 07:08:14 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:08:14 INFO Checking backend latencies : 03/02/2021 07:08:14 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:08:15 INFO Checking backend latencies : 03/02/2021 07:08:15 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:15 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:15 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:08:16 INFO Checking backend latencies : 03/02/2021 07:08:16 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:16 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:16 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:08:17 INFO Checking backend latencies : 03/02/2021 07:08:17 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:17 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:17 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:08:22 INFO Checking backend latencies : 03/02/2021 07:08:22 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:22 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:22 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:08:23 INFO Checking backend latencies : 03/02/2021 07:08:23 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:23 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:23 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:08:23 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:08:24 INFO Checking backend latencies : 03/02/2021 07:08:24 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:24 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:24 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:08:24 INFO Checking backend latencies : 03/02/2021 07:08:24 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:24 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:24 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:08:27 INFO Checking backend latencies : 03/02/2021 07:08:27 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:27 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:27 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:08:34 INFO Checking backend latencies : 03/02/2021 07:08:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:08:34 INFO Checking backend latencies : 03/02/2021 07:08:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:08:38 INFO Checking backend latencies : 03/02/2021 07:08:38 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:38 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:38 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:08:39 INFO Checking backend latencies : 03/02/2021 07:08:39 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:39 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:39 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:08:40 INFO Checking backend latencies : 03/02/2021 07:08:40 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:40 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:40 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:08:40 ERROR Error on consul connection. 03/02/2021 07:08:40 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/iscsi_service.py", line 158, in start self.__current_lock_index = consul_api.current_index() File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 348, in current_index raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 345, in current_index index = consul_obj.get(self.root)[0] File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:08:40 INFO Checking backend latencies : 03/02/2021 07:08:40 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:40 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:40 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:08:42 INFO Checking backend latencies : 03/02/2021 07:08:42 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:42 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:42 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:08:42 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:08:43 INFO Checking backend latencies : 03/02/2021 07:08:43 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:43 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:43 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:08:44 INFO Checking backend latencies : 03/02/2021 07:08:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:08:44 INFO Checking backend latencies : 03/02/2021 07:08:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:08:46 INFO Checking backend latencies : 03/02/2021 07:08:46 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:46 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:46 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:08:47 INFO Checking backend latencies : 03/02/2021 07:08:47 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:47 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:47 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:08:48 INFO Checking backend latencies : 03/02/2021 07:08:48 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:48 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:48 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:08:48 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:08:48 INFO Checking backend latencies : 03/02/2021 07:08:48 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:48 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:48 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:08:49 INFO Checking backend latencies : 03/02/2021 07:08:49 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:49 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:49 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:08:52 INFO Checking backend latencies : 03/02/2021 07:08:52 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:52 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:52 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:08:52 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:08:53 INFO Checking backend latencies : 03/02/2021 07:08:53 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:53 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:53 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:08:54 INFO Checking backend latencies : 03/02/2021 07:08:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:09:02 INFO Checking backend latencies : 03/02/2021 07:09:02 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:09:02 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:09:02 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:09:02 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:09:04 INFO Checking backend latencies : 03/02/2021 07:09:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:09:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:09:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:09:04 INFO Checking backend latencies : 03/02/2021 07:09:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:09:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:09:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:09:04 INFO Checking backend latencies : 03/02/2021 07:09:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:09:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:09:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:09:06 INFO Checking backend latencies : 03/02/2021 07:09:06 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:09:06 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:09:06 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:09:08 INFO Checking backend latencies : 03/02/2021 07:09:08 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:09:08 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:09:08 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:09:10 INFO Checking backend latencies : 03/02/2021 07:09:10 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:09:10 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:09:10 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:09:12 INFO Checking backend latencies : 03/02/2021 07:09:12 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:09:12 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:09:12 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:09:12 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:09:13 INFO Checking backend latencies : 03/02/2021 07:09:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:09:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:09:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:09:13 INFO Checking backend latencies : 03/02/2021 07:09:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:09:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:09:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:09:13 INFO Checking backend latencies : 03/02/2021 07:09:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:09:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:09:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:09:14 INFO Checking backend latencies : 03/02/2021 07:09:14 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:09:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:09:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:09:14 INFO Checking backend latencies : 03/02/2021 07:09:14 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:09:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:09:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:09:18 INFO Checking backend latencies : 03/02/2021 07:09:18 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:09:18 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:09:18 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:09:22 INFO Checking backend latencies : 03/02/2021 07:09:22 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:09:22 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:09:22 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:09:22 INFO Checking backend latencies : 03/02/2021 07:09:22 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:09:22 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:09:22 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:09:22 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:09:24 INFO Checking backend latencies : 03/02/2021 07:09:24 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:09:24 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:09:24 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:09:24 INFO Checking backend latencies : 03/02/2021 07:09:24 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:09:24 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:09:24 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:09:24 INFO Checking backend latencies : 03/02/2021 07:09:24 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:09:24 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:09:24 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:09:24 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader 03/02/2021 07:09:24 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 516, in get_key index, data = cons.get(key) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:09:28 INFO Checking backend latencies : 03/02/2021 07:09:28 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:09:28 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:09:28 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:09:34 INFO Checking backend latencies : 03/02/2021 07:09:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:09:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:09:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:09:34 INFO Checking backend latencies : 03/02/2021 07:09:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:09:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:09:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:09:36 INFO Checking backend latencies : 03/02/2021 07:09:36 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:09:36 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:09:36 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:09:37 INFO Checking backend latencies : 03/02/2021 07:09:37 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:09:37 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:09:37 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:09:39 INFO Checking backend latencies : 03/02/2021 07:09:39 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:09:39 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:09:39 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:09:43 INFO Checking backend latencies : 03/02/2021 07:09:43 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:09:43 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:09:43 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:09:43 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:09:43 INFO Checking backend latencies : 03/02/2021 07:09:43 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:09:43 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:09:43 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:09:44 INFO Checking backend latencies : 03/02/2021 07:09:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:09:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:09:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:09:44 INFO Checking backend latencies : 03/02/2021 07:09:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:09:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:09:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:09:44 INFO Checking backend latencies : 03/02/2021 07:09:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:09:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:09:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:09:46 INFO Checking backend latencies : 03/02/2021 07:09:46 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:09:46 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:09:46 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:09:48 INFO Checking backend latencies : 03/02/2021 07:09:48 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:09:48 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:09:48 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:09:48 INFO Checking backend latencies : 03/02/2021 07:09:48 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:09:48 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:09:48 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:09:48 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:09:49 INFO Checking backend latencies : 03/02/2021 07:09:49 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:09:49 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:09:49 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:09:52 INFO Checking backend latencies : 03/02/2021 07:09:52 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:09:52 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:09:52 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:09:52 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:09:55 INFO Checking backend latencies : 03/02/2021 07:09:55 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:09:55 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:09:55 INFO Network latency for backend 172.16.0.153 = 56.4 us 03/02/2021 07:09:57 INFO Checking backend latencies : 03/02/2021 07:09:57 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:09:57 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:09:57 INFO Network latency for backend 172.16.0.153 = 51.0 us 03/02/2021 07:10:03 INFO Checking backend latencies : 03/02/2021 07:10:03 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:10:03 INFO Network latency for backend 172.16.0.152 = 44.0 us 03/02/2021 07:10:03 INFO Network latency for backend 172.16.0.153 = 56.1 us 03/02/2021 07:10:03 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:10:04 INFO Checking backend latencies : 03/02/2021 07:10:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:10:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:10:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:10:05 INFO Checking backend latencies : 03/02/2021 07:10:05 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:10:05 INFO Network latency for backend 172.16.0.152 = 51.6 us 03/02/2021 07:10:05 INFO Network latency for backend 172.16.0.153 = 54.7 us 03/02/2021 07:10:05 INFO Checking backend latencies : 03/02/2021 07:10:05 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:10:05 INFO Network latency for backend 172.16.0.152 = 52.6 us 03/02/2021 07:10:05 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:10:08 INFO Checking backend latencies : 03/02/2021 07:10:08 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:10:08 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:10:08 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:10:10 INFO Checking backend latencies : 03/02/2021 07:10:10 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:10:10 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:10:10 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:10:12 INFO Checking backend latencies : 03/02/2021 07:10:12 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:10:12 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:10:12 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:10:13 INFO Checking backend latencies : 03/02/2021 07:10:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:10:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:10:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:10:13 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:10:13 INFO Checking backend latencies : 03/02/2021 07:10:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:10:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:10:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:10:14 INFO Checking backend latencies : 03/02/2021 07:10:14 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:10:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:10:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:10:26 INFO call get assignments stats function. 03/02/2021 07:10:37 INFO call get assignments stats function. 03/02/2021 07:10:56 INFO call get assignments stats function. 03/02/2021 07:10:56 INFO call get assignments stats function. 03/02/2021 07:10:56 INFO call get assignments stats function. 03/02/2021 07:10:56 INFO call get assignments stats function. 03/02/2021 07:10:57 INFO call get assignments stats function. 03/02/2021 07:10:58 INFO call get assignments stats function. 03/02/2021 07:10:58 INFO call get assignments stats function. 03/02/2021 07:10:58 INFO call get assignments stats function. 03/02/2021 07:10:58 INFO call get assignments stats function. 03/02/2021 07:10:58 INFO call get assignments stats function. 03/02/2021 07:10:58 INFO call get assignments stats function. 03/02/2021 07:10:58 INFO call get assignments stats function. 03/02/2021 07:11:00 INFO call get assignments stats function. 03/02/2021 07:11:04 INFO Found pool:rbd for disk:00001 via consul 03/02/2021 07:11:04 INFO Could not lock path 00001/1 with session b9a49e91-fd0d-d843-8a93-e4fce7ccbc8b. 03/02/2021 07:11:09 INFO call get assignments stats function. 03/02/2021 07:11:10 INFO call get assignments stats function. 03/02/2021 07:11:10 INFO call get assignments stats function. 03/02/2021 07:11:14 INFO call get assignments stats function. 03/02/2021 07:11:15 INFO call get assignments stats function. 03/02/2021 07:11:20 INFO call get assignments stats function. 03/02/2021 07:11:23 INFO call get assignments stats function. 03/02/2021 07:11:25 INFO call get assignments stats function. 03/02/2021 07:11:25 INFO call get assignments stats function. 03/02/2021 07:11:28 INFO call get assignments stats function. 03/02/2021 07:11:28 INFO call get assignments stats function. 03/02/2021 07:11:30 INFO call get assignments stats function. 03/02/2021 07:11:35 INFO call get assignments stats function. 03/02/2021 07:11:46 INFO call get assignments stats function. 03/02/2021 07:11:55 INFO call get assignments stats function. 03/02/2021 07:12:05 INFO call get assignments stats function. 03/02/2021 07:12:27 INFO Checking backend latencies : 03/02/2021 07:12:27 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:12:27 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:12:27 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:12:34 INFO Checking backend latencies : 03/02/2021 07:12:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:12:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:12:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:12:44 INFO Checking backend latencies : 03/02/2021 07:12:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:12:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:12:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:12:44 INFO Checking backend latencies : 03/02/2021 07:12:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:12:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:12:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:12:50 INFO Checking backend latencies : 03/02/2021 07:12:50 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:12:50 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:12:50 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:12:51 INFO Checking backend latencies : 03/02/2021 07:12:51 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:12:51 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:12:51 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:12:54 INFO Checking backend latencies : 03/02/2021 07:12:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:12:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:12:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:12:59 INFO Checking backend latencies : 03/02/2021 07:12:59 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:12:59 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:12:59 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:13:04 INFO Checking backend latencies : 03/02/2021 07:13:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:13:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:13:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:13:08 INFO Checking backend latencies : 03/02/2021 07:13:08 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:13:08 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:13:08 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:13:08 INFO Checking backend latencies : 03/02/2021 07:13:08 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:13:08 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:13:08 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:13:14 INFO Checking backend latencies : 03/02/2021 07:13:14 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:13:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:13:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:13:14 INFO Checking backend latencies : 03/02/2021 07:13:14 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:13:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:13:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:13:18 INFO Checking backend latencies : 03/02/2021 07:13:18 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:13:18 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:13:18 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:13:18 INFO Checking backend latencies : 03/02/2021 07:13:18 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:13:18 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:13:18 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:13:25 INFO Checking backend latencies : 03/02/2021 07:13:25 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:13:25 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:13:25 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:13:28 INFO Checking backend latencies : 03/02/2021 07:13:28 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:13:28 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:13:28 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:13:34 INFO Checking backend latencies : 03/02/2021 07:13:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:13:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:13:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:13:34 INFO Checking backend latencies : 03/02/2021 07:13:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:13:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:13:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:13:34 INFO Checking backend latencies : 03/02/2021 07:13:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:13:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:13:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:13:34 INFO Checking backend latencies : 03/02/2021 07:13:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:13:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:13:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:13:38 INFO Checking backend latencies : 03/02/2021 07:13:38 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:13:38 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:13:38 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:13:40 INFO Checking backend latencies : 03/02/2021 07:13:40 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:13:40 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:13:40 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:13:44 INFO Checking backend latencies : 03/02/2021 07:13:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:13:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:13:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:13:44 INFO Checking backend latencies : 03/02/2021 07:13:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:13:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:13:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:13:48 INFO Checking backend latencies : 03/02/2021 07:13:48 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:13:48 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:13:48 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:13:54 INFO Checking backend latencies : 03/02/2021 07:13:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:13:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:13:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:13:54 INFO Checking backend latencies : 03/02/2021 07:13:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:13:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:13:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:13:55 INFO Checking backend latencies : 03/02/2021 07:13:55 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:13:55 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:13:55 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:13:58 INFO Checking backend latencies : 03/02/2021 07:13:58 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:13:58 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:13:58 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:13:58 INFO Checking backend latencies : 03/02/2021 07:13:58 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:13:58 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:13:58 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:14:04 INFO Checking backend latencies : 03/02/2021 07:14:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:14:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:14:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:14:04 INFO Checking backend latencies : 03/02/2021 07:14:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:14:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:14:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:14:04 INFO Checking backend latencies : 03/02/2021 07:14:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:14:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:14:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:14:04 INFO Checking backend latencies : 03/02/2021 07:14:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:14:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:14:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:14:08 INFO Checking backend latencies : 03/02/2021 07:14:08 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:14:08 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:14:08 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:14:10 INFO Checking backend latencies : 03/02/2021 07:14:10 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:14:10 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:14:10 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:14:14 INFO Checking backend latencies : 03/02/2021 07:14:14 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:14:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:14:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:14:14 INFO Checking backend latencies : 03/02/2021 07:14:14 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:14:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:14:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:14:18 INFO Checking backend latencies : 03/02/2021 07:14:18 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:14:18 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:14:18 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:14:24 INFO Checking backend latencies : 03/02/2021 07:14:24 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:14:24 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:14:24 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:14:25 INFO Checking backend latencies : 03/02/2021 07:14:25 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:14:25 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:14:25 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:14:25 INFO Checking backend latencies : 03/02/2021 07:14:25 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:14:25 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:14:25 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:14:26 INFO Checking backend latencies : 03/02/2021 07:14:26 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:14:26 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:14:26 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:14:26 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 134, in open_session return ConsulSession(sid,"") File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 39, in __init__ self.permanent= True File "/usr/lib/python3/dist-packages/flask/sessions.py", line 34, in _set_permanent self['_permanent'] = bool(value) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 53, in __setitem__ consul_base_api.write_value("".join( [consul_session_key,self.sid,"/",key]),str(value)) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 52, in write_value cons.kv.put(key, value) File "/usr/lib/python3/dist-packages/consul/base.py", line 621, in put CB.json(), '/v1/kv/%s' % key, params=params, data=value) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 107, in put raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 96, in put raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:14:28 INFO Checking backend latencies : 03/02/2021 07:14:28 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:14:28 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:14:28 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:14:33 INFO Checking backend latencies : 03/02/2021 07:14:33 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:14:33 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:14:33 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:14:33 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:14:34 INFO Checking backend latencies : 03/02/2021 07:14:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:14:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:14:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:14:34 INFO Checking backend latencies : 03/02/2021 07:14:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:14:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:14:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:14:34 INFO Checking backend latencies : 03/02/2021 07:14:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:14:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:14:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:14:34 INFO Checking backend latencies : 03/02/2021 07:14:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:14:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:14:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:14:38 INFO Checking backend latencies : 03/02/2021 07:14:38 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:14:38 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:14:38 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:14:42 INFO Checking backend latencies : 03/02/2021 07:14:42 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:14:42 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:14:42 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:14:42 ERROR Error on consul connection. 03/02/2021 07:14:42 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/iscsi_service.py", line 158, in start self.__current_lock_index = consul_api.current_index() File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 348, in current_index raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 345, in current_index index = consul_obj.get(self.root)[0] File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:14:43 INFO Checking backend latencies : 03/02/2021 07:14:43 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:14:43 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:14:43 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:14:43 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:14:44 INFO Checking backend latencies : 03/02/2021 07:14:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:14:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:14:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:14:44 INFO Checking backend latencies : 03/02/2021 07:14:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:14:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:14:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:14:48 INFO Checking backend latencies : 03/02/2021 07:14:48 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:14:48 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:14:48 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:14:48 ERROR Could not find key resource. 03/02/2021 07:14:48 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 496, in get_key_blocking index, data = consul_obj.get(key, lock_index, False, timeout) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:14:53 INFO Checking backend latencies : 03/02/2021 07:14:53 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:14:53 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:14:53 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:14:53 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:14:54 INFO Checking backend latencies : 03/02/2021 07:14:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:14:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:14:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:14:54 INFO Checking backend latencies : 03/02/2021 07:14:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:14:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:14:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:14:55 INFO Checking backend latencies : 03/02/2021 07:14:55 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:14:55 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:14:55 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:14:55 INFO Checking backend latencies : 03/02/2021 07:14:55 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:14:55 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:14:55 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:14:58 INFO Checking backend latencies : 03/02/2021 07:14:58 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:14:58 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:14:58 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:14:58 INFO Checking backend latencies : 03/02/2021 07:14:58 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:14:58 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:14:58 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:15:02 INFO Checking backend latencies : 03/02/2021 07:15:02 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:15:02 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:15:02 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:15:02 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:15:04 INFO Checking backend latencies : 03/02/2021 07:15:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:15:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:15:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:15:04 INFO Checking backend latencies : 03/02/2021 07:15:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:15:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:15:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:15:05 INFO Checking backend latencies : 03/02/2021 07:15:05 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:15:05 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:15:05 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:15:08 INFO Checking backend latencies : 03/02/2021 07:15:08 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:15:08 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:15:08 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:15:10 INFO Checking backend latencies : 03/02/2021 07:15:10 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:15:10 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:15:10 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:15:13 INFO Checking backend latencies : 03/02/2021 07:15:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:15:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:15:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:15:13 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:15:14 INFO Checking backend latencies : 03/02/2021 07:15:14 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:15:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:15:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:15:14 INFO Checking backend latencies : 03/02/2021 07:15:14 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:15:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:15:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:15:15 INFO Checking backend latencies : 03/02/2021 07:15:15 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:15:15 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:15:15 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:15:18 INFO Checking backend latencies : 03/02/2021 07:15:18 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:15:18 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:15:18 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:15:23 INFO Checking backend latencies : 03/02/2021 07:15:23 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:15:23 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:15:23 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:15:24 INFO Checking backend latencies : 03/02/2021 07:15:24 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:15:24 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:15:24 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:15:24 INFO Checking backend latencies : 03/02/2021 07:15:24 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:15:24 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:15:24 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:15:25 INFO Checking backend latencies : 03/02/2021 07:15:25 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:15:25 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:15:25 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:15:28 INFO Checking backend latencies : 03/02/2021 07:15:28 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:15:28 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:15:28 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:15:33 INFO Checking backend latencies : 03/02/2021 07:15:33 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:15:33 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:15:33 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:15:33 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:15:33 INFO Checking backend latencies : 03/02/2021 07:15:33 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:15:33 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:15:33 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:15:33 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:15:34 INFO Checking backend latencies : 03/02/2021 07:15:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:15:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:15:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:15:34 INFO Checking backend latencies : 03/02/2021 07:15:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:15:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:15:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:15:34 INFO Checking backend latencies : 03/02/2021 07:15:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:15:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:15:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:15:35 INFO Checking backend latencies : 03/02/2021 07:15:35 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:15:35 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:15:35 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:15:35 INFO Checking backend latencies : 03/02/2021 07:15:35 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:15:35 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:15:35 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:15:38 INFO Checking backend latencies : 03/02/2021 07:15:38 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:15:38 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:15:38 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:15:39 INFO Checking backend latencies : 03/02/2021 07:15:39 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:15:39 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:15:39 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:15:43 INFO Checking backend latencies : 03/02/2021 07:15:43 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:15:43 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:15:43 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:15:43 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:15:44 INFO Checking backend latencies : 03/02/2021 07:15:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:15:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:15:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:15:44 INFO Checking backend latencies : 03/02/2021 07:15:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:15:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:15:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:15:47 INFO Checking backend latencies : 03/02/2021 07:15:47 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:15:47 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:15:47 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:15:51 ERROR Node Stats exception. 03/02/2021 07:15:51 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 7.19 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 14.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1392.64 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 81.92 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 61.44 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 102.4 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 0.0 `date +%s`" | nc -q0 10.10.0.153 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 7.19 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 14.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1392.64 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 81.92 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 61.44 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 102.4 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 0.0 `date +%s`" | nc -q0 10.10.0.153 2003 03/02/2021 07:15:52 INFO Checking backend latencies : 03/02/2021 07:15:52 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:15:52 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:15:52 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:15:52 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:15:54 INFO Checking backend latencies : 03/02/2021 07:15:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:15:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:15:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:15:54 INFO Checking backend latencies : 03/02/2021 07:15:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:15:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:15:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:15:54 INFO Checking backend latencies : 03/02/2021 07:15:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:15:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:15:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:15:55 INFO Checking backend latencies : 03/02/2021 07:15:55 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:15:55 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:15:55 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:15:56 INFO Checking backend latencies : 03/02/2021 07:15:56 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:15:56 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:15:56 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:15:56 INFO Checking backend latencies : 03/02/2021 07:15:56 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:15:56 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:15:56 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:15:58 INFO Checking backend latencies : 03/02/2021 07:15:58 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:15:58 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:15:58 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:15:58 INFO Checking backend latencies : 03/02/2021 07:15:58 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:15:58 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:15:58 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:01 INFO Checking backend latencies : 03/02/2021 07:16:01 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:01 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:01 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:03 INFO Checking backend latencies : 03/02/2021 07:16:03 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:03 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:03 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:03 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:16:04 INFO Checking backend latencies : 03/02/2021 07:16:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:04 INFO Checking backend latencies : 03/02/2021 07:16:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:04 INFO Checking backend latencies : 03/02/2021 07:16:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:05 INFO Checking backend latencies : 03/02/2021 07:16:05 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:05 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:05 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:08 INFO Checking backend latencies : 03/02/2021 07:16:08 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:08 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:08 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:13 INFO Checking backend latencies : 03/02/2021 07:16:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:13 INFO Checking backend latencies : 03/02/2021 07:16:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:13 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:16:13 INFO Checking backend latencies : 03/02/2021 07:16:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:14 INFO Checking backend latencies : 03/02/2021 07:16:14 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:14 INFO Checking backend latencies : 03/02/2021 07:16:14 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:18 INFO Checking backend latencies : 03/02/2021 07:16:18 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:18 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:18 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:20 INFO Checking backend latencies : 03/02/2021 07:16:20 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:20 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:20 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:21 INFO Checking backend latencies : 03/02/2021 07:16:21 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:21 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:21 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:24 INFO Checking backend latencies : 03/02/2021 07:16:24 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:24 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:24 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:24 INFO Checking backend latencies : 03/02/2021 07:16:24 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:24 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:24 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:24 INFO Checking backend latencies : 03/02/2021 07:16:24 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:24 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:24 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:25 INFO Checking backend latencies : 03/02/2021 07:16:25 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:25 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:25 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:28 INFO Checking backend latencies : 03/02/2021 07:16:28 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:28 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:28 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:31 INFO Checking backend latencies : 03/02/2021 07:16:31 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:31 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:31 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:32 INFO Checking backend latencies : 03/02/2021 07:16:32 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:32 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:32 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:32 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:16:33 INFO Checking backend latencies : 03/02/2021 07:16:33 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:33 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:33 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:33 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:16:34 INFO Checking backend latencies : 03/02/2021 07:16:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:34 INFO Checking backend latencies : 03/02/2021 07:16:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:34 INFO Checking backend latencies : 03/02/2021 07:16:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:35 INFO Checking backend latencies : 03/02/2021 07:16:35 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:35 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:35 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:37 INFO Checking backend latencies : 03/02/2021 07:16:37 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:37 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:37 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:38 INFO Checking backend latencies : 03/02/2021 07:16:38 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:38 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:38 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:42 INFO Checking backend latencies : 03/02/2021 07:16:42 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:42 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:42 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:42 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:16:44 INFO Checking backend latencies : 03/02/2021 07:16:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:44 INFO Checking backend latencies : 03/02/2021 07:16:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:47 INFO Checking backend latencies : 03/02/2021 07:16:47 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:47 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:47 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:47 INFO Checking backend latencies : 03/02/2021 07:16:47 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:47 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:47 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:48 INFO Checking backend latencies : 03/02/2021 07:16:48 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:48 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:48 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:48 INFO Checking backend latencies : 03/02/2021 07:16:48 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:48 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:48 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:53 INFO Checking backend latencies : 03/02/2021 07:16:53 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:53 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:53 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:53 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:16:54 INFO Checking backend latencies : 03/02/2021 07:16:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:54 INFO Checking backend latencies : 03/02/2021 07:16:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:54 INFO Checking backend latencies : 03/02/2021 07:16:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:55 INFO Checking backend latencies : 03/02/2021 07:16:55 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:55 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:55 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:58 INFO Checking backend latencies : 03/02/2021 07:16:58 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:58 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:58 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:03 INFO Checking backend latencies : 03/02/2021 07:17:03 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:03 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:03 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:03 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:17:03 INFO Checking backend latencies : 03/02/2021 07:17:03 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:03 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:03 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:04 INFO Checking backend latencies : 03/02/2021 07:17:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:04 INFO Checking backend latencies : 03/02/2021 07:17:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:05 INFO Checking backend latencies : 03/02/2021 07:17:05 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:05 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:05 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:05 INFO Checking backend latencies : 03/02/2021 07:17:05 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:05 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:05 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:09 INFO Checking backend latencies : 03/02/2021 07:17:09 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:09 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:09 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:09 ERROR LeaderElectionBase exception 03/02/2021 07:17:09 ERROR 03/02/2021 07:17:11 INFO Checking backend latencies : 03/02/2021 07:17:11 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:11 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:11 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:12 INFO Checking backend latencies : 03/02/2021 07:17:12 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:12 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:12 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:13 INFO Checking backend latencies : 03/02/2021 07:17:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:13 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:17:13 INFO Checking backend latencies : 03/02/2021 07:17:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:13 ERROR Error on consul connection. 03/02/2021 07:17:13 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/iscsi_service.py", line 158, in start self.__current_lock_index = consul_api.current_index() File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 348, in current_index raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 345, in current_index index = consul_obj.get(self.root)[0] File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:17:14 INFO Checking backend latencies : 03/02/2021 07:17:14 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:14 INFO Checking backend latencies : 03/02/2021 07:17:14 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:15 INFO Checking backend latencies : 03/02/2021 07:17:15 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:15 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:15 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:17 INFO Checking backend latencies : 03/02/2021 07:17:17 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:17 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:17 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:17 INFO Checking backend latencies : 03/02/2021 07:17:17 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:17 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:17 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:17 INFO Checking backend latencies : 03/02/2021 07:17:17 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:17 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:17 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:18 INFO Checking backend latencies : 03/02/2021 07:17:18 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:18 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:18 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:22 INFO Checking backend latencies : 03/02/2021 07:17:22 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:22 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:22 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:22 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:17:24 INFO Checking backend latencies : 03/02/2021 07:17:24 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:24 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:24 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:24 INFO Checking backend latencies : 03/02/2021 07:17:24 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:24 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:24 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:25 INFO Checking backend latencies : 03/02/2021 07:17:25 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:25 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:25 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:28 INFO Checking backend latencies : 03/02/2021 07:17:28 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:28 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:28 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:29 INFO Checking backend latencies : 03/02/2021 07:17:29 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:29 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:29 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:33 INFO Checking backend latencies : 03/02/2021 07:17:33 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:33 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:33 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:33 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:17:33 INFO Checking backend latencies : 03/02/2021 07:17:33 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:33 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:33 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:33 INFO Checking backend latencies : 03/02/2021 07:17:33 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:33 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:33 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:33 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:17:34 INFO Checking backend latencies : 03/02/2021 07:17:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:35 INFO Checking backend latencies : 03/02/2021 07:17:35 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:35 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:35 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:35 INFO Checking backend latencies : 03/02/2021 07:17:35 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:35 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:35 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:38 INFO Checking backend latencies : 03/02/2021 07:17:38 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:38 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:38 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:38 INFO Checking backend latencies : 03/02/2021 07:17:38 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:38 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:38 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:42 INFO Checking backend latencies : 03/02/2021 07:17:42 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:42 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:42 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:42 INFO Checking backend latencies : 03/02/2021 07:17:42 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:42 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:42 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:43 INFO Checking backend latencies : 03/02/2021 07:17:43 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:43 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:43 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:43 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:17:44 INFO Checking backend latencies : 03/02/2021 07:17:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:44 INFO Checking backend latencies : 03/02/2021 07:17:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:45 INFO Checking backend latencies : 03/02/2021 07:17:45 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:45 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:45 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:46 INFO Checking backend latencies : 03/02/2021 07:17:46 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:46 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:46 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:48 INFO Checking backend latencies : 03/02/2021 07:17:48 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:48 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:48 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:52 INFO Checking backend latencies : 03/02/2021 07:17:52 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:52 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:52 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:53 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:17:54 INFO Checking backend latencies : 03/02/2021 07:17:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:54 INFO Checking backend latencies : 03/02/2021 07:17:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:55 INFO Checking backend latencies : 03/02/2021 07:17:55 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:55 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:55 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:55 INFO Checking backend latencies : 03/02/2021 07:17:55 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:55 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:55 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:55 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:17:55 INFO Checking backend latencies : 03/02/2021 07:17:55 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:55 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:55 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:55 INFO Checking backend latencies : 03/02/2021 07:17:55 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:55 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:55 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:55 INFO Checking backend latencies : 03/02/2021 07:17:55 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:55 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:55 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:55 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:17:58 INFO Checking backend latencies : 03/02/2021 07:17:58 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:58 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:58 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:18:01 INFO Checking backend latencies : 03/02/2021 07:18:01 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:18:01 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:18:01 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:18:03 INFO Checking backend latencies : 03/02/2021 07:18:03 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:18:03 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:18:03 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:18:03 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:18:04 INFO Checking backend latencies : 03/02/2021 07:18:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:18:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:18:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:18:05 INFO Checking backend latencies : 03/02/2021 07:18:05 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:18:05 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:18:05 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:18:06 INFO Checking backend latencies : 03/02/2021 07:18:06 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:18:06 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:18:06 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:18:08 INFO Checking backend latencies : 03/02/2021 07:18:08 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:18:08 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:18:08 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:18:08 INFO Checking backend latencies : 03/02/2021 07:18:08 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:18:08 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:18:08 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:18:08 INFO Checking backend latencies : 03/02/2021 07:18:08 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:18:08 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:18:08 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:18:11 INFO Checking backend latencies : 03/02/2021 07:18:11 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:18:11 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:18:11 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:18:11 INFO Checking backend latencies : 03/02/2021 07:18:11 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:18:11 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:18:11 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:18:11 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader 03/02/2021 07:18:11 ERROR Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 7.19 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 14.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1392.64 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 81.92 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 61.44 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 102.4 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 0.0 `date +%s`" | nc -q0 10.10.0.153 2003 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 516, in get_key index, data = cons.get(key) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:18:13 INFO Checking backend latencies : 03/02/2021 07:18:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:18:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:18:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:18:13 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:18:15 INFO Checking backend latencies : 03/02/2021 07:18:15 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:18:15 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:18:15 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:18:15 INFO Checking backend latencies : 03/02/2021 07:18:15 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:18:15 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:18:15 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:18:18 INFO Checking backend latencies : 03/02/2021 07:18:18 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:18:18 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:18:18 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:18:18 INFO Checking backend latencies : 03/02/2021 07:18:18 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:18:18 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:18:18 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:18:19 INFO Checking backend latencies : 03/02/2021 07:18:19 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:18:19 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:18:19 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:18:23 INFO Checking backend latencies : 03/02/2021 07:18:23 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:18:23 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:18:23 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:18:23 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:18:24 INFO Checking backend latencies : 03/02/2021 07:18:24 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:18:24 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:18:24 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:18:24 INFO Checking backend latencies : 03/02/2021 07:18:24 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:18:24 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:18:24 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:18:25 INFO Checking backend latencies : 03/02/2021 07:18:25 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:18:25 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:18:25 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:18:29 INFO Checking backend latencies : 03/02/2021 07:18:29 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:18:29 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:18:29 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:18:31 INFO Checking backend latencies : 03/02/2021 07:18:31 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:18:31 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:18:31 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:18:32 INFO Checking backend latencies : 03/02/2021 07:18:32 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:18:32 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:18:32 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:18:33 INFO Checking backend latencies : 03/02/2021 07:18:33 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:18:33 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:18:33 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:18:33 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:18:34 INFO Checking backend latencies : 03/02/2021 07:18:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:18:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:18:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:18:35 INFO Checking backend latencies : 03/02/2021 07:18:35 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:18:35 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:18:35 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:18:37 INFO Checking backend latencies : 03/02/2021 07:18:37 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:18:37 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:18:37 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:18:38 INFO Checking backend latencies : 03/02/2021 07:18:38 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:18:38 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:18:38 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:18:42 INFO Checking backend latencies : 03/02/2021 07:18:42 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:18:42 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:18:42 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:18:45 INFO Checking backend latencies : 03/02/2021 07:18:45 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:18:45 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:18:45 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:18:45 INFO Checking backend latencies : 03/02/2021 07:18:45 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:18:45 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:18:45 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:18:46 INFO Checking backend latencies : 03/02/2021 07:18:46 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:18:46 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:18:46 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:18:47 INFO Checking backend latencies : 03/02/2021 07:18:47 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:18:47 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:18:47 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:18:47 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:18:53 INFO Checking backend latencies : 03/02/2021 07:18:53 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:18:53 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:18:53 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:18:53 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:18:55 INFO Checking backend latencies : 03/02/2021 07:18:55 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:18:55 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:18:55 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:18:55 INFO Checking backend latencies : 03/02/2021 07:18:55 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:18:55 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:18:55 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:19:02 INFO Checking backend latencies : 03/02/2021 07:19:02 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:19:02 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:19:02 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:19:03 INFO Checking backend latencies : 03/02/2021 07:19:03 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:19:03 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:19:03 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:19:03 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:19:04 INFO Checking backend latencies : 03/02/2021 07:19:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:19:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:19:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:19:04 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:19:05 INFO Checking backend latencies : 03/02/2021 07:19:05 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:19:05 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:19:05 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:19:07 INFO Checking backend latencies : 03/02/2021 07:19:07 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:19:07 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:19:07 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:19:08 INFO Checking backend latencies : 03/02/2021 07:19:08 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:19:08 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:19:08 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:19:10 INFO Checking backend latencies : 03/02/2021 07:19:10 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:19:10 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:19:10 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:19:10 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:19:13 INFO Checking backend latencies : 03/02/2021 07:19:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:19:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:19:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:19:13 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:19:15 INFO Checking backend latencies : 03/02/2021 07:19:15 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:19:15 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:19:15 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:19:16 INFO Checking backend latencies : 03/02/2021 07:19:16 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:19:16 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:19:16 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:19:17 INFO Checking backend latencies : 03/02/2021 07:19:17 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:19:17 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:19:17 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:19:17 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:19:20 INFO Checking backend latencies : 03/02/2021 07:19:20 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:19:20 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:19:20 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:19:23 INFO Checking backend latencies : 03/02/2021 07:19:23 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:19:23 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:19:23 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:19:23 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:19:25 INFO Checking backend latencies : 03/02/2021 07:19:25 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:19:25 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:19:25 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:19:33 INFO Checking backend latencies : 03/02/2021 07:19:33 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:19:33 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:19:33 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:19:33 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:19:33 INFO Checking backend latencies : 03/02/2021 07:19:33 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:19:33 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:19:33 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:19:38 INFO Checking backend latencies : 03/02/2021 07:19:38 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:19:38 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:19:38 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:19:40 INFO Checking backend latencies : 03/02/2021 07:19:40 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:19:40 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:19:40 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:19:40 ERROR Could not find key resource. 03/02/2021 07:19:40 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 496, in get_key_blocking index, data = consul_obj.get(key, lock_index, False, timeout) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:19:43 INFO Checking backend latencies : 03/02/2021 07:19:43 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:19:43 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:19:43 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:19:43 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:19:44 INFO Checking backend latencies : 03/02/2021 07:19:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:19:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:19:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:19:45 INFO Checking backend latencies : 03/02/2021 07:19:45 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:19:45 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:19:45 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:19:45 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:19:53 INFO Checking backend latencies : 03/02/2021 07:19:53 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:19:53 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:19:53 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:19:53 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:19:54 INFO Checking backend latencies : 03/02/2021 07:19:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:19:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:19:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:19:54 ERROR Error on consul connection. 03/02/2021 07:19:54 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/iscsi_service.py", line 158, in start self.__current_lock_index = consul_api.current_index() File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 348, in current_index raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 345, in current_index index = consul_obj.get(self.root)[0] File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:19:58 INFO Checking backend latencies : 03/02/2021 07:19:58 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:19:58 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:19:58 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:20:02 INFO Checking backend latencies : 03/02/2021 07:20:02 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:20:02 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:20:02 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:20:03 INFO Checking backend latencies : 03/02/2021 07:20:03 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:20:03 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:20:03 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:20:03 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:20:09 WARNING PetaSAN could not complete process, there are too many exceptions. 03/02/2021 07:20:09 INFO Checking backend latencies : 03/02/2021 07:20:09 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:20:09 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:20:09 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:20:10 INFO Checking backend latencies : 03/02/2021 07:20:10 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:20:10 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:20:10 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:20:17 INFO Checking backend latencies : 03/02/2021 07:20:17 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:20:17 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:20:17 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:20:17 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:20:24 INFO Checking backend latencies : 03/02/2021 07:20:24 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:20:24 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:20:24 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:20:27 INFO Checking backend latencies : 03/02/2021 07:20:27 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:20:27 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:20:27 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:20:32 INFO Checking backend latencies : 03/02/2021 07:20:32 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:20:32 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:20:32 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:20:33 INFO Checking backend latencies : 03/02/2021 07:20:33 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:20:33 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:20:33 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:20:35 INFO Checking backend latencies : 03/02/2021 07:20:35 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:20:35 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:20:35 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:20:41 INFO Checking backend latencies : 03/02/2021 07:20:41 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:20:41 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:20:41 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:20:53 INFO Checking backend latencies : 03/02/2021 07:20:53 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:20:53 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:20:53 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:20:54 INFO Checking backend latencies : 03/02/2021 07:20:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:20:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:20:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:20:56 INFO Checking backend latencies : 03/02/2021 07:20:56 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:20:56 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:20:56 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:20:59 INFO Checking backend latencies : 03/02/2021 07:20:59 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:20:59 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:20:59 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:20:59 INFO Checking backend latencies : 03/02/2021 07:20:59 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:20:59 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:20:59 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:21:19 INFO Checking backend latencies : 03/02/2021 07:21:19 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:21:19 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:21:19 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:21:19 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:21:22 INFO Checking backend latencies : 03/02/2021 07:21:22 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:21:22 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:21:22 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:21:23 INFO Checking backend latencies : 03/02/2021 07:21:23 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:21:23 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:21:23 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:21:25 INFO Checking backend latencies : 03/02/2021 07:21:25 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:21:25 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:21:25 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:21:29 INFO Checking backend latencies : 03/02/2021 07:21:29 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:21:29 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:21:29 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:21:32 INFO Checking backend latencies : 03/02/2021 07:21:32 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:21:32 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:21:32 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:21:32 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader 03/02/2021 07:21:32 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 516, in get_key index, data = cons.get(key) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:21:35 INFO Checking backend latencies : 03/02/2021 07:21:35 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:21:35 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:21:35 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:21:52 INFO Checking backend latencies : 03/02/2021 07:21:52 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:21:52 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:21:52 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:21:55 INFO Checking backend latencies : 03/02/2021 07:21:55 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:21:55 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:21:55 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:21:59 INFO Checking backend latencies : 03/02/2021 07:21:59 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:21:59 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:21:59 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:22:01 INFO Checking backend latencies : 03/02/2021 07:22:01 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:22:01 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:22:01 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:22:01 ERROR LeaderElectionBase exception 03/02/2021 07:22:01 ERROR 03/02/2021 07:22:08 INFO Checking backend latencies : 03/02/2021 07:22:08 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:22:08 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:22:08 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:22:08 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:22:25 INFO Checking backend latencies : 03/02/2021 07:22:25 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:22:25 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:22:25 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:22:30 INFO Checking backend latencies : 03/02/2021 07:22:30 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:22:30 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:22:30 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:22:30 ERROR Error on consul connection. 03/02/2021 07:22:30 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/iscsi_service.py", line 158, in start self.__current_lock_index = consul_api.current_index() File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 348, in current_index raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 345, in current_index index = consul_obj.get(self.root)[0] File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:22:32 INFO Checking backend latencies : 03/02/2021 07:22:32 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:22:32 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:22:32 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:22:33 INFO Checking backend latencies : 03/02/2021 07:22:33 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:22:33 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:22:33 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:22:33 INFO Checking backend latencies : 03/02/2021 07:22:33 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:22:33 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:22:33 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:22:33 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:22:42 WARNING There are too many exceptions.Service will clean this node. 03/02/2021 07:22:42 INFO Cleaning unused configurations. 03/02/2021 07:22:42 INFO Cleaning all mapped disks 03/02/2021 07:22:50 INFO Checking backend latencies : 03/02/2021 07:22:50 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:22:50 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:22:50 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:22:54 INFO Checking backend latencies : 03/02/2021 07:22:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:22:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:22:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:22:56 INFO Checking backend latencies : 03/02/2021 07:22:56 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:22:56 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:22:56 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:22:56 INFO Checking backend latencies : 03/02/2021 07:22:56 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:22:56 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:22:56 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:22:57 INFO Checking backend latencies : 03/02/2021 07:22:57 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:22:57 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:22:57 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:23:05 INFO Checking backend latencies : 03/02/2021 07:23:05 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:23:05 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:23:05 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:23:14 INFO Checking backend latencies : 03/02/2021 07:23:14 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:23:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:23:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:23:18 INFO Checking backend latencies : 03/02/2021 07:23:18 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:23:18 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:23:18 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:23:22 INFO Checking backend latencies : 03/02/2021 07:23:22 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:23:22 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:23:22 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:23:23 INFO Checking backend latencies : 03/02/2021 07:23:23 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:23:23 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:23:23 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:23:29 INFO Checking backend latencies : 03/02/2021 07:23:29 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:23:29 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:23:29 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:23:34 INFO Checking backend latencies : 03/02/2021 07:23:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:23:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:23:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:23:34 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:23:40 INFO Checking backend latencies : 03/02/2021 07:23:40 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:23:40 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:23:40 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:23:44 INFO Checking backend latencies : 03/02/2021 07:23:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:23:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:23:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:23:52 INFO Checking backend latencies : 03/02/2021 07:23:52 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:23:52 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:23:52 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:23:53 INFO Checking backend latencies : 03/02/2021 07:23:53 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:23:53 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:23:53 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:23:55 INFO Checking backend latencies : 03/02/2021 07:23:55 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:23:55 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:23:55 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:24:05 INFO Checking backend latencies : 03/02/2021 07:24:05 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:24:05 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:24:05 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:24:10 INFO Checking backend latencies : 03/02/2021 07:24:10 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:24:10 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:24:10 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:24:14 INFO Checking backend latencies : 03/02/2021 07:24:14 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:24:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:24:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:24:25 INFO Checking backend latencies : 03/02/2021 07:24:25 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:24:25 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:24:25 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:24:29 INFO Checking backend latencies : 03/02/2021 07:24:29 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:24:29 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:24:29 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:24:30 INFO Checking backend latencies : 03/02/2021 07:24:30 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:24:30 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:24:30 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:24:30 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:24:32 INFO Checking backend latencies : 03/02/2021 07:24:32 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:24:32 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:24:32 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:24:32 ERROR Could not find key resource. 03/02/2021 07:24:32 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 496, in get_key_blocking index, data = consul_obj.get(key, lock_index, False, timeout) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:24:49 INFO Checking backend latencies : 03/02/2021 07:24:49 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:24:49 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:24:49 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:24:49 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:24:53 INFO Checking backend latencies : 03/02/2021 07:24:53 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:24:53 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:24:53 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:24:53 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader 03/02/2021 07:24:53 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 516, in get_key index, data = cons.get(key) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:24:54 INFO Checking backend latencies : 03/02/2021 07:24:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:24:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:24:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:24:54 INFO Checking backend latencies : 03/02/2021 07:24:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:24:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:24:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:24:55 INFO Checking backend latencies : 03/02/2021 07:24:55 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:24:55 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:24:55 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:25:03 INFO Checking backend latencies : 03/02/2021 07:25:03 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:25:03 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:25:03 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:25:03 ERROR Error reading application config 03/02/2021 07:25:07 INFO checking uptime 03/02/2021 07:25:07 INFO checking uptime of node: peta1 ip:10.10.0.151 03/02/2021 07:25:07 INFO checking uptime of node: peta2 ip:10.10.0.152 03/02/2021 07:25:18 INFO Checking backend latencies : 03/02/2021 07:25:18 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:25:18 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:25:18 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:25:18 INFO Checking backend latencies : 03/02/2021 07:25:18 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:25:18 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:25:18 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:25:20 INFO Checking backend latencies : 03/02/2021 07:25:20 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:25:20 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:25:20 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:25:25 ERROR checking uptime of node: peta2 ping error to ip: 10.10.0.152 03/02/2021 07:25:25 INFO checking uptime of node: peta3 ip:10.10.0.153 03/02/2021 07:25:25 INFO Service is starting. 03/02/2021 07:25:25 INFO iSCSI Service dropping old sessions 03/02/2021 07:25:25 INFO Checking backend latencies : 03/02/2021 07:25:25 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:25:25 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:25:25 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:25:44 INFO Checking backend latencies : 03/02/2021 07:25:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:25:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:25:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:25:44 INFO Checking backend latencies : 03/02/2021 07:25:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:25:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:25:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:25:45 INFO Checking backend latencies : 03/02/2021 07:25:45 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:25:45 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:25:45 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:26:02 INFO Checking backend latencies : 03/02/2021 07:26:02 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:26:02 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:26:02 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:26:03 INFO Checking backend latencies : 03/02/2021 07:26:03 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:26:03 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:26:03 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:26:03 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:26:10 INFO Checking backend latencies : 03/02/2021 07:26:10 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:26:10 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:26:10 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:26:14 INFO Checking backend latencies : 03/02/2021 07:26:14 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:26:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:26:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:26:15 INFO Checking backend latencies : 03/02/2021 07:26:15 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:26:15 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:26:15 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:26:15 INFO Checking backend latencies : 03/02/2021 07:26:15 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:26:15 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:26:15 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:26:26 INFO Checking backend latencies : 03/02/2021 07:26:26 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:26:26 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:26:26 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:26:35 INFO Checking backend latencies : 03/02/2021 07:26:35 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:26:35 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:26:35 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:26:39 INFO Checking backend latencies : 03/02/2021 07:26:39 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:26:39 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:26:39 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:26:41 INFO Checking backend latencies : 03/02/2021 07:26:41 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:26:41 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:26:41 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:26:52 INFO Checking backend latencies : 03/02/2021 07:26:52 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:26:52 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:26:52 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:26:52 INFO Checking backend latencies : 03/02/2021 07:26:52 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:26:52 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:26:52 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:26:52 ERROR LeaderElectionBase exception 03/02/2021 07:26:52 ERROR 03/02/2021 07:26:53 INFO Checking backend latencies : 03/02/2021 07:26:53 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:26:53 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:26:53 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:26:53 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:26:59 INFO Checking backend latencies : 03/02/2021 07:26:59 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:26:59 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:26:59 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:27:05 INFO Checking backend latencies : 03/02/2021 07:27:05 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:27:05 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:27:05 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:27:17 INFO Checking backend latencies : 03/02/2021 07:27:17 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:27:17 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:27:17 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:27:19 INFO Checking backend latencies : 03/02/2021 07:27:19 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:27:19 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:27:19 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:27:19 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:27:23 INFO Checking backend latencies : 03/02/2021 07:27:23 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:27:23 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:27:23 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:27:25 INFO Checking backend latencies : 03/02/2021 07:27:25 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:27:25 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:27:25 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:27:25 INFO Checking backend latencies : 03/02/2021 07:27:25 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:27:25 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:27:25 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:27:35 INFO Checking backend latencies : 03/02/2021 07:27:35 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:27:35 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:27:35 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:27:41 INFO Checking backend latencies : 03/02/2021 07:27:41 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:27:41 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:27:41 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:27:49 INFO Checking backend latencies : 03/02/2021 07:27:49 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:27:49 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:27:49 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:27:50 INFO Checking backend latencies : 03/02/2021 07:27:50 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:27:50 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:27:50 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:27:56 INFO Checking backend latencies : 03/02/2021 07:27:56 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:27:56 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:27:56 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:28:01 INFO Checking backend latencies : 03/02/2021 07:28:01 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:28:01 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:28:01 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:28:01 ERROR iSCSI Service exception dropping old sessions 03/02/2021 07:28:07 INFO Checking backend latencies : 03/02/2021 07:28:07 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:28:07 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:28:07 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:28:13 INFO Checking backend latencies : 03/02/2021 07:28:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:28:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:28:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:28:13 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader 03/02/2021 07:28:13 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 516, in get_key index, data = cons.get(key) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:28:14 INFO Checking backend latencies : 03/02/2021 07:28:14 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:28:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:28:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:28:15 INFO Checking backend latencies : 03/02/2021 07:28:15 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:28:15 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:28:15 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:28:34 INFO Checking backend latencies : 03/02/2021 07:28:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:28:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:28:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:28:34 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:28:38 INFO Checking backend latencies : 03/02/2021 07:28:38 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:28:38 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:28:38 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:28:38 INFO Checking backend latencies : 03/02/2021 07:28:38 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:28:38 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:28:38 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:28:40 INFO Checking backend latencies : 03/02/2021 07:28:40 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:28:40 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:28:40 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:28:45 INFO Checking backend latencies : 03/02/2021 07:28:45 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:28:45 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:28:45 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:29:02 INFO Checking backend latencies : 03/02/2021 07:29:02 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:29:02 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:29:02 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:29:05 INFO Checking backend latencies : 03/02/2021 07:29:05 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:29:05 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:29:05 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:29:11 INFO Checking backend latencies : 03/02/2021 07:29:11 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:29:11 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:29:11 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:29:16 INFO Checking backend latencies : 03/02/2021 07:29:16 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:29:16 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:29:16 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:29:16 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:29:23 INFO Checking backend latencies : 03/02/2021 07:29:23 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:29:23 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:29:23 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:29:23 ERROR Could not find key resource. 03/02/2021 07:29:23 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 496, in get_key_blocking index, data = consul_obj.get(key, lock_index, False, timeout) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:29:29 INFO Checking backend latencies : 03/02/2021 07:29:29 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:29:29 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:29:29 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:29:29 INFO Checking backend latencies : 03/02/2021 07:29:29 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:29:29 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:29:29 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:29:36 INFO Checking backend latencies : 03/02/2021 07:29:36 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:29:36 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:29:36 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:29:40 INFO Checking backend latencies : 03/02/2021 07:29:40 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:29:40 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:29:40 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:29:45 INFO Checking backend latencies : 03/02/2021 07:29:45 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:29:45 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:29:45 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:29:49 INFO Checking backend latencies : 03/02/2021 07:29:49 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:29:49 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:29:49 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:29:49 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:29:56 INFO Checking backend latencies : 03/02/2021 07:29:56 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:29:56 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:29:56 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:29:59 INFO Checking backend latencies : 03/02/2021 07:29:59 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:29:59 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:29:59 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:30:00 INFO Checking backend latencies : 03/02/2021 07:30:00 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:30:00 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:30:00 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:30:04 INFO Checking backend latencies : 03/02/2021 07:30:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:30:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:30:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:30:09 INFO Checking backend latencies : 03/02/2021 07:30:09 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:30:09 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:30:09 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:30:20 INFO Checking backend latencies : 03/02/2021 07:30:20 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:30:20 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:30:20 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:30:26 INFO Checking backend latencies : 03/02/2021 07:30:26 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:30:26 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:30:26 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:30:26 INFO Checking backend latencies : 03/02/2021 07:30:26 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:30:26 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:30:26 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:30:31 INFO Checking backend latencies : 03/02/2021 07:30:31 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:30:31 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:30:31 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:30:36 INFO Checking backend latencies : 03/02/2021 07:30:36 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:30:36 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:30:36 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:30:38 INFO Checking backend latencies : 03/02/2021 07:30:38 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:30:38 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:30:38 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:30:38 ERROR iSCSI Service exception dropping old sessions 03/02/2021 07:30:44 INFO Checking backend latencies : 03/02/2021 07:30:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:30:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:30:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:30:56 INFO Checking backend latencies : 03/02/2021 07:30:56 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:30:56 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:30:56 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:31:01 INFO Checking backend latencies : 03/02/2021 07:31:01 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:31:01 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:31:01 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:31:04 INFO Checking backend latencies : 03/02/2021 07:31:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:31:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:31:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:31:04 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:31:06 INFO Checking backend latencies : 03/02/2021 07:31:06 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:31:06 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:31:06 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:31:10 INFO Checking backend latencies : 03/02/2021 07:31:10 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:31:10 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:31:10 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:31:15 INFO Checking backend latencies : 03/02/2021 07:31:15 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:31:15 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:31:15 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:31:35 INFO Checking backend latencies : 03/02/2021 07:31:35 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:31:35 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:31:35 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:31:35 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader 03/02/2021 07:31:35 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 516, in get_key index, data = cons.get(key) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:31:35 INFO Checking backend latencies : 03/02/2021 07:31:35 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:31:35 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:31:35 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:31:39 INFO Checking backend latencies : 03/02/2021 07:31:39 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:31:39 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:31:39 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:31:39 INFO Checking backend latencies : 03/02/2021 07:31:39 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:31:39 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:31:39 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:31:39 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:31:41 INFO Checking backend latencies : 03/02/2021 07:31:41 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:31:41 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:31:41 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:31:44 INFO Checking backend latencies : 03/02/2021 07:31:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:31:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:31:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:31:44 ERROR LeaderElectionBase exception 03/02/2021 07:31:44 ERROR 03/02/2021 07:31:59 INFO Checking backend latencies : 03/02/2021 07:31:59 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:31:59 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:31:59 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:32:03 INFO Checking backend latencies : 03/02/2021 07:32:03 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:32:03 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:32:03 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:32:05 INFO Checking backend latencies : 03/02/2021 07:32:05 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:32:05 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:32:05 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:32:16 INFO Checking backend latencies : 03/02/2021 07:32:16 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:32:16 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:32:16 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:32:19 INFO Checking backend latencies : 03/02/2021 07:32:19 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:32:19 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:32:19 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:32:19 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:32:25 INFO Checking backend latencies : 03/02/2021 07:32:25 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:32:25 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:32:25 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:32:28 INFO Checking backend latencies : 03/02/2021 07:32:28 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:32:28 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:32:28 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:32:35 INFO Checking backend latencies : 03/02/2021 07:32:35 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:32:35 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:32:35 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:32:40 INFO Checking backend latencies : 03/02/2021 07:32:40 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:32:40 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:32:40 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:32:50 INFO Checking backend latencies : 03/02/2021 07:32:50 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:32:50 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:32:50 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:32:54 INFO Checking backend latencies : 03/02/2021 07:32:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:32:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:32:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:32:56 INFO Checking backend latencies : 03/02/2021 07:32:56 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:32:56 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:32:56 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:32:57 INFO Checking backend latencies : 03/02/2021 07:32:57 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:32:57 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:32:57 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:33:07 INFO Checking backend latencies : 03/02/2021 07:33:07 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:33:07 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:33:07 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:33:13 INFO Checking backend latencies : 03/02/2021 07:33:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:33:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:33:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:33:13 ERROR iSCSI Service exception dropping old sessions 03/02/2021 07:33:14 INFO Checking backend latencies : 03/02/2021 07:33:14 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:33:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:33:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:33:21 INFO Checking backend latencies : 03/02/2021 07:33:21 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:33:21 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:33:21 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:33:24 INFO Checking backend latencies : 03/02/2021 07:33:24 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:33:24 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:33:24 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:33:34 INFO Checking backend latencies : 03/02/2021 07:33:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:33:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:33:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:33:34 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:33:37 INFO Checking backend latencies : 03/02/2021 07:33:37 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:33:37 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:33:37 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:33:41 INFO Checking backend latencies : 03/02/2021 07:33:41 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:33:41 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:33:41 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:33:47 INFO Checking backend latencies : 03/02/2021 07:33:47 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:33:47 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:33:47 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:33:50 INFO Checking backend latencies : 03/02/2021 07:33:50 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:33:50 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:33:50 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:34:02 INFO Checking backend latencies : 03/02/2021 07:34:02 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:34:02 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:34:02 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:34:02 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:34:05 INFO Checking backend latencies : 03/02/2021 07:34:05 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:34:05 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:34:05 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:34:11 INFO Checking backend latencies : 03/02/2021 07:34:11 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:34:11 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:34:11 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:34:15 INFO Checking backend latencies : 03/02/2021 07:34:15 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:34:15 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:34:15 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:34:15 INFO Checking backend latencies : 03/02/2021 07:34:15 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:34:15 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:34:15 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:34:15 ERROR Could not find key resource. 03/02/2021 07:34:15 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 496, in get_key_blocking index, data = consul_obj.get(key, lock_index, False, timeout) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:34:17 INFO Checking backend latencies : 03/02/2021 07:34:17 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:34:17 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:34:17 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:34:26 INFO Checking backend latencies : 03/02/2021 07:34:26 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:34:26 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:34:26 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:34:29 INFO Checking backend latencies : 03/02/2021 07:34:29 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:34:29 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:34:29 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:34:37 INFO Checking backend latencies : 03/02/2021 07:34:37 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:34:37 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:34:37 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:34:41 INFO Checking backend latencies : 03/02/2021 07:34:41 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:34:41 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:34:41 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:34:49 INFO Checking backend latencies : 03/02/2021 07:34:49 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:34:49 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:34:49 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:34:49 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:34:50 INFO Checking backend latencies : 03/02/2021 07:34:50 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:34:50 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:34:50 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:34:56 INFO Checking backend latencies : 03/02/2021 07:34:56 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:34:56 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:34:56 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:34:56 INFO Checking backend latencies : 03/02/2021 07:34:56 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:34:56 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:34:56 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:34:56 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader 03/02/2021 07:34:56 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 516, in get_key index, data = cons.get(key) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:35:01 INFO Checking backend latencies : 03/02/2021 07:35:01 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:35:01 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:35:01 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:35:11 INFO Checking backend latencies : 03/02/2021 07:35:11 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:35:11 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:35:11 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:35:16 INFO Checking backend latencies : 03/02/2021 07:35:16 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:35:16 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:35:16 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:35:20 INFO Checking backend latencies : 03/02/2021 07:35:20 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:35:20 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:35:20 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:35:26 INFO Checking backend latencies : 03/02/2021 07:35:26 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:35:26 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:35:26 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:35:27 INFO Checking backend latencies : 03/02/2021 07:35:27 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:35:27 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:35:27 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:35:44 INFO Checking backend latencies : 03/02/2021 07:35:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:35:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:35:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:35:46 INFO Checking backend latencies : 03/02/2021 07:35:46 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:35:46 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:35:46 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:35:49 INFO Checking backend latencies : 03/02/2021 07:35:49 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:35:49 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:35:49 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:35:49 ERROR iSCSI Service exception dropping old sessions 03/02/2021 07:35:57 INFO Checking backend latencies : 03/02/2021 07:35:57 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:35:57 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:35:57 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:36:04 INFO Checking backend latencies : 03/02/2021 07:36:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:36:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:36:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:36:04 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:36:11 INFO Checking backend latencies : 03/02/2021 07:36:11 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:36:11 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:36:11 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:36:18 INFO Checking backend latencies : 03/02/2021 07:36:18 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:36:18 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:36:18 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:36:25 INFO Checking backend latencies : 03/02/2021 07:36:25 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:36:25 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:36:25 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:36:25 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:36:26 INFO Checking backend latencies : 03/02/2021 07:36:26 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:36:26 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:36:26 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:36:35 INFO Checking backend latencies : 03/02/2021 07:36:35 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:36:35 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:36:35 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:36:36 INFO Checking backend latencies : 03/02/2021 07:36:36 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:36:36 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:36:36 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:36:36 ERROR LeaderElectionBase exception 03/02/2021 07:36:36 ERROR 03/02/2021 07:36:41 INFO Checking backend latencies : 03/02/2021 07:36:41 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:36:41 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:36:41 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:36:42 INFO Checking backend latencies : 03/02/2021 07:36:42 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:36:42 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:36:42 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:36:49 INFO Checking backend latencies : 03/02/2021 07:36:49 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:36:49 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:36:49 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:36:51 INFO Checking backend latencies : 03/02/2021 07:36:51 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:36:51 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:36:51 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:36:59 INFO Checking backend latencies : 03/02/2021 07:36:59 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:36:59 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:36:59 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:37:08 INFO Checking backend latencies : 03/02/2021 07:37:08 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:37:08 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:37:08 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:37:08 INFO Checking backend latencies : 03/02/2021 07:37:08 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:37:08 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:37:08 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:37:13 INFO Checking backend latencies : 03/02/2021 07:37:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:37:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:37:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:38:14 INFO Start settings IPs 03/02/2021 07:38:27 INFO GlusterFS mount attempt 03/02/2021 07:38:27 INFO str_start_command: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consul agent -raft-protocol 2 -config-dir /opt/petasan/config/etc/consul.d/server -bind 172.16.0.151 -retry-join 172.16.0.152 -retry-join 172.16.0.153 03/02/2021 07:38:31 INFO Starting cluster file sync service 03/02/2021 07:38:31 INFO Starting iSCSI Service 03/02/2021 07:38:31 INFO Starting Cluster Management application 03/02/2021 07:38:31 INFO Starting Node Stats Service 03/02/2021 07:38:31 INFO LeaderElectionBase dropping old sessions 03/02/2021 07:38:31 INFO Starting activating PetaSAN lvs 03/02/2021 07:38:31 INFO Running command: /sbin/vgs --noheadings --readonly --units=b --nosuffix --separator=";" -o vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_count,vg_extent_size 03/02/2021 07:38:31 INFO stdout ceph-847b13a2-67ce-4374-b26b-f42d81ceb154";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 07:38:31 INFO stdout ceph-d0153ed6-0b75-470c-b1c7-906759dee371";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 07:38:31 INFO stdout ceph-e72ec456-4943-4572-8b6f-f20fa4b85417";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 07:38:31 INFO stdout ceph-fe73888d-9945-41db-8d0d-5bb67360981c";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 07:38:31 INFO Starting activating PetaSAN lvs 03/02/2021 07:38:31 INFO checking uptime 03/02/2021 07:38:31 INFO checking uptime of node: peta1 ip:10.10.0.151 03/02/2021 07:38:31 INFO Running command: /sbin/vgs --noheadings --readonly --units=b --nosuffix --separator=";" -o vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_count,vg_extent_size 03/02/2021 07:38:31 INFO stdout ceph-847b13a2-67ce-4374-b26b-f42d81ceb154";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 07:38:31 INFO stdout ceph-d0153ed6-0b75-470c-b1c7-906759dee371";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 07:38:31 INFO stdout ceph-e72ec456-4943-4572-8b6f-f20fa4b85417";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 07:38:31 INFO stdout ceph-fe73888d-9945-41db-8d0d-5bb67360981c";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 07:38:32 INFO checking uptime of node: peta1 is starting 03/02/2021 07:38:32 INFO checking uptime of node: peta2 ip:10.10.0.152 03/02/2021 07:38:33 INFO Starting OSDs 03/02/2021 07:38:33 INFO Starting petasan tuning service 03/02/2021 07:38:33 INFO Starting Config Upload service 03/02/2021 07:38:33 INFO Starting petasan-qperf server 03/02/2021 07:38:50 INFO Checking backend latencies : 03/02/2021 07:38:50 INFO Network latency for backend 172.16.0.151 = 6.07 us 03/02/2021 07:38:50 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:38:50 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:38:52 ERROR checking uptime of node: peta2 ping error to ip: 10.10.0.152 03/02/2021 07:38:52 INFO checking uptime of node: peta3 ip:10.10.0.153 03/02/2021 07:38:52 INFO Service is starting. 03/02/2021 07:38:52 INFO iSCSI Service dropping old sessions 03/02/2021 07:38:52 INFO Checking backend latencies : 03/02/2021 07:38:52 INFO Network latency for backend 172.16.0.151 = 7.22 us 03/02/2021 07:38:52 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:38:52 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:39:00 INFO Checking backend latencies : 03/02/2021 07:39:00 INFO Network latency for backend 172.16.0.151 = 7.22 us 03/02/2021 07:39:00 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:39:00 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:39:02 INFO Checking backend latencies : 03/02/2021 07:39:02 INFO Network latency for backend 172.16.0.151 = 7.28 us 03/02/2021 07:39:02 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:39:02 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:39:04 INFO GlusterFS mount attempt 03/02/2021 07:39:11 INFO Checking backend latencies : 03/02/2021 07:39:11 INFO Network latency for backend 172.16.0.151 = 7.19 us 03/02/2021 07:39:11 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:39:11 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:39:13 INFO Checking backend latencies : 03/02/2021 07:39:13 INFO Network latency for backend 172.16.0.151 = 7.01 us 03/02/2021 07:39:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:39:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:39:21 INFO Checking backend latencies : 03/02/2021 07:39:21 INFO Network latency for backend 172.16.0.151 = 7.13 us 03/02/2021 07:39:21 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:39:21 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:39:23 INFO Checking backend latencies : 03/02/2021 07:39:23 INFO Network latency for backend 172.16.0.151 = 7.04 us 03/02/2021 07:39:23 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:39:23 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:39:26 INFO Checking backend latencies : 03/02/2021 07:39:26 INFO Network latency for backend 172.16.0.151 = 7.15 us 03/02/2021 07:39:26 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:39:26 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:39:34 INFO Checking backend latencies : 03/02/2021 07:39:34 INFO Network latency for backend 172.16.0.151 = 7.38 us 03/02/2021 07:39:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:39:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:39:35 INFO GlusterFS mount attempt 03/02/2021 07:39:37 INFO Checking backend latencies : 03/02/2021 07:39:37 INFO Network latency for backend 172.16.0.151 = 7.07 us 03/02/2021 07:39:37 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:39:37 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:39:44 INFO Checking backend latencies : 03/02/2021 07:39:44 INFO Network latency for backend 172.16.0.151 = 7.24 us 03/02/2021 07:39:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:39:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:39:46 INFO Checking backend latencies : 03/02/2021 07:39:46 INFO Network latency for backend 172.16.0.151 = 7.35 us 03/02/2021 07:39:46 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:39:46 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:39:48 INFO Checking backend latencies : 03/02/2021 07:39:48 INFO Network latency for backend 172.16.0.151 = 7.06 us 03/02/2021 07:39:48 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:39:48 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:40:02 INFO Checking backend latencies : 03/02/2021 07:40:02 INFO Network latency for backend 172.16.0.151 = 7.15 us 03/02/2021 07:40:02 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:40:02 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:40:04 INFO Checking backend latencies : 03/02/2021 07:40:04 INFO Network latency for backend 172.16.0.151 = 6.99 us 03/02/2021 07:40:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:40:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:40:06 INFO GlusterFS mount attempt 03/02/2021 07:40:11 INFO Checking backend latencies : 03/02/2021 07:40:11 INFO Network latency for backend 172.16.0.151 = 6.99 us 03/02/2021 07:40:11 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:40:11 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:40:13 INFO Checking backend latencies : 03/02/2021 07:40:13 INFO Network latency for backend 172.16.0.151 = 7.31 us 03/02/2021 07:40:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:40:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:40:15 INFO Checking backend latencies : 03/02/2021 07:40:15 INFO Network latency for backend 172.16.0.151 = 6.53 us 03/02/2021 07:40:15 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:40:15 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:40:17 INFO Checking backend latencies : 03/02/2021 07:40:17 INFO Network latency for backend 172.16.0.151 = 7.15 us 03/02/2021 07:40:17 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:40:17 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:40:36 INFO GlusterFS mount attempt 03/02/2021 07:40:37 INFO Checking backend latencies : 03/02/2021 07:40:37 INFO Network latency for backend 172.16.0.151 = 7.25 us 03/02/2021 07:40:37 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:40:37 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:40:37 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:40:39 INFO Checking backend latencies : 03/02/2021 07:40:39 INFO Network latency for backend 172.16.0.151 = 7.11 us 03/02/2021 07:40:39 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:40:39 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:40:39 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader 03/02/2021 07:40:39 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 516, in get_key index, data = cons.get(key) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:40:41 INFO Checking backend latencies : 03/02/2021 07:40:41 INFO Network latency for backend 172.16.0.151 = 6.18 us 03/02/2021 07:40:41 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:40:41 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:40:43 INFO Checking backend latencies : 03/02/2021 07:40:43 INFO Network latency for backend 172.16.0.151 = 7.27 us 03/02/2021 07:40:43 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:40:43 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:40:47 INFO Checking backend latencies : 03/02/2021 07:40:47 INFO Network latency for backend 172.16.0.151 = 8.27 us 03/02/2021 07:40:47 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:40:47 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:40:47 ERROR LeaderElectionBase exception dropping old sessions 03/02/2021 07:40:50 INFO Checking backend latencies : 03/02/2021 07:40:50 INFO Network latency for backend 172.16.0.151 = 6.98 us 03/02/2021 07:40:50 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:40:50 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:40:50 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:40:58 INFO Checking backend latencies : 03/02/2021 07:40:58 INFO Network latency for backend 172.16.0.151 = 6.83 us 03/02/2021 07:40:58 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:40:58 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:41:04 INFO Checking backend latencies : 03/02/2021 07:41:04 INFO Network latency for backend 172.16.0.151 = 6.86 us 03/02/2021 07:41:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:41:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:41:07 INFO GlusterFS mount attempt 03/02/2021 07:41:16 INFO Checking backend latencies : 03/02/2021 07:41:16 INFO Network latency for backend 172.16.0.151 = 7.05 us 03/02/2021 07:41:16 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:41:16 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:41:18 INFO Checking backend latencies : 03/02/2021 07:41:18 INFO Network latency for backend 172.16.0.151 = 7.15 us 03/02/2021 07:41:18 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:41:18 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:41:18 ERROR iSCSI Service exception dropping old sessions 03/02/2021 07:41:20 INFO Checking backend latencies : 03/02/2021 07:41:20 INFO Network latency for backend 172.16.0.151 = 6.67 us 03/02/2021 07:41:20 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:41:20 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:41:31 INFO Checking backend latencies : 03/02/2021 07:41:31 INFO Network latency for backend 172.16.0.151 = 7.03 us 03/02/2021 07:41:31 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:41:31 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:41:33 INFO Checking backend latencies : 03/02/2021 07:41:33 INFO Network latency for backend 172.16.0.151 = 6.69 us 03/02/2021 07:41:33 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:41:33 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:41:37 INFO Checking backend latencies : 03/02/2021 07:41:37 INFO Network latency for backend 172.16.0.151 = 6.67 us 03/02/2021 07:41:37 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:41:37 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:41:38 INFO GlusterFS mount attempt 03/02/2021 07:41:43 INFO Checking backend latencies : 03/02/2021 07:41:43 INFO Network latency for backend 172.16.0.151 = 6.84 us 03/02/2021 07:41:43 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:41:43 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:41:52 INFO Checking backend latencies : 03/02/2021 07:41:52 INFO Network latency for backend 172.16.0.151 = 6.88 us 03/02/2021 07:41:52 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:41:52 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:41:54 INFO Checking backend latencies : 03/02/2021 07:41:54 INFO Network latency for backend 172.16.0.151 = 7.28 us 03/02/2021 07:41:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:41:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:41:58 INFO Checking backend latencies : 03/02/2021 07:41:58 INFO Network latency for backend 172.16.0.151 = 7.01 us 03/02/2021 07:41:58 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:41:58 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:42:00 INFO Checking backend latencies : 03/02/2021 07:42:00 INFO Network latency for backend 172.16.0.151 = 6.71 us 03/02/2021 07:42:00 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:42:00 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:42:06 INFO Checking backend latencies : 03/02/2021 07:42:06 INFO Network latency for backend 172.16.0.151 = 6.53 us 03/02/2021 07:42:06 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:42:06 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:42:06 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:42:10 INFO Checking backend latencies : 03/02/2021 07:42:10 INFO Network latency for backend 172.16.0.151 = 6.95 us 03/02/2021 07:42:10 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:42:10 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:42:13 INFO Checking backend latencies : 03/02/2021 07:42:13 INFO Network latency for backend 172.16.0.151 = 6.98 us 03/02/2021 07:42:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:42:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:42:17 INFO GlusterFS mount attempt 03/02/2021 07:42:18 INFO Checking backend latencies : 03/02/2021 07:42:18 INFO Network latency for backend 172.16.0.151 = 7.03 us 03/02/2021 07:42:18 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:42:18 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:42:20 INFO Checking backend latencies : 03/02/2021 07:42:20 INFO Network latency for backend 172.16.0.151 = 7.06 us 03/02/2021 07:42:20 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:42:20 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:42:27 INFO Checking backend latencies : 03/02/2021 07:42:27 INFO Network latency for backend 172.16.0.151 = 6.59 us 03/02/2021 07:42:27 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:42:27 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:42:36 INFO Checking backend latencies : 03/02/2021 07:42:36 INFO Network latency for backend 172.16.0.151 = 6.67 us 03/02/2021 07:42:36 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:42:36 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:42:43 INFO Checking backend latencies : 03/02/2021 07:42:43 INFO Network latency for backend 172.16.0.151 = 7.08 us 03/02/2021 07:42:43 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:42:43 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:42:45 INFO Checking backend latencies : 03/02/2021 07:42:45 INFO Network latency for backend 172.16.0.151 = 7.05 us 03/02/2021 07:42:45 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:42:45 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:42:47 INFO Checking backend latencies : 03/02/2021 07:42:47 INFO Network latency for backend 172.16.0.151 = 6.49 us 03/02/2021 07:42:47 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:42:47 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:42:47 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:42:49 INFO Checking backend latencies : 03/02/2021 07:42:49 INFO Network latency for backend 172.16.0.151 = 7.2 us 03/02/2021 07:42:49 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:42:49 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:42:49 INFO GlusterFS mount attempt 03/02/2021 07:43:02 INFO Checking backend latencies : 03/02/2021 07:43:02 INFO Network latency for backend 172.16.0.151 = 7.33 us 03/02/2021 07:43:02 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:43:02 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:43:02 ERROR LeaderElectionBase exception dropping old sessions 03/02/2021 07:43:04 INFO Checking backend latencies : 03/02/2021 07:43:04 INFO Network latency for backend 172.16.0.151 = 6.58 us 03/02/2021 07:43:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:43:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:43:08 INFO Checking backend latencies : 03/02/2021 07:43:08 INFO Network latency for backend 172.16.0.151 = 7.04 us 03/02/2021 07:43:08 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:43:08 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:43:10 INFO Checking backend latencies : 03/02/2021 07:43:10 INFO Network latency for backend 172.16.0.151 = 6.74 us 03/02/2021 07:43:10 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:43:10 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:43:12 INFO Checking backend latencies : 03/02/2021 07:43:12 INFO Network latency for backend 172.16.0.151 = 6.78 us 03/02/2021 07:43:12 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:43:12 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:43:19 INFO GlusterFS mount attempt 03/02/2021 07:43:20 INFO Checking backend latencies : 03/02/2021 07:43:20 INFO Network latency for backend 172.16.0.151 = 6.98 us 03/02/2021 07:43:20 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:43:20 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:43:20 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:43:29 INFO Checking backend latencies : 03/02/2021 07:43:29 INFO Network latency for backend 172.16.0.151 = 6.86 us 03/02/2021 07:43:29 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:43:29 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:43:32 INFO Checking backend latencies : 03/02/2021 07:43:32 INFO Network latency for backend 172.16.0.151 = 7.24 us 03/02/2021 07:43:32 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:43:32 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:43:35 INFO Checking backend latencies : 03/02/2021 07:43:35 INFO Network latency for backend 172.16.0.151 = 6.84 us 03/02/2021 07:43:35 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:43:35 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:43:40 INFO Checking backend latencies : 03/02/2021 07:43:40 INFO Network latency for backend 172.16.0.151 = 6.91 us 03/02/2021 07:43:40 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:43:40 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:43:40 ERROR iSCSI Service exception dropping old sessions 03/02/2021 07:43:45 INFO Checking backend latencies : 03/02/2021 07:43:45 INFO Network latency for backend 172.16.0.151 = 6.89 us 03/02/2021 07:43:45 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:43:45 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:43:45 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader 03/02/2021 07:43:45 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 516, in get_key index, data = cons.get(key) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:43:50 INFO GlusterFS mount attempt 03/02/2021 07:43:52 INFO Checking backend latencies : 03/02/2021 07:43:52 INFO Network latency for backend 172.16.0.151 = 6.77 us 03/02/2021 07:43:52 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:43:52 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:43:54 INFO Checking backend latencies : 03/02/2021 07:43:54 INFO Network latency for backend 172.16.0.151 = 6.43 us 03/02/2021 07:43:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:43:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:44:02 INFO Checking backend latencies : 03/02/2021 07:44:02 INFO Network latency for backend 172.16.0.151 = 7.1 us 03/02/2021 07:44:02 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:44:02 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:44:04 INFO Checking backend latencies : 03/02/2021 07:44:04 INFO Network latency for backend 172.16.0.151 = 6.26 us 03/02/2021 07:44:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:44:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:44:14 INFO Checking backend latencies : 03/02/2021 07:44:14 INFO Network latency for backend 172.16.0.151 = 6.66 us 03/02/2021 07:44:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:44:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:44:17 INFO Checking backend latencies : 03/02/2021 07:44:17 INFO Network latency for backend 172.16.0.151 = 6.9 us 03/02/2021 07:44:17 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:44:17 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:44:19 INFO Checking backend latencies : 03/02/2021 07:44:19 INFO Network latency for backend 172.16.0.151 = 6.41 us 03/02/2021 07:44:19 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:44:19 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:44:23 INFO GlusterFS mount attempt 03/02/2021 07:44:25 INFO Checking backend latencies : 03/02/2021 07:44:25 INFO Network latency for backend 172.16.0.151 = 6.84 us 03/02/2021 07:44:25 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:44:25 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:44:35 INFO Checking backend latencies : 03/02/2021 07:44:35 INFO Network latency for backend 172.16.0.151 = 7.08 us 03/02/2021 07:44:35 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:44:35 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:44:37 INFO Checking backend latencies : 03/02/2021 07:44:37 INFO Network latency for backend 172.16.0.151 = 6.02 us 03/02/2021 07:44:37 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:44:37 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:44:37 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:44:44 INFO Checking backend latencies : 03/02/2021 07:44:44 INFO Network latency for backend 172.16.0.151 = 6.81 us 03/02/2021 07:44:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:44:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:44:48 INFO Checking backend latencies : 03/02/2021 07:44:48 INFO Network latency for backend 172.16.0.151 = 7.06 us 03/02/2021 07:44:48 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:44:48 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:44:54 INFO Checking backend latencies : 03/02/2021 07:44:54 INFO Network latency for backend 172.16.0.151 = 6.59 us 03/02/2021 07:44:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:44:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:44:54 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:44:57 INFO GlusterFS mount attempt 03/02/2021 07:44:58 INFO Checking backend latencies : 03/02/2021 07:44:58 INFO Network latency for backend 172.16.0.151 = 7.05 us 03/02/2021 07:44:58 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:44:58 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:45:04 INFO Checking backend latencies : 03/02/2021 07:45:04 INFO Network latency for backend 172.16.0.151 = 7.04 us 03/02/2021 07:45:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:45:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:45:15 INFO Checking backend latencies : 03/02/2021 07:45:15 INFO Network latency for backend 172.16.0.151 = 7.01 us 03/02/2021 07:45:15 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:45:15 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:45:17 INFO Checking backend latencies : 03/02/2021 07:45:17 INFO Network latency for backend 172.16.0.151 = 6.91 us 03/02/2021 07:45:17 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:45:17 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:45:19 INFO Checking backend latencies : 03/02/2021 07:45:19 INFO Network latency for backend 172.16.0.151 = 7.31 us 03/02/2021 07:45:19 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:45:19 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:45:21 INFO Checking backend latencies : 03/02/2021 07:45:21 INFO Network latency for backend 172.16.0.151 = 6.04 us 03/02/2021 07:45:21 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:45:21 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:45:21 ERROR LeaderElectionBase exception dropping old sessions 03/02/2021 07:45:25 INFO Checking backend latencies : 03/02/2021 07:45:25 INFO Network latency for backend 172.16.0.151 = 6.92 us 03/02/2021 07:45:25 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:45:25 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:45:27 INFO Checking backend latencies : 03/02/2021 07:45:27 INFO Network latency for backend 172.16.0.151 = 6.9 us 03/02/2021 07:45:27 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:45:27 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:45:31 INFO GlusterFS mount attempt 03/02/2021 07:45:39 INFO Checking backend latencies : 03/02/2021 07:45:39 INFO Network latency for backend 172.16.0.151 = 7.11 us 03/02/2021 07:45:39 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:45:39 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:45:41 INFO Checking backend latencies : 03/02/2021 07:45:41 INFO Network latency for backend 172.16.0.151 = 6.43 us 03/02/2021 07:45:41 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:45:41 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:45:50 INFO Checking backend latencies : 03/02/2021 07:45:50 INFO Network latency for backend 172.16.0.151 = 7.04 us 03/02/2021 07:45:50 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:45:50 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:45:52 INFO Checking backend latencies : 03/02/2021 07:45:52 INFO Network latency for backend 172.16.0.151 = 7.06 us 03/02/2021 07:45:52 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:45:52 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:45:54 INFO Checking backend latencies : 03/02/2021 07:45:54 INFO Network latency for backend 172.16.0.151 = 7.09 us 03/02/2021 07:45:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:45:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:45:54 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:46:00 INFO Checking backend latencies : 03/02/2021 07:46:00 INFO Network latency for backend 172.16.0.151 = 7.07 us 03/02/2021 07:46:00 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:46:00 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:46:00 ERROR iSCSI Service exception dropping old sessions 03/02/2021 07:46:02 INFO Checking backend latencies : 03/02/2021 07:46:02 INFO Network latency for backend 172.16.0.151 = 6.71 us 03/02/2021 07:46:02 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:46:02 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:46:04 INFO Checking backend latencies : 03/02/2021 07:46:04 INFO Network latency for backend 172.16.0.151 = 6.75 us 03/02/2021 07:46:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:46:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:46:05 INFO GlusterFS mount attempt 03/02/2021 07:46:14 INFO Checking backend latencies : 03/02/2021 07:46:14 INFO Network latency for backend 172.16.0.151 = 7.07 us 03/02/2021 07:46:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:46:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:46:18 INFO Checking backend latencies : 03/02/2021 07:46:18 INFO Network latency for backend 172.16.0.151 = 7.04 us 03/02/2021 07:46:18 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:46:18 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:46:29 INFO Checking backend latencies : 03/02/2021 07:46:29 INFO Network latency for backend 172.16.0.151 = 6.88 us 03/02/2021 07:46:29 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:46:29 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:46:31 INFO Checking backend latencies : 03/02/2021 07:46:31 INFO Network latency for backend 172.16.0.151 = 7.18 us 03/02/2021 07:46:31 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:46:31 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:46:33 INFO Checking backend latencies : 03/02/2021 07:46:33 INFO Network latency for backend 172.16.0.151 = 7.33 us 03/02/2021 07:46:33 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:46:33 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:46:35 INFO Checking backend latencies : 03/02/2021 07:46:35 INFO Network latency for backend 172.16.0.151 = 6.8 us 03/02/2021 07:46:35 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:46:35 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:46:37 INFO GlusterFS mount attempt 03/02/2021 07:46:37 INFO Checking backend latencies : 03/02/2021 07:46:37 INFO Network latency for backend 172.16.0.151 = 6.79 us 03/02/2021 07:46:37 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:46:37 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:46:53 INFO Checking backend latencies : 03/02/2021 07:46:53 INFO Network latency for backend 172.16.0.151 = 6.97 us 03/02/2021 07:46:53 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:46:53 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:46:53 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader 03/02/2021 07:46:53 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 516, in get_key index, data = cons.get(key) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:46:55 INFO Checking backend latencies : 03/02/2021 07:46:55 INFO Network latency for backend 172.16.0.151 = 6.42 us 03/02/2021 07:46:55 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:46:55 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:46:57 INFO Checking backend latencies : 03/02/2021 07:46:57 INFO Network latency for backend 172.16.0.151 = 7.19 us 03/02/2021 07:46:57 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:46:57 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:47:05 INFO Checking backend latencies : 03/02/2021 07:47:05 INFO Network latency for backend 172.16.0.151 = 7.1 us 03/02/2021 07:47:05 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:47:05 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:47:07 INFO Checking backend latencies : 03/02/2021 07:47:07 INFO Network latency for backend 172.16.0.151 = 7.1 us 03/02/2021 07:47:07 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:47:07 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:47:07 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:47:07 INFO GlusterFS mount attempt 03/02/2021 07:47:09 INFO Checking backend latencies : 03/02/2021 07:47:09 INFO Network latency for backend 172.16.0.151 = 7.32 us 03/02/2021 07:47:09 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:47:09 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:47:09 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:47:18 INFO Checking backend latencies : 03/02/2021 07:47:18 INFO Network latency for backend 172.16.0.151 = 7.28 us 03/02/2021 07:47:18 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:47:18 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:47:20 INFO Checking backend latencies : 03/02/2021 07:47:20 INFO Network latency for backend 172.16.0.151 = 7.17 us 03/02/2021 07:47:20 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:47:20 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:47:28 INFO Checking backend latencies : 03/02/2021 07:47:28 INFO Network latency for backend 172.16.0.151 = 6.52 us 03/02/2021 07:47:28 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:47:28 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:47:40 INFO Checking backend latencies : 03/02/2021 07:47:40 INFO Network latency for backend 172.16.0.151 = 7.2 us 03/02/2021 07:47:40 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:47:40 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:47:40 ERROR LeaderElectionBase exception dropping old sessions 03/02/2021 07:47:40 INFO GlusterFS mount attempt 03/02/2021 07:47:45 INFO Checking backend latencies : 03/02/2021 07:47:45 INFO Network latency for backend 172.16.0.151 = 7.13 us 03/02/2021 07:47:45 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:47:45 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:47:47 INFO Checking backend latencies : 03/02/2021 07:47:47 INFO Network latency for backend 172.16.0.151 = 7.23 us 03/02/2021 07:47:47 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:47:47 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:47:49 INFO Checking backend latencies : 03/02/2021 07:47:49 INFO Network latency for backend 172.16.0.151 = 7.09 us 03/02/2021 07:47:49 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:47:49 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:47:51 INFO Checking backend latencies : 03/02/2021 07:47:51 INFO Network latency for backend 172.16.0.151 = 7.18 us 03/02/2021 07:47:51 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:47:51 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:48:09 INFO Checking backend latencies : 03/02/2021 07:48:09 INFO Network latency for backend 172.16.0.151 = 7.21 us 03/02/2021 07:48:09 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:48:09 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:48:11 INFO Checking backend latencies : 03/02/2021 07:48:11 INFO Network latency for backend 172.16.0.151 = 7.15 us 03/02/2021 07:48:11 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:48:11 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:48:13 INFO Checking backend latencies : 03/02/2021 07:48:13 INFO Network latency for backend 172.16.0.151 = 7.23 us 03/02/2021 07:48:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:48:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:48:15 INFO Checking backend latencies : 03/02/2021 07:48:15 INFO Network latency for backend 172.16.0.151 = 6.7 us 03/02/2021 07:48:15 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:48:15 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:48:20 INFO GlusterFS mount attempt 03/02/2021 07:48:20 INFO Checking backend latencies : 03/02/2021 07:48:20 INFO Network latency for backend 172.16.0.151 = 7.14 us 03/02/2021 07:48:20 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:48:20 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:48:20 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:48:22 INFO Checking backend latencies : 03/02/2021 07:48:22 INFO Network latency for backend 172.16.0.151 = 7.0 us 03/02/2021 07:48:22 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:48:22 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:48:22 ERROR iSCSI Service exception dropping old sessions 03/02/2021 07:48:30 INFO Checking backend latencies : 03/02/2021 07:48:30 INFO Network latency for backend 172.16.0.151 = 7.3 us 03/02/2021 07:48:30 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:48:30 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:48:34 INFO Checking backend latencies : 03/02/2021 07:48:34 INFO Network latency for backend 172.16.0.151 = 7.11 us 03/02/2021 07:48:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:48:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:48:36 INFO Checking backend latencies : 03/02/2021 07:48:36 INFO Network latency for backend 172.16.0.151 = 7.25 us 03/02/2021 07:48:36 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:48:36 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:48:42 INFO Checking backend latencies : 03/02/2021 07:48:42 INFO Network latency for backend 172.16.0.151 = 7.13 us 03/02/2021 07:48:42 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:48:42 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:48:52 INFO GlusterFS mount attempt 03/02/2021 07:48:53 INFO Checking backend latencies : 03/02/2021 07:48:53 INFO Network latency for backend 172.16.0.151 = 7.26 us 03/02/2021 07:48:53 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:48:53 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:48:56 INFO Checking backend latencies : 03/02/2021 07:48:56 INFO Network latency for backend 172.16.0.151 = 7.08 us 03/02/2021 07:48:56 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:48:56 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:48:59 INFO Checking backend latencies : 03/02/2021 07:48:59 INFO Network latency for backend 172.16.0.151 = 7.16 us 03/02/2021 07:48:59 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:48:59 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:49:01 INFO Checking backend latencies : 03/02/2021 07:49:01 INFO Network latency for backend 172.16.0.151 = 7.17 us 03/02/2021 07:49:01 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:49:01 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:49:03 INFO Checking backend latencies : 03/02/2021 07:49:03 INFO Network latency for backend 172.16.0.151 = 7.25 us 03/02/2021 07:49:03 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:49:03 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:49:17 INFO Checking backend latencies : 03/02/2021 07:49:17 INFO Network latency for backend 172.16.0.151 = 7.23 us 03/02/2021 07:49:17 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:49:17 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:49:17 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:49:19 INFO Checking backend latencies : 03/02/2021 07:49:19 INFO Network latency for backend 172.16.0.151 = 6.94 us 03/02/2021 07:49:19 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:49:19 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:49:21 INFO Checking backend latencies : 03/02/2021 07:49:21 INFO Network latency for backend 172.16.0.151 = 6.53 us 03/02/2021 07:49:21 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:49:21 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:49:25 INFO Checking backend latencies : 03/02/2021 07:49:25 INFO Network latency for backend 172.16.0.151 = 7.18 us 03/02/2021 07:49:25 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:49:25 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:49:26 INFO GlusterFS mount attempt 03/02/2021 07:49:27 INFO Checking backend latencies : 03/02/2021 07:49:27 INFO Network latency for backend 172.16.0.151 = 7.25 us 03/02/2021 07:49:27 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:49:27 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:49:37 INFO Checking backend latencies : 03/02/2021 07:49:37 INFO Network latency for backend 172.16.0.151 = 7.28 us 03/02/2021 07:49:37 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:49:37 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:49:37 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:49:39 INFO Checking backend latencies : 03/02/2021 07:49:39 INFO Network latency for backend 172.16.0.151 = 7.22 us 03/02/2021 07:49:39 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:49:39 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:49:42 INFO Checking backend latencies : 03/02/2021 07:49:42 INFO Network latency for backend 172.16.0.151 = 6.23 us 03/02/2021 07:49:42 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:49:42 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:49:48 INFO Checking backend latencies : 03/02/2021 07:49:48 INFO Network latency for backend 172.16.0.151 = 7.21 us 03/02/2021 07:49:48 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:49:48 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:49:56 INFO Checking backend latencies : 03/02/2021 07:49:56 INFO Network latency for backend 172.16.0.151 = 7.1 us 03/02/2021 07:49:56 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:49:56 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:49:56 ERROR LeaderElectionBase exception dropping old sessions 03/02/2021 07:49:59 INFO GlusterFS mount attempt 03/02/2021 07:50:00 INFO Checking backend latencies : 03/02/2021 07:50:00 INFO Network latency for backend 172.16.0.151 = 6.03 us 03/02/2021 07:50:00 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:50:00 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:50:02 INFO Checking backend latencies : 03/02/2021 07:50:02 INFO Network latency for backend 172.16.0.151 = 7.1 us 03/02/2021 07:50:02 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:50:02 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:50:02 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader 03/02/2021 07:50:02 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 516, in get_key index, data = cons.get(key) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:50:04 INFO Checking backend latencies : 03/02/2021 07:50:04 INFO Network latency for backend 172.16.0.151 = 7.17 us 03/02/2021 07:50:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:50:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:50:10 INFO Checking backend latencies : 03/02/2021 07:50:10 INFO Network latency for backend 172.16.0.151 = 7.06 us 03/02/2021 07:50:10 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:50:10 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:50:15 INFO Checking backend latencies : 03/02/2021 07:50:15 INFO Network latency for backend 172.16.0.151 = 7.21 us 03/02/2021 07:50:15 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:50:15 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:50:23 INFO Checking backend latencies : 03/02/2021 07:50:23 INFO Network latency for backend 172.16.0.151 = 7.03 us 03/02/2021 07:50:23 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:50:23 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:50:25 INFO Checking backend latencies : 03/02/2021 07:50:25 INFO Network latency for backend 172.16.0.151 = 7.38 us 03/02/2021 07:50:25 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:50:25 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:50:27 INFO Checking backend latencies : 03/02/2021 07:50:27 INFO Network latency for backend 172.16.0.151 = 6.6 us 03/02/2021 07:50:27 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:50:27 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:50:31 INFO GlusterFS mount attempt 03/02/2021 07:50:45 INFO Checking backend latencies : 03/02/2021 07:50:45 INFO Network latency for backend 172.16.0.151 = 7.31 us 03/02/2021 07:50:45 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:50:45 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:50:45 ERROR iSCSI Service exception dropping old sessions 03/02/2021 07:50:48 INFO Checking backend latencies : 03/02/2021 07:50:48 INFO Network latency for backend 172.16.0.151 = 7.24 us 03/02/2021 07:50:48 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:50:48 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:50:50 INFO Checking backend latencies : 03/02/2021 07:50:50 INFO Network latency for backend 172.16.0.151 = 7.31 us 03/02/2021 07:50:50 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:50:50 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:50:52 INFO Checking backend latencies : 03/02/2021 07:50:52 INFO Network latency for backend 172.16.0.151 = 6.14 us 03/02/2021 07:50:52 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:50:52 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:50:54 INFO Checking backend latencies : 03/02/2021 07:50:54 INFO Network latency for backend 172.16.0.151 = 6.47 us 03/02/2021 07:50:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:50:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:50:54 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:51:02 INFO GlusterFS mount attempt 03/02/2021 07:51:11 INFO Checking backend latencies : 03/02/2021 07:51:11 INFO Network latency for backend 172.16.0.151 = 7.28 us 03/02/2021 07:51:11 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:51:11 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:51:17 INFO Checking backend latencies : 03/02/2021 07:51:17 INFO Network latency for backend 172.16.0.151 = 7.32 us 03/02/2021 07:51:17 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:51:17 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:51:19 INFO Checking backend latencies : 03/02/2021 07:51:19 INFO Network latency for backend 172.16.0.151 = 6.06 us 03/02/2021 07:51:19 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:51:19 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:51:21 INFO Checking backend latencies : 03/02/2021 07:51:21 INFO Network latency for backend 172.16.0.151 = 7.26 us 03/02/2021 07:51:21 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:51:21 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:51:23 INFO Checking backend latencies : 03/02/2021 07:51:23 INFO Network latency for backend 172.16.0.151 = 7.42 us 03/02/2021 07:51:23 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:51:23 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:51:27 INFO Checking backend latencies : 03/02/2021 07:51:27 INFO Network latency for backend 172.16.0.151 = 6.98 us 03/02/2021 07:51:27 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:51:27 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:51:27 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:51:33 INFO GlusterFS mount attempt 03/02/2021 07:51:38 INFO Checking backend latencies : 03/02/2021 07:51:38 INFO Network latency for backend 172.16.0.151 = 7.26 us 03/02/2021 07:51:38 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:51:38 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:51:41 INFO Checking backend latencies : 03/02/2021 07:51:41 INFO Network latency for backend 172.16.0.151 = 7.19 us 03/02/2021 07:51:41 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:51:41 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:51:43 INFO Checking backend latencies : 03/02/2021 07:51:43 INFO Network latency for backend 172.16.0.151 = 7.17 us 03/02/2021 07:51:43 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:51:43 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:51:45 INFO Checking backend latencies : 03/02/2021 07:51:45 INFO Network latency for backend 172.16.0.151 = 7.25 us 03/02/2021 07:51:45 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:51:45 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:51:48 INFO Checking backend latencies : 03/02/2021 07:51:48 INFO Network latency for backend 172.16.0.151 = 6.03 us 03/02/2021 07:51:48 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:51:48 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:51:53 INFO Checking backend latencies : 03/02/2021 07:51:53 INFO Network latency for backend 172.16.0.151 = 6.02 us 03/02/2021 07:51:53 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:51:53 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:51:53 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:52:04 INFO Checking backend latencies : 03/02/2021 07:52:04 INFO Network latency for backend 172.16.0.151 = 6.84 us 03/02/2021 07:52:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:52:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:52:05 INFO GlusterFS mount attempt 03/02/2021 07:52:06 INFO Checking backend latencies : 03/02/2021 07:52:06 INFO Network latency for backend 172.16.0.151 = 7.22 us 03/02/2021 07:52:06 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:52:06 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:52:08 INFO Checking backend latencies : 03/02/2021 07:52:08 INFO Network latency for backend 172.16.0.151 = 7.37 us 03/02/2021 07:52:08 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:52:08 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:52:10 INFO Checking backend latencies : 03/02/2021 07:52:10 INFO Network latency for backend 172.16.0.151 = 6.63 us 03/02/2021 07:52:10 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:52:10 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:52:13 INFO Checking backend latencies : 03/02/2021 07:52:13 INFO Network latency for backend 172.16.0.151 = 7.18 us 03/02/2021 07:52:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:52:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:52:13 ERROR LeaderElectionBase exception dropping old sessions 03/02/2021 07:52:31 INFO Checking backend latencies : 03/02/2021 07:52:31 INFO Network latency for backend 172.16.0.151 = 7.12 us 03/02/2021 07:52:31 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:52:31 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:52:33 INFO Checking backend latencies : 03/02/2021 07:52:33 INFO Network latency for backend 172.16.0.151 = 7.1 us 03/02/2021 07:52:33 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:52:33 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:52:35 INFO Checking backend latencies : 03/02/2021 07:52:35 INFO Network latency for backend 172.16.0.151 = 7.18 us 03/02/2021 07:52:35 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:52:35 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:52:37 INFO Checking backend latencies : 03/02/2021 07:52:37 INFO Network latency for backend 172.16.0.151 = 7.21 us 03/02/2021 07:52:37 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:52:37 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:52:37 INFO GlusterFS mount attempt 03/02/2021 07:52:39 INFO Checking backend latencies : 03/02/2021 07:52:39 INFO Network latency for backend 172.16.0.151 = 7.28 us 03/02/2021 07:52:39 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:52:39 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:52:43 INFO Checking backend latencies : 03/02/2021 07:52:43 INFO Network latency for backend 172.16.0.151 = 7.26 us 03/02/2021 07:52:43 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:52:43 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:52:55 INFO Checking backend latencies : 03/02/2021 07:52:55 INFO Network latency for backend 172.16.0.151 = 7.25 us 03/02/2021 07:52:55 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:52:55 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:53:04 INFO Checking backend latencies : 03/02/2021 07:53:04 INFO Network latency for backend 172.16.0.151 = 7.12 us 03/02/2021 07:53:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:53:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:53:06 INFO Checking backend latencies : 03/02/2021 07:53:06 INFO Network latency for backend 172.16.0.151 = 7.14 us 03/02/2021 07:53:06 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:53:06 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:53:08 INFO Checking backend latencies : 03/02/2021 07:53:08 INFO Network latency for backend 172.16.0.151 = 7.16 us 03/02/2021 07:53:08 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:53:08 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:53:08 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:53:10 INFO GlusterFS mount attempt 03/02/2021 07:53:10 INFO Checking backend latencies : 03/02/2021 07:53:10 INFO Network latency for backend 172.16.0.151 = 7.34 us 03/02/2021 07:53:10 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:53:10 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:53:10 ERROR iSCSI Service exception dropping old sessions 03/02/2021 07:53:14 INFO Checking backend latencies : 03/02/2021 07:53:14 INFO Network latency for backend 172.16.0.151 = 7.18 us 03/02/2021 07:53:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:53:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:53:14 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader 03/02/2021 07:53:14 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 516, in get_key index, data = cons.get(key) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:53:18 INFO Checking backend latencies : 03/02/2021 07:53:18 INFO Network latency for backend 172.16.0.151 = 7.15 us 03/02/2021 07:53:18 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:53:18 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:53:27 INFO Checking backend latencies : 03/02/2021 07:53:27 INFO Network latency for backend 172.16.0.151 = 7.35 us 03/02/2021 07:53:27 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:53:27 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:53:41 INFO Checking backend latencies : 03/02/2021 07:53:41 INFO Network latency for backend 172.16.0.151 = 7.22 us 03/02/2021 07:53:41 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:53:41 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:53:41 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:53:42 INFO GlusterFS mount attempt 03/02/2021 07:53:44 INFO Checking backend latencies : 03/02/2021 07:53:44 INFO Network latency for backend 172.16.0.151 = 7.33 us 03/02/2021 07:53:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:53:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:53:46 INFO Checking backend latencies : 03/02/2021 07:53:46 INFO Network latency for backend 172.16.0.151 = 6.56 us 03/02/2021 07:53:46 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:53:46 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:53:48 INFO Checking backend latencies : 03/02/2021 07:53:48 INFO Network latency for backend 172.16.0.151 = 6.36 us 03/02/2021 07:53:48 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:53:48 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:53:55 INFO Checking backend latencies : 03/02/2021 07:53:55 INFO Network latency for backend 172.16.0.151 = 7.02 us 03/02/2021 07:53:55 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:53:55 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:54:02 INFO Checking backend latencies : 03/02/2021 07:54:02 INFO Network latency for backend 172.16.0.151 = 7.36 us 03/02/2021 07:54:02 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:54:02 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:54:06 INFO Checking backend latencies : 03/02/2021 07:54:06 INFO Network latency for backend 172.16.0.151 = 7.16 us 03/02/2021 07:54:06 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:54:06 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:54:09 INFO Checking backend latencies : 03/02/2021 07:54:09 INFO Network latency for backend 172.16.0.151 = 7.33 us 03/02/2021 07:54:09 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:54:09 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:54:14 INFO GlusterFS mount attempt 03/02/2021 07:54:22 INFO Checking backend latencies : 03/02/2021 07:54:22 INFO Network latency for backend 172.16.0.151 = 7.31 us 03/02/2021 07:54:22 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:54:22 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:54:22 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:54:24 INFO Checking backend latencies : 03/02/2021 07:54:24 INFO Network latency for backend 172.16.0.151 = 6.23 us 03/02/2021 07:54:24 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:54:24 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:54:29 INFO Checking backend latencies : 03/02/2021 07:54:29 INFO Network latency for backend 172.16.0.151 = 7.17 us 03/02/2021 07:54:29 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:54:29 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:54:31 INFO Checking backend latencies : 03/02/2021 07:54:31 INFO Network latency for backend 172.16.0.151 = 7.04 us 03/02/2021 07:54:31 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:54:31 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:54:31 ERROR LeaderElectionBase exception dropping old sessions 03/02/2021 07:54:33 INFO Checking backend latencies : 03/02/2021 07:54:33 INFO Network latency for backend 172.16.0.151 = 7.22 us 03/02/2021 07:54:33 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:54:33 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:54:35 INFO Checking backend latencies : 03/02/2021 07:54:35 INFO Network latency for backend 172.16.0.151 = 7.09 us 03/02/2021 07:54:35 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:54:35 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:54:46 INFO GlusterFS mount attempt 03/02/2021 07:54:47 INFO Checking backend latencies : 03/02/2021 07:54:47 INFO Network latency for backend 172.16.0.151 = 7.28 us 03/02/2021 07:54:47 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:54:47 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:54:56 INFO Checking backend latencies : 03/02/2021 07:54:56 INFO Network latency for backend 172.16.0.151 = 7.36 us 03/02/2021 07:54:56 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:54:56 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:54:58 INFO Checking backend latencies : 03/02/2021 07:54:58 INFO Network latency for backend 172.16.0.151 = 7.27 us 03/02/2021 07:54:58 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:54:58 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:55:00 INFO Checking backend latencies : 03/02/2021 07:55:00 INFO Network latency for backend 172.16.0.151 = 7.17 us 03/02/2021 07:55:00 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:55:00 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:55:02 INFO Checking backend latencies : 03/02/2021 07:55:02 INFO Network latency for backend 172.16.0.151 = 6.08 us 03/02/2021 07:55:02 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:55:02 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:55:04 INFO Checking backend latencies : 03/02/2021 07:55:04 INFO Network latency for backend 172.16.0.151 = 7.04 us 03/02/2021 07:55:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:55:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:55:14 INFO Checking backend latencies : 03/02/2021 07:55:14 INFO Network latency for backend 172.16.0.151 = 7.25 us 03/02/2021 07:55:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:55:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:55:19 INFO GlusterFS mount attempt 03/02/2021 07:55:19 INFO Checking backend latencies : 03/02/2021 07:55:19 INFO Network latency for backend 172.16.0.151 = 7.22 us 03/02/2021 07:55:19 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:55:19 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:55:21 INFO Checking backend latencies : 03/02/2021 07:55:21 INFO Network latency for backend 172.16.0.151 = 7.29 us 03/02/2021 07:55:21 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:55:21 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:55:25 INFO Checking backend latencies : 03/02/2021 07:55:25 INFO Network latency for backend 172.16.0.151 = 7.16 us 03/02/2021 07:55:25 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:55:25 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:55:33 INFO Checking backend latencies : 03/02/2021 07:55:33 INFO Network latency for backend 172.16.0.151 = 7.28 us 03/02/2021 07:55:33 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:55:33 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:55:33 ERROR iSCSI Service exception dropping old sessions 03/02/2021 07:55:37 INFO Checking backend latencies : 03/02/2021 07:55:37 INFO Network latency for backend 172.16.0.151 = 7.1 us 03/02/2021 07:55:37 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:55:37 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:55:37 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:55:44 INFO Checking backend latencies : 03/02/2021 07:55:44 INFO Network latency for backend 172.16.0.151 = 7.25 us 03/02/2021 07:55:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:55:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:55:46 INFO Checking backend latencies : 03/02/2021 07:55:46 INFO Network latency for backend 172.16.0.151 = 6.26 us 03/02/2021 07:55:46 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:55:46 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:55:48 INFO Checking backend latencies : 03/02/2021 07:55:48 INFO Network latency for backend 172.16.0.151 = 7.36 us 03/02/2021 07:55:48 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:55:48 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:55:50 INFO Checking backend latencies : 03/02/2021 07:55:50 INFO Network latency for backend 172.16.0.151 = 7.3 us 03/02/2021 07:55:50 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:55:50 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:55:50 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:55:51 INFO GlusterFS mount attempt 03/02/2021 07:56:07 INFO Checking backend latencies : 03/02/2021 07:56:07 INFO Network latency for backend 172.16.0.151 = 7.04 us 03/02/2021 07:56:07 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:56:07 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:56:11 INFO Checking backend latencies : 03/02/2021 07:56:11 INFO Network latency for backend 172.16.0.151 = 7.03 us 03/02/2021 07:56:11 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:56:11 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:56:13 INFO Checking backend latencies : 03/02/2021 07:56:13 INFO Network latency for backend 172.16.0.151 = 7.07 us 03/02/2021 07:56:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:56:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:56:15 INFO Checking backend latencies : 03/02/2021 07:56:15 INFO Network latency for backend 172.16.0.151 = 7.16 us 03/02/2021 07:56:15 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:56:15 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:56:18 INFO Checking backend latencies : 03/02/2021 07:56:18 INFO Network latency for backend 172.16.0.151 = 7.43 us 03/02/2021 07:56:18 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:56:18 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:56:21 INFO Checking backend latencies : 03/02/2021 07:56:21 INFO Network latency for backend 172.16.0.151 = 7.28 us 03/02/2021 07:56:21 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:56:21 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:56:21 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader 03/02/2021 07:56:21 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 516, in get_key index, data = cons.get(key) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:56:22 INFO GlusterFS mount attempt 03/02/2021 07:56:28 INFO Checking backend latencies : 03/02/2021 07:56:28 INFO Network latency for backend 172.16.0.151 = 7.12 us 03/02/2021 07:56:28 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:56:28 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:56:33 INFO Checking backend latencies : 03/02/2021 07:56:33 INFO Network latency for backend 172.16.0.151 = 7.31 us 03/02/2021 07:56:33 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:56:33 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:56:39 INFO Checking backend latencies : 03/02/2021 07:56:39 INFO Network latency for backend 172.16.0.151 = 7.18 us 03/02/2021 07:56:39 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:56:39 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:56:49 INFO Checking backend latencies : 03/02/2021 07:56:49 INFO Network latency for backend 172.16.0.151 = 6.98 us 03/02/2021 07:56:49 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:56:49 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:56:49 ERROR LeaderElectionBase exception dropping old sessions 03/02/2021 07:56:51 INFO Checking backend latencies : 03/02/2021 07:56:51 INFO Network latency for backend 172.16.0.151 = 7.42 us 03/02/2021 07:56:51 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:56:51 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:56:51 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:56:53 INFO Checking backend latencies : 03/02/2021 07:56:53 INFO Network latency for backend 172.16.0.151 = 7.23 us 03/02/2021 07:56:53 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:56:53 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:56:53 INFO GlusterFS mount attempt 03/02/2021 07:56:56 INFO Checking backend latencies : 03/02/2021 07:56:56 INFO Network latency for backend 172.16.0.151 = 7.23 us 03/02/2021 07:56:56 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:56:56 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:57:02 INFO Checking backend latencies : 03/02/2021 07:57:02 INFO Network latency for backend 172.16.0.151 = 7.38 us 03/02/2021 07:57:02 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:57:02 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:57:18 INFO Checking backend latencies : 03/02/2021 07:57:18 INFO Network latency for backend 172.16.0.151 = 6.78 us 03/02/2021 07:57:18 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:57:18 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:57:20 INFO Checking backend latencies : 03/02/2021 07:57:20 INFO Network latency for backend 172.16.0.151 = 7.18 us 03/02/2021 07:57:20 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:57:20 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:57:23 INFO Checking backend latencies : 03/02/2021 07:57:23 INFO Network latency for backend 172.16.0.151 = 6.17 us 03/02/2021 07:57:23 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:57:23 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:57:24 INFO GlusterFS mount attempt 03/02/2021 07:57:29 INFO Checking backend latencies : 03/02/2021 07:57:29 INFO Network latency for backend 172.16.0.151 = 6.89 us 03/02/2021 07:57:29 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:57:29 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:57:33 INFO Checking backend latencies : 03/02/2021 07:57:33 INFO Network latency for backend 172.16.0.151 = 7.16 us 03/02/2021 07:57:33 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:57:33 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:57:39 INFO Checking backend latencies : 03/02/2021 07:57:39 INFO Network latency for backend 172.16.0.151 = 7.34 us 03/02/2021 07:57:39 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:57:39 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:57:41 INFO Checking backend latencies : 03/02/2021 07:57:41 INFO Network latency for backend 172.16.0.151 = 7.12 us 03/02/2021 07:57:41 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:57:41 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:57:54 INFO Checking backend latencies : 03/02/2021 07:57:54 INFO Network latency for backend 172.16.0.151 = 6.06 us 03/02/2021 07:57:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:57:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:57:55 INFO GlusterFS mount attempt 03/02/2021 07:57:56 INFO Checking backend latencies : 03/02/2021 07:57:56 INFO Network latency for backend 172.16.0.151 = 6.03 us 03/02/2021 07:57:56 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:57:56 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:57:56 ERROR iSCSI Service exception dropping old sessions 03/02/2021 07:57:58 INFO Checking backend latencies : 03/02/2021 07:57:58 INFO Network latency for backend 172.16.0.151 = 7.34 us 03/02/2021 07:57:58 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:57:58 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:57:58 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:58:02 INFO Checking backend latencies : 03/02/2021 07:58:02 INFO Network latency for backend 172.16.0.151 = 7.21 us 03/02/2021 07:58:02 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:58:02 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:58:04 INFO Checking backend latencies : 03/02/2021 07:58:04 INFO Network latency for backend 172.16.0.151 = 7.24 us 03/02/2021 07:58:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:58:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:58:06 INFO Checking backend latencies : 03/02/2021 07:58:06 INFO Network latency for backend 172.16.0.151 = 7.39 us 03/02/2021 07:58:06 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:58:06 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:58:06 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:58:17 INFO Checking backend latencies : 03/02/2021 07:58:17 INFO Network latency for backend 172.16.0.151 = 7.09 us 03/02/2021 07:58:17 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:58:17 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:58:20 INFO Checking backend latencies : 03/02/2021 07:58:20 INFO Network latency for backend 172.16.0.151 = 7.3 us 03/02/2021 07:58:20 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:58:20 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:58:26 INFO GlusterFS mount attempt 03/02/2021 07:58:27 INFO Checking backend latencies : 03/02/2021 07:58:27 INFO Network latency for backend 172.16.0.151 = 7.21 us 03/02/2021 07:58:27 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:58:27 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:58:29 INFO Checking backend latencies : 03/02/2021 07:58:29 INFO Network latency for backend 172.16.0.151 = 6.43 us 03/02/2021 07:58:29 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:58:29 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:58:31 INFO Checking backend latencies : 03/02/2021 07:58:31 INFO Network latency for backend 172.16.0.151 = 7.32 us 03/02/2021 07:58:31 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:58:31 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:58:41 INFO Checking backend latencies : 03/02/2021 07:58:41 INFO Network latency for backend 172.16.0.151 = 6.87 us 03/02/2021 07:58:41 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:58:41 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:58:44 INFO Checking backend latencies : 03/02/2021 07:58:44 INFO Network latency for backend 172.16.0.151 = 7.36 us 03/02/2021 07:58:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:58:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:58:48 INFO Checking backend latencies : 03/02/2021 07:58:48 INFO Network latency for backend 172.16.0.151 = 7.28 us 03/02/2021 07:58:48 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:58:48 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:58:52 INFO Checking backend latencies : 03/02/2021 07:58:52 INFO Network latency for backend 172.16.0.151 = 7.09 us 03/02/2021 07:58:52 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:58:52 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:58:54 INFO Checking backend latencies : 03/02/2021 07:58:54 INFO Network latency for backend 172.16.0.151 = 7.31 us 03/02/2021 07:58:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:58:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:58:57 INFO GlusterFS mount attempt 03/02/2021 07:59:04 INFO Checking backend latencies : 03/02/2021 07:59:04 INFO Network latency for backend 172.16.0.151 = 7.25 us 03/02/2021 07:59:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:59:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:59:06 INFO Checking backend latencies : 03/02/2021 07:59:06 INFO Network latency for backend 172.16.0.151 = 7.28 us 03/02/2021 07:59:06 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:59:06 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:59:06 ERROR LeaderElectionBase exception dropping old sessions 03/02/2021 07:59:09 INFO Checking backend latencies : 03/02/2021 07:59:09 INFO Network latency for backend 172.16.0.151 = 7.28 us 03/02/2021 07:59:09 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:59:09 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:59:15 INFO Checking backend latencies : 03/02/2021 07:59:15 INFO Network latency for backend 172.16.0.151 = 7.09 us 03/02/2021 07:59:15 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:59:15 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:59:20 INFO Checking backend latencies : 03/02/2021 07:59:20 INFO Network latency for backend 172.16.0.151 = 7.01 us 03/02/2021 07:59:20 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:59:20 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:59:20 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:59:29 INFO GlusterFS mount attempt 03/02/2021 07:59:30 INFO Checking backend latencies : 03/02/2021 07:59:30 INFO Network latency for backend 172.16.0.151 = 7.3 us 03/02/2021 07:59:30 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:59:30 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:59:30 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader 03/02/2021 07:59:30 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 516, in get_key index, data = cons.get(key) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:59:32 INFO Checking backend latencies : 03/02/2021 07:59:32 INFO Network latency for backend 172.16.0.151 = 7.29 us 03/02/2021 07:59:32 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:59:32 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:59:34 INFO Checking backend latencies : 03/02/2021 07:59:34 INFO Network latency for backend 172.16.0.151 = 6.3 us 03/02/2021 07:59:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:59:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:59:36 INFO Checking backend latencies : 03/02/2021 07:59:36 INFO Network latency for backend 172.16.0.151 = 6.82 us 03/02/2021 07:59:36 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:59:36 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:59:43 INFO Checking backend latencies : 03/02/2021 07:59:43 INFO Network latency for backend 172.16.0.151 = 7.36 us 03/02/2021 07:59:43 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:59:43 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:59:57 INFO Checking backend latencies : 03/02/2021 07:59:57 INFO Network latency for backend 172.16.0.151 = 7.21 us 03/02/2021 07:59:57 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:59:57 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:00:01 INFO Checking backend latencies : 03/02/2021 08:00:01 INFO Network latency for backend 172.16.0.151 = 7.19 us 03/02/2021 08:00:01 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:00:01 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:00:01 INFO GlusterFS mount attempt 03/02/2021 08:00:03 INFO Checking backend latencies : 03/02/2021 08:00:03 INFO Network latency for backend 172.16.0.151 = 6.74 us 03/02/2021 08:00:03 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:00:03 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:00:07 INFO Checking backend latencies : 03/02/2021 08:00:07 INFO Network latency for backend 172.16.0.151 = 7.33 us 03/02/2021 08:00:07 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:00:07 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:00:07 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:00:18 INFO Checking backend latencies : 03/02/2021 08:00:18 INFO Network latency for backend 172.16.0.151 = 7.22 us 03/02/2021 08:00:18 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:00:18 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:00:18 ERROR iSCSI Service exception dropping old sessions 03/02/2021 08:00:20 INFO Checking backend latencies : 03/02/2021 08:00:20 INFO Network latency for backend 172.16.0.151 = 6.56 us 03/02/2021 08:00:20 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:00:20 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:00:24 INFO Checking backend latencies : 03/02/2021 08:00:24 INFO Network latency for backend 172.16.0.151 = 6.47 us 03/02/2021 08:00:24 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:00:24 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:00:28 INFO Checking backend latencies : 03/02/2021 08:00:28 INFO Network latency for backend 172.16.0.151 = 7.23 us 03/02/2021 08:00:28 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:00:28 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:00:33 INFO GlusterFS mount attempt 03/02/2021 08:00:36 INFO Checking backend latencies : 03/02/2021 08:00:36 INFO Network latency for backend 172.16.0.151 = 7.31 us 03/02/2021 08:00:36 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:00:36 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:00:36 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:00:47 INFO Checking backend latencies : 03/02/2021 08:00:47 INFO Network latency for backend 172.16.0.151 = 7.13 us 03/02/2021 08:00:47 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:00:47 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:00:49 INFO Checking backend latencies : 03/02/2021 08:00:49 INFO Network latency for backend 172.16.0.151 = 6.95 us 03/02/2021 08:00:49 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:00:49 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:00:51 INFO Checking backend latencies : 03/02/2021 08:00:51 INFO Network latency for backend 172.16.0.151 = 6.02 us 03/02/2021 08:00:51 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:00:51 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:00:53 INFO Checking backend latencies : 03/02/2021 08:00:53 INFO Network latency for backend 172.16.0.151 = 6.21 us 03/02/2021 08:00:53 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:00:53 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:00:55 INFO Checking backend latencies : 03/02/2021 08:00:55 INFO Network latency for backend 172.16.0.151 = 7.11 us 03/02/2021 08:00:55 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:00:55 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:01:14 INFO Checking backend latencies : 03/02/2021 08:01:14 INFO Network latency for backend 172.16.0.151 = 6.15 us 03/02/2021 08:01:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:01:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:01:14 INFO GlusterFS mount attempt 03/02/2021 08:01:16 INFO Checking backend latencies : 03/02/2021 08:01:16 INFO Network latency for backend 172.16.0.151 = 7.41 us 03/02/2021 08:01:16 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:01:16 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:01:18 INFO Checking backend latencies : 03/02/2021 08:01:18 INFO Network latency for backend 172.16.0.151 = 6.02 us 03/02/2021 08:01:18 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:01:18 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:01:19 INFO Checking backend latencies : 03/02/2021 08:01:19 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:01:19 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:01:19 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:01:20 INFO Checking backend latencies : 03/02/2021 08:01:20 INFO Network latency for backend 172.16.0.151 = 6.07 us 03/02/2021 08:01:20 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:01:20 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:01:21 INFO Checking backend latencies : 03/02/2021 08:01:21 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:01:21 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:01:21 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:01:25 INFO Checking backend latencies : 03/02/2021 08:01:25 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:01:25 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:01:25 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:01:25 ERROR LeaderElectionBase exception dropping old sessions 03/02/2021 08:01:38 INFO Checking backend latencies : 03/02/2021 08:01:38 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:01:38 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:01:38 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:01:44 INFO Checking backend latencies : 03/02/2021 08:01:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:01:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:01:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:01:45 INFO Checking backend latencies : 03/02/2021 08:01:45 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:01:45 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:01:45 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:01:46 INFO Checking backend latencies : 03/02/2021 08:01:46 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:01:46 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:01:46 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:01:46 INFO Checking backend latencies : 03/02/2021 08:01:46 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:01:46 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:01:46 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:01:55 INFO GlusterFS mount attempt 03/02/2021 08:01:58 INFO Checking backend latencies : 03/02/2021 08:01:58 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:01:58 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:01:58 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:01:58 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:01:58 INFO Checking backend latencies : 03/02/2021 08:01:58 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:01:58 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:01:58 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:02:04 INFO Checking backend latencies : 03/02/2021 08:02:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:02:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:02:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:02:11 INFO Checking backend latencies : 03/02/2021 08:02:11 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:02:11 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:02:11 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:02:14 INFO Checking backend latencies : 03/02/2021 08:02:14 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:02:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:02:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:02:16 INFO Checking backend latencies : 03/02/2021 08:02:16 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:02:16 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:02:16 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:02:21 INFO Checking backend latencies : 03/02/2021 08:02:21 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:02:21 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:02:21 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:02:22 INFO Checking backend latencies : 03/02/2021 08:02:22 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:02:22 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:02:22 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:02:24 INFO Checking backend latencies : 03/02/2021 08:02:24 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:02:24 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:02:24 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:02:24 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:02:34 INFO Checking backend latencies : 03/02/2021 08:02:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:02:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:02:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:02:36 INFO GlusterFS mount attempt 03/02/2021 08:02:41 INFO Checking backend latencies : 03/02/2021 08:02:41 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:02:41 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:02:41 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:02:45 INFO Checking backend latencies : 03/02/2021 08:02:45 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:02:45 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:02:45 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:02:48 INFO Checking backend latencies : 03/02/2021 08:02:48 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:02:48 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:02:48 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:02:48 INFO Checking backend latencies : 03/02/2021 08:02:48 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:02:48 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:02:48 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:02:53 INFO Checking backend latencies : 03/02/2021 08:02:53 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:02:53 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:02:53 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:02:53 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader 03/02/2021 08:02:53 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 516, in get_key index, data = cons.get(key) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:02:54 INFO Checking backend latencies : 03/02/2021 08:02:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:02:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:02:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:02:54 ERROR iSCSI Service exception dropping old sessions 03/02/2021 08:03:09 INFO GlusterFS mount attempt 03/02/2021 08:03:11 INFO Checking backend latencies : 03/02/2021 08:03:11 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:03:11 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:03:11 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:03:12 INFO Checking backend latencies : 03/02/2021 08:03:12 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:03:12 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:03:12 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:03:12 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:03:13 INFO Checking backend latencies : 03/02/2021 08:03:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:03:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:03:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:03:19 INFO Checking backend latencies : 03/02/2021 08:03:19 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:03:19 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:03:19 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:03:19 INFO Checking backend latencies : 03/02/2021 08:03:19 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:03:19 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:03:19 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:03:19 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:03:21 INFO Checking backend latencies : 03/02/2021 08:03:21 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:03:21 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:03:21 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:03:31 INFO Checking backend latencies : 03/02/2021 08:03:31 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:03:31 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:03:31 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:03:39 INFO Checking backend latencies : 03/02/2021 08:03:39 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:03:39 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:03:39 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:03:40 INFO GlusterFS mount attempt 03/02/2021 08:03:41 INFO Checking backend latencies : 03/02/2021 08:03:41 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:03:41 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:03:41 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:03:45 INFO Checking backend latencies : 03/02/2021 08:03:45 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:03:45 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:03:45 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:03:55 INFO Checking backend latencies : 03/02/2021 08:03:55 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:03:55 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:03:55 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:03:57 INFO Checking backend latencies : 03/02/2021 08:03:57 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:03:57 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:03:57 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:03:57 ERROR LeaderElectionBase exception dropping old sessions 03/02/2021 08:04:09 INFO Checking backend latencies : 03/02/2021 08:04:09 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:04:09 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:04:09 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:04:11 INFO GlusterFS mount attempt 03/02/2021 08:04:11 INFO Checking backend latencies : 03/02/2021 08:04:11 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:04:11 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:04:11 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:04:15 INFO Checking backend latencies : 03/02/2021 08:04:15 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:04:15 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:04:15 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:04:20 INFO Checking backend latencies : 03/02/2021 08:04:20 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:04:20 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:04:20 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:04:20 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:04:21 INFO Checking backend latencies : 03/02/2021 08:04:21 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:04:21 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:04:21 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:04:21 INFO Checking backend latencies : 03/02/2021 08:04:21 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:04:21 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:04:21 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:04:29 INFO Checking backend latencies : 03/02/2021 08:04:29 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:04:29 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:04:29 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:04:39 INFO Checking backend latencies : 03/02/2021 08:04:39 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:04:39 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:04:39 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:04:41 INFO Checking backend latencies : 03/02/2021 08:04:41 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:04:41 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:04:41 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:04:43 INFO GlusterFS mount attempt 03/02/2021 08:04:45 INFO Checking backend latencies : 03/02/2021 08:04:45 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:04:45 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:04:45 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:04:47 INFO Checking backend latencies : 03/02/2021 08:04:47 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:04:47 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:04:47 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:04:47 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:04:52 INFO Checking backend latencies : 03/02/2021 08:04:52 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:04:52 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:04:52 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:04:53 INFO Checking backend latencies : 03/02/2021 08:04:53 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:04:53 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:04:53 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:05:05 INFO Checking backend latencies : 03/02/2021 08:05:05 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:05:05 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:05:05 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:05:11 INFO Checking backend latencies : 03/02/2021 08:05:11 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:05:11 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:05:11 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:05:11 INFO Checking backend latencies : 03/02/2021 08:05:11 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:05:11 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:05:11 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:05:18 INFO GlusterFS mount attempt 03/02/2021 08:05:19 INFO Checking backend latencies : 03/02/2021 08:05:19 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:05:19 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:05:19 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:05:20 INFO Checking backend latencies : 03/02/2021 08:05:20 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:05:20 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:05:20 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:05:20 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:05:30 INFO Checking backend latencies : 03/02/2021 08:05:30 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:05:30 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:05:30 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:05:30 ERROR iSCSI Service exception dropping old sessions 03/02/2021 08:05:35 INFO Checking backend latencies : 03/02/2021 08:05:35 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:05:35 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:05:35 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:05:36 INFO Checking backend latencies : 03/02/2021 08:05:36 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:05:36 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:05:36 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:05:36 INFO Checking backend latencies : 03/02/2021 08:05:36 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:05:36 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:05:36 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:05:42 INFO Checking backend latencies : 03/02/2021 08:05:42 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:05:42 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:05:42 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:05:49 INFO Checking backend latencies : 03/02/2021 08:05:49 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:05:49 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:05:49 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:05:51 INFO GlusterFS mount attempt 03/02/2021 08:06:00 INFO Checking backend latencies : 03/02/2021 08:06:00 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:06:00 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:06:00 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:06:02 INFO Checking backend latencies : 03/02/2021 08:06:02 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:06:02 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:06:02 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:06:07 INFO Checking backend latencies : 03/02/2021 08:06:07 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:06:07 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:06:07 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:06:13 INFO Checking backend latencies : 03/02/2021 08:06:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:06:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:06:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:06:13 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader 03/02/2021 08:06:13 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 516, in get_key index, data = cons.get(key) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:06:20 INFO Checking backend latencies : 03/02/2021 08:06:20 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:06:20 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:06:20 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:06:20 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:06:23 INFO GlusterFS mount attempt 03/02/2021 08:06:26 INFO Checking backend latencies : 03/02/2021 08:06:26 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:06:26 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:06:26 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:06:27 INFO Checking backend latencies : 03/02/2021 08:06:27 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:06:27 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:06:27 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:06:27 ERROR LeaderElectionBase exception dropping old sessions 03/02/2021 08:06:31 INFO Checking backend latencies : 03/02/2021 08:06:31 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:06:31 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:06:31 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:06:32 INFO Checking backend latencies : 03/02/2021 08:06:32 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:06:32 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:06:32 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:06:51 INFO Checking backend latencies : 03/02/2021 08:06:51 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:06:51 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:06:51 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:06:56 INFO Checking backend latencies : 03/02/2021 08:06:56 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:06:56 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:06:56 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:06:57 INFO GlusterFS mount attempt 03/02/2021 08:06:58 INFO Checking backend latencies : 03/02/2021 08:06:58 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:06:58 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:06:58 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:06:59 INFO Checking backend latencies : 03/02/2021 08:06:59 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:06:59 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:06:59 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:07:10 INFO Checking backend latencies : 03/02/2021 08:07:10 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:07:10 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:07:10 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:07:10 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:07:15 INFO Checking backend latencies : 03/02/2021 08:07:15 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:07:15 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:07:15 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:07:24 INFO Checking backend latencies : 03/02/2021 08:07:24 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:07:24 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:07:24 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:07:28 INFO Checking backend latencies : 03/02/2021 08:07:28 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:07:28 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:07:28 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:07:31 INFO GlusterFS mount attempt 03/02/2021 08:07:34 INFO Checking backend latencies : 03/02/2021 08:07:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:07:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:07:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:07:34 INFO Checking backend latencies : 03/02/2021 08:07:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:07:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:07:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:07:34 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:07:35 INFO Checking backend latencies : 03/02/2021 08:07:35 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:07:35 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:07:35 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:07:41 INFO Checking backend latencies : 03/02/2021 08:07:41 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:07:41 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:07:41 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:07:50 INFO Checking backend latencies : 03/02/2021 08:07:50 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:07:50 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:07:50 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:07:58 INFO Checking backend latencies : 03/02/2021 08:07:58 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:07:58 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:07:58 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:08:00 INFO Checking backend latencies : 03/02/2021 08:08:00 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:08:00 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:08:00 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:08:02 INFO GlusterFS mount attempt 03/02/2021 08:08:06 INFO Checking backend latencies : 03/02/2021 08:08:06 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:08:06 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:08:06 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:08:06 ERROR iSCSI Service exception dropping old sessions 03/02/2021 08:08:06 INFO Checking backend latencies : 03/02/2021 08:08:06 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:08:06 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:08:06 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:08:12 INFO Checking backend latencies : 03/02/2021 08:08:12 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:08:12 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:08:12 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:08:20 INFO Checking backend latencies : 03/02/2021 08:08:20 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:08:20 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:08:20 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:08:24 INFO Checking backend latencies : 03/02/2021 08:08:24 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:08:24 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:08:24 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:08:26 INFO Checking backend latencies : 03/02/2021 08:08:26 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:08:26 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:08:26 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:08:30 INFO Checking backend latencies : 03/02/2021 08:08:30 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:08:30 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:08:30 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:08:34 INFO GlusterFS mount attempt 03/02/2021 08:08:43 INFO Checking backend latencies : 03/02/2021 08:08:43 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:08:43 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:08:43 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:08:50 INFO Checking backend latencies : 03/02/2021 08:08:50 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:08:50 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:08:50 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:08:50 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:08:54 INFO Checking backend latencies : 03/02/2021 08:08:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:08:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:08:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:08:56 INFO Checking backend latencies : 03/02/2021 08:08:56 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:08:56 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:08:56 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:08:56 INFO Checking backend latencies : 03/02/2021 08:08:56 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:08:56 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:08:56 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:08:59 INFO Checking backend latencies : 03/02/2021 08:08:59 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:08:59 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:08:59 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:08:59 ERROR LeaderElectionBase exception dropping old sessions 03/02/2021 08:09:08 INFO Checking backend latencies : 03/02/2021 08:09:08 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:09:08 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:09:08 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:09:15 INFO GlusterFS mount attempt 03/02/2021 08:09:21 INFO Checking backend latencies : 03/02/2021 08:09:21 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:09:21 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:09:21 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:09:27 INFO Checking backend latencies : 03/02/2021 08:09:27 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:09:27 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:09:27 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:09:31 INFO Checking backend latencies : 03/02/2021 08:09:31 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:09:31 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:09:31 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:09:33 INFO Checking backend latencies : 03/02/2021 08:09:33 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:09:33 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:09:33 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:09:33 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:09:34 INFO Checking backend latencies : 03/02/2021 08:09:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:09:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:09:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:09:34 INFO Checking backend latencies : 03/02/2021 08:09:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:09:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:09:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:09:34 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader 03/02/2021 08:09:34 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 516, in get_key index, data = cons.get(key) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:09:45 INFO Checking backend latencies : 03/02/2021 08:09:45 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:09:45 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:09:45 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:09:52 INFO GlusterFS mount attempt 03/02/2021 08:09:55 INFO Checking backend latencies : 03/02/2021 08:09:55 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:09:55 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:09:55 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:09:57 INFO Checking backend latencies : 03/02/2021 08:09:57 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:09:57 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:09:57 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:10:04 INFO Checking backend latencies : 03/02/2021 08:10:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:10:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:10:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:10:05 INFO Checking backend latencies : 03/02/2021 08:10:05 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:10:05 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:10:05 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:10:05 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:10:12 INFO Checking backend latencies : 03/02/2021 08:10:12 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:10:12 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:10:12 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:10:21 INFO Checking backend latencies : 03/02/2021 08:10:21 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:10:21 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:10:21 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:10:21 INFO Checking backend latencies : 03/02/2021 08:10:21 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:10:21 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:10:21 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:10:25 INFO GlusterFS mount attempt 03/02/2021 08:10:36 INFO Checking backend latencies : 03/02/2021 08:10:36 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:10:36 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:10:36 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:10:42 INFO Checking backend latencies : 03/02/2021 08:10:42 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:10:42 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:10:42 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:10:42 INFO Checking backend latencies : 03/02/2021 08:10:42 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:10:42 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:10:42 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:10:42 ERROR iSCSI Service exception dropping old sessions 03/02/2021 08:10:47 INFO Checking backend latencies : 03/02/2021 08:10:47 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:10:47 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:10:47 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:10:51 INFO Checking backend latencies : 03/02/2021 08:10:51 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:10:51 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:10:51 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:10:56 INFO Checking backend latencies : 03/02/2021 08:10:56 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:10:56 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:10:56 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:11:00 INFO Checking backend latencies : 03/02/2021 08:11:00 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:11:00 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:11:00 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:11:04 INFO GlusterFS mount attempt 03/02/2021 08:11:17 INFO Checking backend latencies : 03/02/2021 08:11:17 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:11:17 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:11:17 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:11:19 INFO Checking backend latencies : 03/02/2021 08:11:19 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:11:19 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:11:19 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:11:20 INFO Checking backend latencies : 03/02/2021 08:11:20 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:11:20 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:11:20 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:11:20 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:11:20 INFO Checking backend latencies : 03/02/2021 08:11:20 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:11:20 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:11:20 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:11:26 INFO Checking backend latencies : 03/02/2021 08:11:26 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:11:26 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:11:26 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:11:29 INFO Checking backend latencies : 03/02/2021 08:11:29 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:11:29 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:11:29 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:11:29 ERROR LeaderElectionBase exception dropping old sessions 03/02/2021 08:11:37 INFO GlusterFS mount attempt 03/02/2021 08:11:43 INFO Checking backend latencies : 03/02/2021 08:11:43 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:11:43 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:11:43 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:11:47 INFO Checking backend latencies : 03/02/2021 08:11:47 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:11:47 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:11:47 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:11:51 INFO Checking backend latencies : 03/02/2021 08:11:51 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:11:51 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:11:51 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:11:56 INFO Checking backend latencies : 03/02/2021 08:11:56 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:11:56 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:11:56 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:11:56 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:11:56 INFO Checking backend latencies : 03/02/2021 08:11:56 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:11:56 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:11:56 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:12:01 INFO Checking backend latencies : 03/02/2021 08:12:01 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:12:01 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:12:01 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:12:09 INFO GlusterFS mount attempt 03/02/2021 08:12:10 INFO Checking backend latencies : 03/02/2021 08:12:10 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:12:10 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:12:10 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:12:16 INFO Checking backend latencies : 03/02/2021 08:12:16 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:12:16 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:12:16 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:12:17 INFO Checking backend latencies : 03/02/2021 08:12:17 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:12:17 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:12:17 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:12:20 INFO Checking backend latencies : 03/02/2021 08:12:20 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:12:20 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:12:20 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:12:25 INFO Checking backend latencies : 03/02/2021 08:12:25 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:12:25 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:12:25 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:12:35 INFO Checking backend latencies : 03/02/2021 08:12:35 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:12:35 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:12:35 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:12:35 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:12:40 INFO Checking backend latencies : 03/02/2021 08:12:40 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:12:40 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:12:40 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:12:42 INFO Checking backend latencies : 03/02/2021 08:12:42 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:12:42 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:12:42 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:12:43 INFO GlusterFS mount attempt 03/02/2021 08:12:44 INFO Checking backend latencies : 03/02/2021 08:12:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:12:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:12:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:12:52 INFO Checking backend latencies : 03/02/2021 08:12:52 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:12:52 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:12:52 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:12:55 INFO Checking backend latencies : 03/02/2021 08:12:55 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:12:55 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:12:55 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:12:55 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader 03/02/2021 08:12:55 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 516, in get_key index, data = cons.get(key) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:13:06 INFO Checking backend latencies : 03/02/2021 08:13:06 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:13:06 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:13:06 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:13:10 INFO Checking backend latencies : 03/02/2021 08:13:10 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:13:10 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:13:10 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:13:12 INFO Checking backend latencies : 03/02/2021 08:13:12 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:13:12 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:13:12 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:13:15 INFO Checking backend latencies : 03/02/2021 08:13:15 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:13:15 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:13:15 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:13:18 INFO Checking backend latencies : 03/02/2021 08:13:18 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:13:18 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:13:18 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:13:18 ERROR iSCSI Service exception dropping old sessions 03/02/2021 08:13:22 INFO Checking backend latencies : 03/02/2021 08:13:22 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:13:22 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:13:22 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:13:24 INFO GlusterFS mount attempt 03/02/2021 08:13:31 INFO Checking backend latencies : 03/02/2021 08:13:31 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:13:31 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:13:31 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:13:39 INFO Checking backend latencies : 03/02/2021 08:13:39 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:13:39 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:13:39 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:13:41 INFO Checking backend latencies : 03/02/2021 08:13:41 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:13:41 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:13:41 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:13:50 INFO Checking backend latencies : 03/02/2021 08:13:50 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:13:50 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:13:50 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:13:50 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:13:55 INFO Checking backend latencies : 03/02/2021 08:13:55 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:13:55 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:13:55 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:13:56 INFO GlusterFS mount attempt 03/02/2021 08:13:57 INFO Checking backend latencies : 03/02/2021 08:13:57 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:13:57 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:13:57 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:14:00 INFO Checking backend latencies : 03/02/2021 08:14:00 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:14:00 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:14:00 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:14:00 ERROR LeaderElectionBase exception dropping old sessions 03/02/2021 08:14:05 INFO Checking backend latencies : 03/02/2021 08:14:05 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:14:05 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:14:05 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:14:06 INFO Checking backend latencies : 03/02/2021 08:14:06 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:14:06 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:14:06 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:14:17 INFO Checking backend latencies : 03/02/2021 08:14:17 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:14:17 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:14:17 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:14:19 INFO Checking backend latencies : 03/02/2021 08:14:19 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:14:19 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:14:19 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:14:19 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:14:19 INFO Checking backend latencies : 03/02/2021 08:14:19 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:14:19 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:14:19 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:14:27 INFO Checking backend latencies : 03/02/2021 08:14:27 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:14:27 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:14:27 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:14:31 INFO Checking backend latencies : 03/02/2021 08:14:31 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:14:31 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:14:31 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:14:32 INFO Checking backend latencies : 03/02/2021 08:14:32 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:14:32 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:14:32 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:14:35 INFO Checking backend latencies : 03/02/2021 08:14:35 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:14:35 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:14:35 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:14:37 INFO GlusterFS mount attempt 03/02/2021 08:14:41 INFO Checking backend latencies : 03/02/2021 08:14:41 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:14:41 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:14:41 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:14:43 INFO Checking backend latencies : 03/02/2021 08:14:43 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:14:43 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:14:43 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:14:45 INFO Checking backend latencies : 03/02/2021 08:14:45 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:14:45 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:14:45 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:14:56 INFO Checking backend latencies : 03/02/2021 08:14:56 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:14:56 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:14:56 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:14:57 INFO Checking backend latencies : 03/02/2021 08:14:57 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:14:57 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:14:57 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:15:05 INFO Checking backend latencies : 03/02/2021 08:15:05 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:15:05 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:15:05 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:15:05 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:15:07 INFO Checking backend latencies : 03/02/2021 08:15:07 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:15:07 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:15:07 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:15:07 INFO Checking backend latencies : 03/02/2021 08:15:07 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:15:07 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:15:07 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:15:13 INFO Checking backend latencies : 03/02/2021 08:15:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:15:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:15:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:15:13 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:15:16 INFO Checking backend latencies : 03/02/2021 08:15:16 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:15:16 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:15:16 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:15:18 INFO GlusterFS mount attempt 03/02/2021 08:15:21 INFO Checking backend latencies : 03/02/2021 08:15:21 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:15:21 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:15:21 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:15:23 INFO Checking backend latencies : 03/02/2021 08:15:23 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:15:23 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:15:23 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:15:27 INFO Checking backend latencies : 03/02/2021 08:15:27 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:15:27 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:15:27 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:15:33 INFO Checking backend latencies : 03/02/2021 08:15:33 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:15:33 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:15:33 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:15:38 INFO Checking backend latencies : 03/02/2021 08:15:38 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:15:38 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:15:38 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:15:45 INFO Checking backend latencies : 03/02/2021 08:15:45 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:15:45 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:15:45 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:15:53 INFO Checking backend latencies : 03/02/2021 08:15:53 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:15:53 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:15:53 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:15:54 INFO Checking backend latencies : 03/02/2021 08:15:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:15:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:15:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:15:54 ERROR iSCSI Service exception dropping old sessions 03/02/2021 08:15:54 INFO GlusterFS mount attempt 03/02/2021 08:16:04 INFO Checking backend latencies : 03/02/2021 08:16:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:16:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:16:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:16:05 INFO Checking backend latencies : 03/02/2021 08:16:05 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:16:05 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:16:05 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:16:05 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:16:12 INFO Checking backend latencies : 03/02/2021 08:16:12 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:16:12 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:16:12 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:16:16 INFO Checking backend latencies : 03/02/2021 08:16:16 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:16:16 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:16:16 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:16:16 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader 03/02/2021 08:16:16 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 516, in get_key index, data = cons.get(key) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:16:28 INFO GlusterFS mount attempt 03/02/2021 08:16:31 INFO Checking backend latencies : 03/02/2021 08:16:31 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:16:31 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:16:31 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:16:31 ERROR LeaderElectionBase exception dropping old sessions 03/02/2021 08:16:31 INFO Checking backend latencies : 03/02/2021 08:16:31 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:16:31 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:16:31 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:16:36 INFO Checking backend latencies : 03/02/2021 08:16:36 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:16:36 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:16:36 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:16:42 INFO Checking backend latencies : 03/02/2021 08:16:42 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:16:42 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:16:42 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:16:42 INFO Checking backend latencies : 03/02/2021 08:16:42 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:16:42 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:16:42 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:16:42 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:16:55 INFO Checking backend latencies : 03/02/2021 08:16:55 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:16:55 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:16:55 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:16:59 INFO GlusterFS mount attempt 03/02/2021 08:17:00 INFO Checking backend latencies : 03/02/2021 08:17:00 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:17:00 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:17:00 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:17:03 INFO Checking backend latencies : 03/02/2021 08:17:03 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:17:03 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:17:03 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:17:06 INFO Checking backend latencies : 03/02/2021 08:17:06 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:17:06 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:17:06 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:17:20 INFO Checking backend latencies : 03/02/2021 08:17:20 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:17:20 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:17:20 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:17:20 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:17:21 INFO Checking backend latencies : 03/02/2021 08:17:21 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:17:21 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:17:21 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:17:27 INFO Checking backend latencies : 03/02/2021 08:17:27 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:17:27 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:17:27 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:17:27 INFO Checking backend latencies : 03/02/2021 08:17:27 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:17:27 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:17:27 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:17:30 INFO Checking backend latencies : 03/02/2021 08:17:30 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:17:30 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:17:30 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:17:33 INFO GlusterFS mount attempt 03/02/2021 08:17:38 INFO Checking backend latencies : 03/02/2021 08:17:38 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:17:38 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:17:38 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:17:51 INFO Checking backend latencies : 03/02/2021 08:17:51 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:17:51 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:17:51 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:17:51 INFO Checking backend latencies : 03/02/2021 08:17:51 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:17:51 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:17:51 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:17:53 INFO Checking backend latencies : 03/02/2021 08:17:53 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:17:53 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:17:53 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:17:56 INFO Checking backend latencies : 03/02/2021 08:17:56 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:17:56 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:17:56 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:17:57 INFO Checking backend latencies : 03/02/2021 08:17:57 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:17:57 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:17:57 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:18:02 INFO Checking backend latencies : 03/02/2021 08:18:02 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:18:02 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:18:02 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:18:07 INFO GlusterFS mount attempt 03/02/2021 08:18:15 INFO Checking backend latencies : 03/02/2021 08:18:15 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:18:15 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:18:15 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:18:24 INFO Checking backend latencies : 03/02/2021 08:18:24 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:18:24 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:18:24 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:18:26 INFO Checking backend latencies : 03/02/2021 08:18:26 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:18:26 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:18:26 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:18:29 INFO Checking backend latencies : 03/02/2021 08:18:29 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:18:29 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:18:29 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:18:30 INFO Checking backend latencies : 03/02/2021 08:18:30 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:18:30 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:18:30 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:18:30 ERROR iSCSI Service exception dropping old sessions 03/02/2021 08:18:35 INFO Checking backend latencies : 03/02/2021 08:18:35 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:18:35 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:18:35 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:18:35 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:18:40 INFO GlusterFS mount attempt 03/02/2021 08:18:42 INFO Checking backend latencies : 03/02/2021 08:18:42 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:18:42 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:18:42 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:18:59 INFO Checking backend latencies : 03/02/2021 08:18:59 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:18:59 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:18:59 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:19:02 INFO Checking backend latencies : 03/02/2021 08:19:02 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:19:02 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:19:02 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:19:02 ERROR LeaderElectionBase exception dropping old sessions 03/02/2021 08:19:04 INFO Checking backend latencies : 03/02/2021 08:19:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:19:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:19:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:19:04 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:19:06 INFO Checking backend latencies : 03/02/2021 08:19:06 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:19:06 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:19:06 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:19:07 INFO Checking backend latencies : 03/02/2021 08:19:07 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:19:07 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:19:07 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:19:12 INFO Checking backend latencies : 03/02/2021 08:19:12 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:19:12 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:19:12 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:19:13 INFO GlusterFS mount attempt 03/02/2021 08:19:29 INFO Checking backend latencies : 03/02/2021 08:19:29 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:19:29 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:19:29 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:19:31 INFO Checking backend latencies : 03/02/2021 08:19:31 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:19:31 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:19:31 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:19:31 INFO Checking backend latencies : 03/02/2021 08:19:31 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:19:31 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:19:31 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:19:34 INFO Checking backend latencies : 03/02/2021 08:19:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:19:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:19:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:19:37 INFO Checking backend latencies : 03/02/2021 08:19:37 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:19:37 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:19:37 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:19:37 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader 03/02/2021 08:19:37 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 516, in get_key index, data = cons.get(key) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:19:48 INFO GlusterFS mount attempt 03/02/2021 08:19:50 INFO Checking backend latencies : 03/02/2021 08:19:50 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:19:50 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:19:50 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:19:50 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:19:53 INFO Checking backend latencies : 03/02/2021 08:19:53 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:19:53 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:19:53 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:19:57 INFO Checking backend latencies : 03/02/2021 08:19:57 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:19:57 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:19:57 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:19:57 INFO Checking backend latencies : 03/02/2021 08:19:57 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:19:57 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:19:57 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:19:58 INFO Checking backend latencies : 03/02/2021 08:19:58 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:19:58 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:19:58 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:20:19 INFO Checking backend latencies : 03/02/2021 08:20:19 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:20:19 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:20:19 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:20:21 INFO Checking backend latencies : 03/02/2021 08:20:21 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:20:21 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:20:21 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:20:24 INFO Checking backend latencies : 03/02/2021 08:20:24 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:20:24 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:20:24 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:20:27 INFO Checking backend latencies : 03/02/2021 08:20:27 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:20:27 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:20:27 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:20:27 INFO Checking backend latencies : 03/02/2021 08:20:27 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:20:27 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:20:27 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:20:28 INFO GlusterFS mount attempt 03/02/2021 08:20:46 INFO Checking backend latencies : 03/02/2021 08:20:46 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:20:46 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:20:46 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:20:49 INFO Checking backend latencies : 03/02/2021 08:20:49 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:20:49 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:20:49 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:20:54 INFO Checking backend latencies : 03/02/2021 08:20:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:20:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:20:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:20:59 INFO Checking backend latencies : 03/02/2021 08:20:59 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:20:59 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:20:59 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:21:02 INFO GlusterFS mount attempt 03/02/2021 08:21:05 INFO Checking backend latencies : 03/02/2021 08:21:05 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:21:05 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:21:05 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:21:05 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:21:05 INFO Checking backend latencies : 03/02/2021 08:21:05 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:21:05 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:21:05 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:21:05 ERROR iSCSI Service exception dropping old sessions 03/02/2021 08:21:12 INFO Checking backend latencies : 03/02/2021 08:21:12 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:21:12 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:21:12 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:21:21 INFO Checking backend latencies : 03/02/2021 08:21:21 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:21:21 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:21:21 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:21:23 INFO Checking backend latencies : 03/02/2021 08:21:23 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:21:23 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:21:23 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:21:27 INFO Checking backend latencies : 03/02/2021 08:21:27 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:21:27 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:21:27 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:21:27 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:21:32 INFO Checking backend latencies : 03/02/2021 08:21:32 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:21:32 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:21:32 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:21:32 ERROR LeaderElectionBase exception dropping old sessions 03/02/2021 08:21:33 INFO GlusterFS mount attempt 03/02/2021 08:21:42 INFO Checking backend latencies : 03/02/2021 08:21:42 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:21:42 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:21:42 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:21:42 INFO Checking backend latencies : 03/02/2021 08:21:42 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:21:42 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:21:42 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:21:46 INFO Checking backend latencies : 03/02/2021 08:21:46 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:21:46 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:21:46 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:21:50 INFO Checking backend latencies : 03/02/2021 08:21:50 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:21:50 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:21:50 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:21:51 INFO Checking backend latencies : 03/02/2021 08:21:51 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:21:51 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:21:51 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:22:05 INFO Checking backend latencies : 03/02/2021 08:22:05 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:22:05 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:22:05 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:22:05 INFO GlusterFS mount attempt 03/02/2021 08:22:07 INFO Checking backend latencies : 03/02/2021 08:22:07 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:22:07 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:22:07 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:22:12 INFO Checking backend latencies : 03/02/2021 08:22:12 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:22:12 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:22:12 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:22:15 INFO Checking backend latencies : 03/02/2021 08:22:15 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:22:15 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:22:15 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:22:20 INFO Checking backend latencies : 03/02/2021 08:22:20 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:22:20 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:22:20 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:22:20 INFO Checking backend latencies : 03/02/2021 08:22:20 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:22:20 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:22:20 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:22:20 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:22:29 INFO Checking backend latencies : 03/02/2021 08:22:29 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:22:29 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:22:29 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:22:33 INFO Checking backend latencies : 03/02/2021 08:22:33 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:22:33 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:22:33 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:22:37 INFO Checking backend latencies : 03/02/2021 08:22:37 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:22:37 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:22:37 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:22:37 INFO GlusterFS mount attempt 03/02/2021 08:22:38 INFO Checking backend latencies : 03/02/2021 08:22:38 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:22:38 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:22:38 INFO Network latency for backend 172.16.0.153 = 58.6 us 03/02/2021 08:22:40 INFO Checking backend latencies : 03/02/2021 08:22:40 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:22:40 INFO Network latency for backend 172.16.0.152 = 51.2 us 03/02/2021 08:22:40 INFO Network latency for backend 172.16.0.153 = 54.0 us 03/02/2021 08:22:41 INFO iSCSI Service successfully dropped old sessions 03/02/2021 08:22:41 INFO Cleaning unused configurations. 03/02/2021 08:22:41 INFO Cleaning all mapped disks 03/02/2021 08:22:41 INFO Cleaning unused rbd images. 03/02/2021 08:22:41 INFO Cleaning unused ips. 03/02/2021 08:22:44 ERROR Node Stats exception. 03/02/2021 08:22:44 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 23.09 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 7.37 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 3553.28 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 4956.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.02 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 21790.72 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 19322.88 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.01 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 18739.2 `date +%s`" | nc -q0 10.10.0.153 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 23.09 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 7.37 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 3553.28 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 4956.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.02 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 21790.72 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 19322.88 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.01 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 18739.2 `date +%s`" | nc -q0 10.10.0.153 2003 03/02/2021 08:22:49 INFO Checking backend latencies : 03/02/2021 08:22:49 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:22:49 INFO Network latency for backend 172.16.0.152 = 56.8 us 03/02/2021 08:22:49 INFO Network latency for backend 172.16.0.153 = 65.2 us 03/02/2021 08:22:57 INFO LeaderElectionBase successfully dropped old sessions 03/02/2021 08:23:00 INFO Found pool:rbd for disk:00001 via consul 03/02/2021 08:23:00 INFO Image image-00001 mapped successfully. 03/02/2021 08:23:02 INFO ClusterLeader start action 03/02/2021 08:23:02 INFO ClusterLeader starting services 03/02/2021 08:23:03 INFO LIO add_target() disk wwn is c74c6e0600001 03/02/2021 08:23:03 INFO Path 00001/3 acquired successfully 03/02/2021 08:23:23 INFO Checking backend latencies : 03/02/2021 08:23:23 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:23:23 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:23:23 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:23:23 INFO Checking backend latencies : 03/02/2021 08:23:23 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:23:23 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:23:23 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:23:24 INFO Checking backend latencies : 03/02/2021 08:23:24 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:23:24 INFO Network latency for backend 172.16.0.152 = 50.2 us 03/02/2021 08:23:24 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:23:36 INFO Checking backend latencies : 03/02/2021 08:23:36 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:23:36 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:23:36 INFO Network latency for backend 172.16.0.153 = 03/02/2021 08:23:41 INFO Found pool:rbd for disk:00001 via consul 03/02/2021 08:23:42 INFO Could not lock path 00001/2 with session 1841a6bf-4482-aa78-6225-469a2c59de79. 03/02/2021 08:24:05 ERROR Node Stats exception. 03/02/2021 08:24:05 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 0.51 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 14.24 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1320.96 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 419.84 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.03 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 40857.6 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 37253.12 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.03 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 37939.2 `date +%s`" | nc -q0 10.10.0.153 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 0.51 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 14.24 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1320.96 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 419.84 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.03 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 40857.6 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 37253.12 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.03 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 37939.2 `date +%s`" | nc -q0 10.10.0.153 2003 03/02/2021 10:28:20 INFO call get assignments stats function. 03/02/2021 10:28:30 INFO call get assignments stats function. 03/02/2021 11:15:21 INFO Start add osd job 9840 03/02/2021 11:16:26 INFO Start add osd job 12527 03/02/2021 11:34:20 ERROR Node Stats exception. 03/02/2021 11:34:20 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.52 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1699.84 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 51.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.18 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 155125.76 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 224296.96 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.17 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 144721.92 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.52 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1699.84 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 51.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.18 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 155125.76 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 224296.96 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.17 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 144721.92 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:35:23 ERROR Node Stats exception. 03/02/2021 11:35:23 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 2.35 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.5 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1024.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 100782.08 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 256215.04 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.19 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 93296.64 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 2.35 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.5 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1024.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 100782.08 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 256215.04 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.19 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 93296.64 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:36:27 ERROR Node Stats exception. 03/02/2021 11:36:27 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.67 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.49 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1044.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 51.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.22 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 135239.68 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 279736.32 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.21 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 125061.12 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.67 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.49 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1044.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 51.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.22 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 135239.68 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 279736.32 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.21 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 125061.12 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:37:30 ERROR Node Stats exception. 03/02/2021 11:37:30 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 4.84 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.63 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1146.88 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 20.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 143697.92 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 201984.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.15 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 134195.2 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 4.84 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.63 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1146.88 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 20.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 143697.92 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 201984.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.15 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 134195.2 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:38:33 ERROR Node Stats exception. 03/02/2021 11:38:33 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 6.34 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.53 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1208.32 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 235.52 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.22 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 164218.88 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 278282.24 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.21 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 153927.68 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 6.34 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.53 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1208.32 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 235.52 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.22 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 164218.88 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 278282.24 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.21 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 153927.68 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:39:36 ERROR Node Stats exception. 03/02/2021 11:39:36 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.67 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.49 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1146.88 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.24 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 155392.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 302295.04 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.23 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 144424.96 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.67 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.49 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1146.88 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.24 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 155392.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 302295.04 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.23 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 144424.96 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:40:39 ERROR Node Stats exception. 03/02/2021 11:40:39 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.17 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.52 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 880.64 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.12 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 79042.56 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 149821.44 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.11 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 73113.6 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.17 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.52 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 880.64 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.12 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 79042.56 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 149821.44 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.11 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 73113.6 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:41:43 ERROR Node Stats exception. 03/02/2021 11:41:43 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 2.17 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.59 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 768.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 20.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.22 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 143493.12 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 276961.28 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.21 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 133744.64 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 2.17 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.59 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 768.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 20.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.22 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 143493.12 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 276961.28 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.21 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 133744.64 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:42:47 ERROR Node Stats exception. 03/02/2021 11:42:47 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.disks.smart.sda.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sda.power_on_hours 84501 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sda.temperature_celsius 34 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.power_on_hours 84500 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.temperature_celsius 33 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.power_on_hours 84501 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.temperature_celsius 34 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdd.reallocated_sector_ct 0 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.disks.smart.sda.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sda.power_on_hours 84501 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sda.temperature_celsius 34 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.power_on_hours 84500 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.temperature_celsius 33 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.power_on_hours 84501 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.temperature_celsius 34 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdd.reallocated_sector_ct 0 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:43:51 ERROR Node Stats exception. 03/02/2021 11:43:51 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.17 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.59 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 839.68 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.15 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 122992.64 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 191744.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.15 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 114493.44 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.17 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.59 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 839.68 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.15 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 122992.64 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 191744.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.15 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 114493.44 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:44:54 ERROR Node Stats exception. 03/02/2021 11:44:54 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.5 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.56 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1515.52 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 122.88 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.22 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 161474.56 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 273285.12 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.21 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 149719.04 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.5 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.56 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1515.52 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 122.88 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.22 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 161474.56 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 273285.12 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.21 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 149719.04 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:45:57 ERROR Node Stats exception. 03/02/2021 11:45:57 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 7.21 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.6 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1085.44 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 20.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.21 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 123279.36 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 263782.4 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 114493.44 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 7.21 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.6 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1085.44 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 20.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.21 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 123279.36 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 263782.4 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 114493.44 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:47:00 ERROR Node Stats exception. 03/02/2021 11:47:00 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 4.5 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.66 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1392.64 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 61.44 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.14 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 103905.28 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 169359.36 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.13 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 96286.72 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 4.5 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.66 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1392.64 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 61.44 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.14 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 103905.28 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 169359.36 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.13 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 96286.72 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:48:03 ERROR Node Stats exception. 03/02/2021 11:48:03 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 5.19 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.58 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1146.88 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 235.52 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.22 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 150210.56 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 271288.32 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.21 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 140165.12 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 5.19 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.58 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1146.88 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 235.52 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.22 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 150210.56 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 271288.32 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.21 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 140165.12 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:49:07 ERROR Node Stats exception. 03/02/2021 11:49:07 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.33 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.6 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 911.36 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 40.96 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.22 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 123924.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 277852.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.21 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 115558.4 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.33 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.6 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 911.36 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 40.96 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.22 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 123924.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 277852.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.21 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 115558.4 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:50:02 INFO backfill_speed choice : slow 03/02/2021 11:50:03 INFO Done changing the backfill speed parameter (osd_max_backfills) to 1 03/02/2021 11:50:03 INFO Done changing the backfill speed parameter (osd_recovery_max_active) to 1 03/02/2021 11:50:04 INFO Done changing the backfill speed parameter (osd_recovery_sleep) to 0.3 03/02/2021 11:50:10 ERROR Node Stats exception. 03/02/2021 11:50:10 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.83 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.71 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1126.4 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 71.68 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 129269.76 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 201103.36 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.15 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 120483.84 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.83 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.71 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1126.4 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 71.68 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 129269.76 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 201103.36 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.15 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 120483.84 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:51:13 ERROR Node Stats exception. 03/02/2021 11:51:13 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.5 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.82 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1044.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 61.44 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.23 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 149903.36 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 284231.68 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.22 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 139888.64 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.5 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.82 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1044.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 61.44 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.23 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 149903.36 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 284231.68 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.22 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 139888.64 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:52:16 ERROR Node Stats exception. 03/02/2021 11:52:16 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 3.33 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.82 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1218.56 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 3676.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.24 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 167464.96 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 302684.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.23 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 156477.44 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 3.33 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.82 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1218.56 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 3676.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.24 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 167464.96 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 302684.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.23 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 156477.44 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:53:21 ERROR Node Stats exception. 03/02/2021 11:53:21 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.disks.smart.sda.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sda.power_on_hours 84501 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sda.temperature_celsius 34 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.power_on_hours 84500 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.temperature_celsius 33 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.power_on_hours 84501 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.temperature_celsius 33 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdd.reallocated_sector_ct 0 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.disks.smart.sda.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sda.power_on_hours 84501 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sda.temperature_celsius 34 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.power_on_hours 84500 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.temperature_celsius 33 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.power_on_hours 84501 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.temperature_celsius 33 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdd.reallocated_sector_ct 0 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:54:24 ERROR Node Stats exception. 03/02/2021 11:54:24 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 11.21 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.86 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1239.04 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.14 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 156436.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 179394.56 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.14 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 146513.92 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 11.21 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.86 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1239.04 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.14 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 156436.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 179394.56 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.14 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 146513.92 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:55:27 ERROR Node Stats exception. 03/02/2021 11:55:27 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.84 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.88 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1617.92 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.21 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 105359.36 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 261273.6 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 97925.12 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.84 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.88 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1617.92 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.21 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 105359.36 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 261273.6 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 97925.12 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:56:31 ERROR Node Stats exception. 03/02/2021 11:56:31 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.17 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.85 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 880.64 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.17 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 142131.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 211537.92 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 132597.76 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.17 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.85 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 880.64 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.17 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 142131.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 211537.92 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 132597.76 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:57:34 ERROR Node Stats exception. 03/02/2021 11:57:34 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 2.66 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.85 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 727.04 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 136161.28 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 249937.92 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.19 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 126545.92 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 2.66 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.85 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 727.04 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 136161.28 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 249937.92 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.19 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 126545.92 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:58:37 ERROR Node Stats exception. 03/02/2021 11:58:37 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 17.52 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.89 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 778.24 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 20.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.21 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 206551.04 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 268492.8 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 193730.56 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 17.52 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.89 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 778.24 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 20.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.21 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 206551.04 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 268492.8 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 193730.56 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:59:40 ERROR Node Stats exception. 03/02/2021 11:59:40 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 2.17 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.92 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1075.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 20.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.17 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 159016.96 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 214159.36 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 149022.72 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 2.17 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.92 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1075.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 20.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.17 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 159016.96 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 214159.36 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 149022.72 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:00:43 ERROR Node Stats exception. 03/02/2021 12:00:43 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.67 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.94 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1218.56 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 81.92 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.24 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 162293.76 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 295270.4 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.22 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 151377.92 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.67 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.94 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1218.56 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 81.92 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.24 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 162293.76 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 295270.4 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.22 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 151377.92 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:01:46 ERROR Node Stats exception. 03/02/2021 12:01:46 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.5 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.94 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1576.96 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 81.92 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.24 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 148930.56 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 296007.68 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.22 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 138403.84 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.5 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.94 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1576.96 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 81.92 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.24 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 148930.56 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 296007.68 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.22 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 138403.84 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:02:50 ERROR Node Stats exception. 03/02/2021 12:02:50 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.17 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.95 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 993.28 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 20.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.17 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 131768.32 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 207370.24 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 122777.6 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.17 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.95 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 993.28 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 20.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.17 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 131768.32 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 207370.24 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 122777.6 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:03:54 ERROR Node Stats exception. 03/02/2021 12:03:54 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.disks.smart.sda.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sda.power_on_hours 84501 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sda.temperature_celsius 34 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.power_on_hours 84500 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.temperature_celsius 33 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.power_on_hours 84501 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.temperature_celsius 34 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdd.reallocated_sector_ct 0 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.disks.smart.sda.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sda.power_on_hours 84501 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sda.temperature_celsius 34 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.power_on_hours 84500 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.temperature_celsius 33 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.power_on_hours 84501 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.temperature_celsius 34 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdd.reallocated_sector_ct 0 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:04:58 ERROR Node Stats exception. 03/02/2021 12:04:58 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 7.32 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.97 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1157.12 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 81.92 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.23 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 144773.12 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 288665.6 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.22 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 134778.88 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 7.32 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.97 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1157.12 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 81.92 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.23 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 144773.12 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 288665.6 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.22 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 134778.88 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:06:01 ERROR Node Stats exception. 03/02/2021 12:06:01 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 2.17 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.95 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 3123.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 471.04 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.21 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 172656.64 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 263096.32 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 161730.56 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 2.17 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.95 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 3123.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 471.04 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.21 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 172656.64 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 263096.32 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 161730.56 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:07:04 ERROR Node Stats exception. 03/02/2021 12:07:04 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 2.68 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.95 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 3205.12 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 1423.36 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.19 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 182312.96 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 239462.4 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.18 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 171100.16 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 2.68 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.95 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 3205.12 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 1423.36 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.19 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 182312.96 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 239462.4 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.18 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 171100.16 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:08:07 ERROR Node Stats exception. 03/02/2021 12:08:07 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.5 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.96 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1054.72 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 61.44 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 105553.92 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 255447.04 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.19 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 97761.28 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.5 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.96 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1054.72 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 61.44 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 105553.92 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 255447.04 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.19 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 97761.28 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:09:10 ERROR Node Stats exception. 03/02/2021 12:09:10 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.33 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.97 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 849.92 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.17 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 141096.96 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 210462.72 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 131010.56 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.33 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.97 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 849.92 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.17 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 141096.96 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 210462.72 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 131010.56 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:10:14 ERROR Node Stats exception. 03/02/2021 12:10:14 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.5 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.96 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 737.28 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 153.6 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.22 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 138711.04 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 276039.68 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.21 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 128839.68 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.5 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.96 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 737.28 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 153.6 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.22 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 138711.04 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 276039.68 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.21 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 128839.68 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:11:17 ERROR Node Stats exception. 03/02/2021 12:11:17 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.34 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.02 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 829.44 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 61.44 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.21 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 110991.36 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 257986.56 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 102973.44 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.34 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.02 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 829.44 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 61.44 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.21 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 110991.36 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 257986.56 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 102973.44 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:12:20 ERROR Node Stats exception. 03/02/2021 12:12:20 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 2.17 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.97 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 2068.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 491.52 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.18 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 147548.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 225689.6 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.17 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 137134.08 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 2.17 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.97 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 2068.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 491.52 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.18 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 147548.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 225689.6 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.17 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 137134.08 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:13:23 ERROR Node Stats exception. 03/02/2021 12:13:23 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 2.69 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.99 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 2621.44 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 1423.36 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.21 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 116715.52 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 262256.64 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 107796.48 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 2.69 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.99 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 2621.44 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 1423.36 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.21 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 116715.52 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 262256.64 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 107796.48 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:14:28 ERROR Node Stats exception. 03/02/2021 12:14:28 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.disks.smart.sda.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sda.power_on_hours 84501 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sda.temperature_celsius 35 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.power_on_hours 84501 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.temperature_celsius 33 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.power_on_hours 84501 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.temperature_celsius 34 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdd.reallocated_sector_ct 0 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.disks.smart.sda.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sda.power_on_hours 84501 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sda.temperature_celsius 35 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.power_on_hours 84501 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.temperature_celsius 33 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.power_on_hours 84501 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.temperature_celsius 34 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdd.reallocated_sector_ct 0 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:15:31 ERROR Node Stats exception. 03/02/2021 12:15:31 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 2.5 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.99 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1136.64 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.23 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 146718.72 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 293580.8 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.22 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 136540.16 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 2.5 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.99 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1136.64 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.23 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 146718.72 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 293580.8 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.22 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 136540.16 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:16:34 ERROR Node Stats exception. 03/02/2021 12:16:34 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 2.33 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.98 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 839.68 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 92.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.15 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 126709.76 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 181555.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.14 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 117493.76 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 2.33 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 19.98 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 839.68 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 92.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.15 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 126709.76 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 181555.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.14 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 117493.76 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:17:38 ERROR Node Stats exception. 03/02/2021 12:17:38 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.83 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.05 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1730.56 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 103895.04 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 250726.4 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.19 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 96327.68 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.83 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.05 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1730.56 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 103895.04 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 250726.4 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.19 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 96327.68 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:18:41 ERROR Node Stats exception. 03/02/2021 12:18:41 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 2.0 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.05 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 2478.08 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 573.44 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.18 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 169420.8 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 222013.44 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.17 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 158115.84 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 2.0 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.05 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 2478.08 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 573.44 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.18 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 169420.8 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 222013.44 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.17 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 158115.84 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:19:44 ERROR Node Stats exception. 03/02/2021 12:19:44 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 3.83 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.09 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 2396.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 1423.36 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 130887.68 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 245688.32 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.19 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 122009.6 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 3.83 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.09 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 2396.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 1423.36 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 130887.68 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 245688.32 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.19 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 122009.6 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:20:47 ERROR Node Stats exception. 03/02/2021 12:20:47 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.82 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.07 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 972.8 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.21 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 110909.44 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 258191.36 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 102625.28 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.82 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.07 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 972.8 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.21 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 110909.44 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 258191.36 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 102625.28 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:21:50 ERROR Node Stats exception. 03/02/2021 12:21:50 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.17 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.06 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 962.56 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 61.44 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.19 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 162816.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 236974.08 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.18 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 151808.0 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.17 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.06 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 962.56 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 61.44 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.19 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 162816.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 236974.08 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.18 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 151808.0 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:22:53 ERROR Node Stats exception. 03/02/2021 12:22:53 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 2.0 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.1 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1402.88 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 40.96 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.21 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 131665.92 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 264724.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 122777.6 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 2.0 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.1 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1402.88 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 40.96 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.21 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 131665.92 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 264724.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 122777.6 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:23:05 INFO Start add osd job 69001 03/02/2021 12:23:57 ERROR Node Stats exception. 03/02/2021 12:23:57 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 3.0 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.11 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 665.6 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 92.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.27 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 175544.32 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 331724.8 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.25 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 163563.52 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 3.0 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.11 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 665.6 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 92.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.27 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 175544.32 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 331724.8 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.25 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 163563.52 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:25:00 INFO Start add osd job 71316 03/02/2021 12:25:02 ERROR Node Stats exception. 03/02/2021 12:25:02 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.disks.smart.sda.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sda.power_on_hours 84501 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sda.temperature_celsius 34 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.power_on_hours 84501 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.temperature_celsius 34 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.power_on_hours 84501 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.temperature_celsius 34 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdd.reallocated_sector_ct 0 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.disks.smart.sda.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sda.power_on_hours 84501 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sda.temperature_celsius 34 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.power_on_hours 84501 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.temperature_celsius 34 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.power_on_hours 84501 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.temperature_celsius 34 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdd.reallocated_sector_ct 0 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:26:05 ERROR Node Stats exception. 03/02/2021 12:26:05 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.33 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.09 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 911.36 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.17 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 163143.68 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 214251.52 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 152268.8 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.33 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.09 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 911.36 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.17 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 163143.68 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 214251.52 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 152268.8 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:27:08 ERROR Node Stats exception. 03/02/2021 12:27:08 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 2.01 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.12 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 2109.44 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 552.96 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.25 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 138936.32 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 310251.52 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.24 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 128368.64 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 2.01 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.12 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 2109.44 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 552.96 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.25 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 138936.32 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 310251.52 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.24 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 128368.64 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:28:02 INFO Start add osd job 75527 03/02/2021 12:28:11 ERROR Node Stats exception. 03/02/2021 12:28:11 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.33 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.1 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1423.36 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.24 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 146083.84 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 294389.76 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.22 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 135587.84 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.33 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.1 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1423.36 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.24 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 146083.84 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 294389.76 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.22 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 135587.84 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:29:14 ERROR Node Stats exception. 03/02/2021 12:29:14 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 4.17 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.14 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 4147.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 2949.12 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 121466.88 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 199700.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.15 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 113315.84 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 4.17 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.14 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 4147.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 2949.12 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 121466.88 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 199700.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.15 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 113315.84 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:29:18 INFO Start add osd job 78456 03/02/2021 12:30:17 ERROR Node Stats exception. 03/02/2021 12:30:17 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.83 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.11 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 921.6 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.21 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 117166.08 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 264058.88 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 108503.04 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.83 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.11 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 921.6 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.21 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 117166.08 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 264058.88 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 108503.04 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:31:21 ERROR Node Stats exception. 03/02/2021 12:31:21 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 3.5 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.13 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 3911.68 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 1884.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.18 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 163942.4 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 220426.24 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.17 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 152340.48 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 3.5 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.13 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 3911.68 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 1884.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.18 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 163942.4 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 220426.24 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.17 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 152340.48 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:32:24 ERROR Node Stats exception. 03/02/2021 12:32:24 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 17.82 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.13 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 890.88 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.21 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 234926.08 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 257863.68 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 220876.8 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 17.82 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.13 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 890.88 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.21 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 234926.08 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 257863.68 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 220876.8 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:33:27 ERROR Node Stats exception. 03/02/2021 12:33:27 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.5 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.11 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 808.96 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.23 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 136284.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 286494.72 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.22 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 126771.2 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.5 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.11 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 808.96 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.23 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 136284.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 286494.72 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.22 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 126771.2 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:34:30 ERROR Node Stats exception. 03/02/2021 12:34:30 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 14.38 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.14 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 2498.56 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 1740.8 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 139468.8 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 248309.76 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.19 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 129546.24 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 14.38 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.14 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 2498.56 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 1740.8 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 139468.8 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 248309.76 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.19 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 129546.24 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:34:31 INFO Start add osd job 86425 03/02/2021 12:35:35 ERROR Node Stats exception. 03/02/2021 12:35:35 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.disks.smart.sda.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sda.power_on_hours 84502 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sda.temperature_celsius 35 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.power_on_hours 84501 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.temperature_celsius 33 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.power_on_hours 84502 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.temperature_celsius 34 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdd.reallocated_sector_ct 0 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.disks.smart.sda.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sda.power_on_hours 84502 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sda.temperature_celsius 35 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.power_on_hours 84501 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdb.temperature_celsius 33 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.power_on_hours 84502 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdc.temperature_celsius 34 `date +%s`" " PetaSAN.NodeStats.peta1.disks.smart.sdd.reallocated_sector_ct 0 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:36:38 ERROR Node Stats exception. 03/02/2021 12:36:38 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.34 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1474.56 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 20.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.23 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 130938.88 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 282941.44 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.22 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 121354.24 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.34 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1474.56 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 20.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.23 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 130938.88 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 282941.44 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.22 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 121354.24 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:37:15 INFO Start add osd job 91248 03/02/2021 12:37:41 ERROR Node Stats exception. 03/02/2021 12:37:41 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.01 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.19 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 727.04 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.24 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 162662.4 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 296796.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.23 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 152053.76 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.01 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.19 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 727.04 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.24 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 162662.4 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 296796.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.23 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 152053.76 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:38:45 ERROR Node Stats exception. 03/02/2021 12:38:45 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 2.33 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.21 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1556.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.17 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 172718.08 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 208517.12 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 161474.56 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 2.33 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.21 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1556.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.17 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 172718.08 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 208517.12 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 161474.56 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:39:48 ERROR Node Stats exception. 03/02/2021 12:39:48 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 2.66 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.25 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 2201.6 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 471.04 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 117954.56 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 247900.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.19 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 109414.4 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 2.66 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.25 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 2201.6 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 471.04 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 117954.56 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 247900.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.19 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 109414.4 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:39:56 INFO Start add osd job 3267 03/02/2021 12:40:51 ERROR Node Stats exception. 03/02/2021 12:40:51 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.01 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.31 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1024.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 81.92 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.26 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 179220.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 322785.28 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.25 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 166676.48 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.01 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.31 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1024.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 81.92 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.26 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 179220.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 322785.28 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.25 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 166676.48 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:40:59 INFO Start add osd job 7427 03/02/2021 12:41:54 ERROR Node Stats exception. 03/02/2021 12:41:54 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 3.69 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.45 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1064.96 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 51.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.18 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 211906.56 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 229457.92 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.17 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 198594.56 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 3.69 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.45 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1064.96 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 51.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.18 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 211906.56 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 229457.92 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.17 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 198594.56 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:42:03 INFO Start add osd job 12439 03/02/2021 12:42:57 ERROR Node Stats exception. 03/02/2021 12:42:57 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 2.66 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.51 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1310.72 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 81.92 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.26 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 194938.88 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 318904.32 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.24 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 182097.92 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 2.66 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.51 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1310.72 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 81.92 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.26 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 194938.88 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 318904.32 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.24 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 182097.92 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:43:15 INFO Start add osd job 17442 03/02/2021 12:44:01 ERROR Node Stats exception. 03/02/2021 12:44:01 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 3.33 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.64 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1177.6 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 133.12 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.21 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 188846.08 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 263608.32 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 176445.44 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 3.33 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 20.64 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1177.6 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 133.12 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.21 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 188846.08 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 263608.32 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 176445.44 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:48:32 ERROR 400 Bad Request: The browser (or proxy) sent a request that this server could not understand. 03/02/2021 12:48:32 ERROR 400 Bad Request: The browser (or proxy) sent a request that this server could not understand. 03/02/2021 12:48:32 ERROR 400 Bad Request: The browser (or proxy) sent a request that this server could not understand. 03/02/2021 12:48:32 ERROR 400 Bad Request: The browser (or proxy) sent a request that this server could not understand. 03/02/2021 12:48:51 INFO call get assignments stats function. 03/02/2021 12:49:01 INFO call get assignments stats function. 03/02/2021 12:49:11 INFO call get assignments stats function. 03/02/2021 12:49:21 INFO call get assignments stats function. 03/02/2021 12:49:31 INFO call get assignments stats function. 03/02/2021 12:49:41 INFO call get assignments stats function. 03/02/2021 12:49:52 INFO call get assignments stats function. 03/02/2021 12:50:15 INFO Success saving application config 03/02/2021 12:50:19 INFO Success saving application config 03/02/2021 12:50:19 INFO Success saving application config 03/02/2021 12:53:31 INFO call get assignments stats function. 03/02/2021 12:53:41 INFO call get assignments stats function. 03/02/2021 13:00:31 INFO Successfully created key 00001 for new disk. 03/02/2021 13:00:33 INFO Cleaned disk path 00001/3. 03/02/2021 13:00:33 INFO PetaSAN cleaned local paths not locked by this node in consul. 03/02/2021 13:00:33 INFO Stopping disk 00001 03/02/2021 13:00:33 ERROR Could not find ips for image-00001 03/02/2021 13:00:33 INFO LIO deleted backstore image image-00001 03/02/2021 13:00:33 INFO LIO deleted Target iqn.2016-05.com.petasan:00001 03/02/2021 13:00:33 INFO Image image-00001 unmapped successfully. 03/02/2021 13:00:35 INFO Found pool:rbd for disk:00001 via consul 03/02/2021 13:00:36 INFO PetaSAN removed key of stopped disk 00001 from consul. 03/02/2021 13:01:37 INFO script 03/02/2021 13:01:37 INFO /opt/petasan/scripts/util/delete_iscsi_disk.py 03/02/2021 13:01:37 INFO params 03/02/2021 13:01:37 INFO image-00001 rbd 03/02/2021 13:01:37 INFO Deleting disk: image-00001 has been started as a job 03/02/2021 13:01:37 INFO Start deleting disk: image-00001 ... 03/02/2021 13:01:44 ERROR Cannot get image meta object from rbd header. 03/02/2021 13:01:44 ERROR Get ceph image size image-00001 error 03/02/2021 13:01:44 ERROR [errno 2] error opening image b'image-00001' at snapshot None Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/ceph/api.py", line 1044, in __get_image_size size = int(math.ceil(float(rbd.Image(ioctx, image).size()) / float(1024 ** 3))) File "rbd.pyx", line 2958, in rbd.Image.__init__ rbd.ImageNotFound: [errno 2] error opening image b'image-00001' at snapshot None 03/02/2021 13:01:45 ERROR Cannot get image meta object from rbd header. 03/02/2021 13:01:45 ERROR Get ceph image size image-00001 error 03/02/2021 13:01:45 ERROR [errno 2] error opening image b'image-00001' at snapshot None Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/ceph/api.py", line 1044, in __get_image_size size = int(math.ceil(float(rbd.Image(ioctx, image).size()) / float(1024 ** 3))) File "rbd.pyx", line 2958, in rbd.Image.__init__ rbd.ImageNotFound: [errno 2] error opening image b'image-00001' at snapshot None 03/02/2021 13:02:01 ERROR Cannot get image meta object from rbd header. 03/02/2021 13:02:01 ERROR Get ceph image size image-00001 error 03/02/2021 13:02:01 ERROR [errno 2] error opening image b'image-00001' at snapshot None Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/ceph/api.py", line 1044, in __get_image_size size = int(math.ceil(float(rbd.Image(ioctx, image).size()) / float(1024 ** 3))) File "rbd.pyx", line 2958, in rbd.Image.__init__ rbd.ImageNotFound: [errno 2] error opening image b'image-00001' at snapshot None 03/02/2021 13:02:11 ERROR Cannot get image meta object from rbd header. 03/02/2021 13:02:11 ERROR Get ceph image size image-00001 error 03/02/2021 13:02:11 ERROR [errno 2] error opening image b'image-00001' at snapshot None Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/ceph/api.py", line 1044, in __get_image_size size = int(math.ceil(float(rbd.Image(ioctx, image).size()) / float(1024 ** 3))) File "rbd.pyx", line 2958, in rbd.Image.__init__ rbd.ImageNotFound: [errno 2] error opening image b'image-00001' at snapshot None 03/02/2021 13:02:20 ERROR Cannot get image meta object from rbd header. 03/02/2021 13:02:20 ERROR Get ceph image size image-00001 error 03/02/2021 13:02:20 ERROR [errno 2] error opening image b'image-00001' at snapshot None Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/ceph/api.py", line 1044, in __get_image_size size = int(math.ceil(float(rbd.Image(ioctx, image).size()) / float(1024 ** 3))) File "rbd.pyx", line 2958, in rbd.Image.__init__ rbd.ImageNotFound: [errno 2] error opening image b'image-00001' at snapshot None 03/02/2021 13:02:30 ERROR Cannot get image meta object from rbd header. 03/02/2021 13:02:30 ERROR Get ceph image size image-00001 error 03/02/2021 13:02:30 ERROR [errno 2] error opening image b'image-00001' at snapshot None Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/ceph/api.py", line 1044, in __get_image_size size = int(math.ceil(float(rbd.Image(ioctx, image).size()) / float(1024 ** 3))) File "rbd.pyx", line 2958, in rbd.Image.__init__ rbd.ImageNotFound: [errno 2] error opening image b'image-00001' at snapshot None 03/02/2021 13:02:40 ERROR Cannot get image meta object from rbd header. 03/02/2021 13:02:40 ERROR Get ceph image size image-00001 error 03/02/2021 13:02:40 ERROR [errno 2] error opening image b'image-00001' at snapshot None Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/ceph/api.py", line 1044, in __get_image_size size = int(math.ceil(float(rbd.Image(ioctx, image).size()) / float(1024 ** 3))) File "rbd.pyx", line 2958, in rbd.Image.__init__ rbd.ImageNotFound: [errno 2] error opening image b'image-00001' at snapshot None 03/02/2021 13:02:50 ERROR Cannot get image meta object from rbd header. 03/02/2021 13:02:50 ERROR Get ceph image size image-00001 error 03/02/2021 13:02:50 ERROR [errno 2] error opening image b'image-00001' at snapshot None Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/ceph/api.py", line 1044, in __get_image_size size = int(math.ceil(float(rbd.Image(ioctx, image).size()) / float(1024 ** 3))) File "rbd.pyx", line 2958, in rbd.Image.__init__ rbd.ImageNotFound: [errno 2] error opening image b'image-00001' at snapshot None 03/02/2021 13:03:00 ERROR Cannot get image meta object from rbd header. 03/02/2021 13:03:00 ERROR Get ceph image size image-00001 error 03/02/2021 13:03:00 ERROR [errno 2] error opening image b'image-00001' at snapshot None Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/ceph/api.py", line 1044, in __get_image_size size = int(math.ceil(float(rbd.Image(ioctx, image).size()) / float(1024 ** 3))) File "rbd.pyx", line 2958, in rbd.Image.__init__ rbd.ImageNotFound: [errno 2] error opening image b'image-00001' at snapshot None 03/02/2021 13:03:11 ERROR Cannot get image meta object from rbd header. 03/02/2021 13:03:11 ERROR Get ceph image size image-00001 error 03/02/2021 13:03:11 ERROR [errno 2] error opening image b'image-00001' at snapshot None Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/ceph/api.py", line 1044, in __get_image_size size = int(math.ceil(float(rbd.Image(ioctx, image).size()) / float(1024 ** 3))) File "rbd.pyx", line 2958, in rbd.Image.__init__ rbd.ImageNotFound: [errno 2] error opening image b'image-00001' at snapshot None 03/02/2021 13:03:20 ERROR Cannot get image meta object from rbd header. 03/02/2021 13:03:21 ERROR Get ceph image size image-00001 error 03/02/2021 13:03:21 ERROR [errno 2] error opening image b'image-00001' at snapshot None Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/ceph/api.py", line 1044, in __get_image_size size = int(math.ceil(float(rbd.Image(ioctx, image).size()) / float(1024 ** 3))) File "rbd.pyx", line 2958, in rbd.Image.__init__ rbd.ImageNotFound: [errno 2] error opening image b'image-00001' at snapshot None 03/02/2021 13:03:30 ERROR Cannot get image meta object from rbd header. 03/02/2021 13:03:30 ERROR Get ceph image size image-00001 error 03/02/2021 13:03:30 ERROR [errno 2] error opening image b'image-00001' at snapshot None Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/ceph/api.py", line 1044, in __get_image_size size = int(math.ceil(float(rbd.Image(ioctx, image).size()) / float(1024 ** 3))) File "rbd.pyx", line 2958, in rbd.Image.__init__ rbd.ImageNotFound: [errno 2] error opening image b'image-00001' at snapshot None 03/02/2021 13:03:41 ERROR Cannot get image meta object from rbd header. 03/02/2021 13:03:41 ERROR Get ceph image size image-00001 error 03/02/2021 13:03:41 ERROR [errno 2] error opening image b'image-00001' at snapshot None Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/ceph/api.py", line 1044, in __get_image_size size = int(math.ceil(float(rbd.Image(ioctx, image).size()) / float(1024 ** 3))) File "rbd.pyx", line 2958, in rbd.Image.__init__ rbd.ImageNotFound: [errno 2] error opening image b'image-00001' at snapshot None 03/02/2021 13:03:50 ERROR Cannot get image meta object from rbd header. 03/02/2021 13:03:50 ERROR Get ceph image size image-00001 error 03/02/2021 13:03:50 ERROR [errno 2] error opening image b'image-00001' at snapshot None Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/ceph/api.py", line 1044, in __get_image_size size = int(math.ceil(float(rbd.Image(ioctx, image).size()) / float(1024 ** 3))) File "rbd.pyx", line 2958, in rbd.Image.__init__ rbd.ImageNotFound: [errno 2] error opening image b'image-00001' at snapshot None 03/02/2021 13:04:00 ERROR Cannot get image meta object from rbd header. 03/02/2021 13:04:00 ERROR Get ceph image size image-00001 error 03/02/2021 13:04:00 ERROR [errno 2] error opening image b'image-00001' at snapshot None Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/ceph/api.py", line 1044, in __get_image_size size = int(math.ceil(float(rbd.Image(ioctx, image).size()) / float(1024 ** 3))) File "rbd.pyx", line 2958, in rbd.Image.__init__ rbd.ImageNotFound: [errno 2] error opening image b'image-00001' at snapshot None 03/02/2021 13:04:10 ERROR Cannot get image meta object from rbd header. 03/02/2021 13:04:10 ERROR Get ceph image size image-00001 error 03/02/2021 13:04:10 ERROR [errno 2] error opening image b'image-00001' at snapshot None Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/ceph/api.py", line 1044, in __get_image_size size = int(math.ceil(float(rbd.Image(ioctx, image).size()) / float(1024 ** 3))) File "rbd.pyx", line 2958, in rbd.Image.__init__ rbd.ImageNotFound: [errno 2] error opening image b'image-00001' at snapshot None 03/02/2021 13:04:20 ERROR Cannot get image meta object from rbd header. 03/02/2021 13:04:21 ERROR Get ceph image size image-00001 error 03/02/2021 13:04:21 ERROR [errno 2] error opening image b'image-00001' at snapshot None Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/ceph/api.py", line 1044, in __get_image_size size = int(math.ceil(float(rbd.Image(ioctx, image).size()) / float(1024 ** 3))) File "rbd.pyx", line 2958, in rbd.Image.__init__ rbd.ImageNotFound: [errno 2] error opening image b'image-00001' at snapshot None 03/02/2021 13:04:30 ERROR Cannot get image meta object from rbd header. 03/02/2021 13:04:30 ERROR Get ceph image size image-00001 error 03/02/2021 13:04:30 ERROR [errno 2] error opening image b'image-00001' at snapshot None Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/ceph/api.py", line 1044, in __get_image_size size = int(math.ceil(float(rbd.Image(ioctx, image).size()) / float(1024 ** 3))) File "rbd.pyx", line 2958, in rbd.Image.__init__ rbd.ImageNotFound: [errno 2] error opening image b'image-00001' at snapshot None 03/02/2021 13:04:40 ERROR Cannot get image meta object from rbd header. 03/02/2021 13:04:40 ERROR Get ceph image size image-00001 error 03/02/2021 13:04:40 ERROR [errno 2] error opening image b'image-00001' at snapshot None Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/ceph/api.py", line 1044, in __get_image_size size = int(math.ceil(float(rbd.Image(ioctx, image).size()) / float(1024 ** 3))) File "rbd.pyx", line 2958, in rbd.Image.__init__ rbd.ImageNotFound: [errno 2] error opening image b'image-00001' at snapshot None 03/02/2021 13:04:50 ERROR Cannot get image meta object from rbd header. 03/02/2021 13:04:50 ERROR Get ceph image size image-00001 error 03/02/2021 13:04:50 ERROR [errno 2] error opening image b'image-00001' at snapshot None Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/ceph/api.py", line 1044, in __get_image_size size = int(math.ceil(float(rbd.Image(ioctx, image).size()) / float(1024 ** 3))) File "rbd.pyx", line 2958, in rbd.Image.__init__ rbd.ImageNotFound: [errno 2] error opening image b'image-00001' at snapshot None 03/02/2021 13:05:00 ERROR Cannot get image meta object from rbd header. 03/02/2021 13:05:00 ERROR Get ceph image size image-00001 error 03/02/2021 13:05:00 ERROR [errno 2] error opening image b'image-00001' at snapshot None Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/ceph/api.py", line 1044, in __get_image_size size = int(math.ceil(float(rbd.Image(ioctx, image).size()) / float(1024 ** 3))) File "rbd.pyx", line 2958, in rbd.Image.__init__ rbd.ImageNotFound: [errno 2] error opening image b'image-00001' at snapshot None 03/02/2021 13:05:10 ERROR Cannot get image meta object from rbd header. 03/02/2021 13:05:10 ERROR Get ceph image size image-00001 error 03/02/2021 13:05:10 ERROR [errno 2] error opening image b'image-00001' at snapshot None Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/ceph/api.py", line 1044, in __get_image_size size = int(math.ceil(float(rbd.Image(ioctx, image).size()) / float(1024 ** 3))) File "rbd.pyx", line 2958, in rbd.Image.__init__ rbd.ImageNotFound: [errno 2] error opening image b'image-00001' at snapshot None 03/02/2021 13:05:20 ERROR Cannot get image meta object from rbd header. 03/02/2021 13:05:20 ERROR Get ceph image size image-00001 error 03/02/2021 13:05:20 ERROR [errno 2] error opening image b'image-00001' at snapshot None Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/ceph/api.py", line 1044, in __get_image_size size = int(math.ceil(float(rbd.Image(ioctx, image).size()) / float(1024 ** 3))) File "rbd.pyx", line 2958, in rbd.Image.__init__ rbd.ImageNotFound: [errno 2] error opening image b'image-00001' at snapshot None 03/02/2021 13:05:30 ERROR Cannot get image meta object from rbd header. 03/02/2021 13:05:30 ERROR Get ceph image size image-00001 error 03/02/2021 13:05:30 ERROR [errno 2] error opening image b'image-00001' at snapshot None Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/ceph/api.py", line 1044, in __get_image_size size = int(math.ceil(float(rbd.Image(ioctx, image).size()) / float(1024 ** 3))) File "rbd.pyx", line 2958, in rbd.Image.__init__ rbd.ImageNotFound: [errno 2] error opening image b'image-00001' at snapshot None 03/02/2021 13:05:40 ERROR Cannot get image meta object from rbd header. 03/02/2021 13:05:40 ERROR Get ceph image size image-00001 error 03/02/2021 13:05:40 ERROR [errno 2] error opening image b'image-00001' at snapshot None Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/ceph/api.py", line 1044, in __get_image_size size = int(math.ceil(float(rbd.Image(ioctx, image).size()) / float(1024 ** 3))) File "rbd.pyx", line 2958, in rbd.Image.__init__ rbd.ImageNotFound: [errno 2] error opening image b'image-00001' at snapshot None 03/02/2021 13:07:19 ERROR Cannot get image meta object from rbd header. 03/02/2021 13:07:20 ERROR Get ceph image size image-00001 error 03/02/2021 13:07:20 ERROR [errno 2] error opening image b'image-00001' at snapshot None Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/ceph/api.py", line 1044, in __get_image_size size = int(math.ceil(float(rbd.Image(ioctx, image).size()) / float(1024 ** 3))) File "rbd.pyx", line 2958, in rbd.Image.__init__ rbd.ImageNotFound: [errno 2] error opening image b'image-00001' at snapshot None 03/02/2021 13:51:43 INFO Checking backend latencies : 03/02/2021 13:51:43 INFO Network latency for backend 172.16.0.151 = 03/02/2021 13:51:43 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:51:43 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:51:46 INFO Checking backend latencies : 03/02/2021 13:51:46 INFO Network latency for backend 172.16.0.151 = 03/02/2021 13:51:46 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:51:46 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:51:47 INFO Checking backend latencies : 03/02/2021 13:51:47 INFO Network latency for backend 172.16.0.151 = 03/02/2021 13:51:47 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:51:47 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:51:56 INFO Checking backend latencies : 03/02/2021 13:51:56 INFO Network latency for backend 172.16.0.151 = 03/02/2021 13:51:56 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:51:56 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:51:56 INFO Checking backend latencies : 03/02/2021 13:51:56 INFO Network latency for backend 172.16.0.151 = 03/02/2021 13:51:56 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:51:56 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:52:07 INFO Checking backend latencies : 03/02/2021 13:52:07 INFO Network latency for backend 172.16.0.151 = 03/02/2021 13:52:07 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:52:07 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:52:10 INFO Checking backend latencies : 03/02/2021 13:52:10 INFO Network latency for backend 172.16.0.151 = 03/02/2021 13:52:10 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:52:10 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:52:11 INFO Checking backend latencies : 03/02/2021 13:52:11 INFO Network latency for backend 172.16.0.151 = 03/02/2021 13:52:11 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:52:11 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:52:21 INFO Checking backend latencies : 03/02/2021 13:52:21 INFO Network latency for backend 172.16.0.151 = 03/02/2021 13:52:21 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:52:21 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:52:21 INFO Checking backend latencies : 03/02/2021 13:52:21 INFO Network latency for backend 172.16.0.151 = 03/02/2021 13:52:21 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:52:21 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:52:34 INFO Checking backend latencies : 03/02/2021 13:52:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 13:52:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:52:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:52:36 INFO Checking backend latencies : 03/02/2021 13:52:36 INFO Network latency for backend 172.16.0.151 = 03/02/2021 13:52:36 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:52:36 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:52:37 INFO Checking backend latencies : 03/02/2021 13:52:37 INFO Network latency for backend 172.16.0.151 = 03/02/2021 13:52:37 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:52:37 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:52:43 INFO Checking backend latencies : 03/02/2021 13:52:43 INFO Network latency for backend 172.16.0.151 = 03/02/2021 13:52:43 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:52:43 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:52:47 INFO Checking backend latencies : 03/02/2021 13:52:47 INFO Network latency for backend 172.16.0.151 = 03/02/2021 13:52:47 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:52:47 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:52:47 INFO Checking backend latencies : 03/02/2021 13:52:47 INFO Network latency for backend 172.16.0.151 = 03/02/2021 13:52:47 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:52:47 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:53:04 INFO Checking backend latencies : 03/02/2021 13:53:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 13:53:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:53:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:53:06 INFO Checking backend latencies : 03/02/2021 13:53:06 INFO Network latency for backend 172.16.0.151 = 03/02/2021 13:53:06 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:53:06 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:53:07 INFO Checking backend latencies : 03/02/2021 13:53:07 INFO Network latency for backend 172.16.0.151 = 03/02/2021 13:53:07 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:53:07 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:53:07 INFO Checking backend latencies : 03/02/2021 13:53:07 INFO Network latency for backend 172.16.0.151 = 03/02/2021 13:53:07 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:53:07 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:53:17 INFO Checking backend latencies : 03/02/2021 13:53:17 INFO Network latency for backend 172.16.0.151 = 03/02/2021 13:53:17 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:53:17 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:53:17 INFO Checking backend latencies : 03/02/2021 13:53:17 INFO Network latency for backend 172.16.0.151 = 03/02/2021 13:53:17 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:53:17 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:53:33 INFO Checking backend latencies : 03/02/2021 13:53:33 INFO Network latency for backend 172.16.0.151 = 03/02/2021 13:53:33 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:53:33 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:53:42 INFO Checking backend latencies : 03/02/2021 13:53:42 INFO Network latency for backend 172.16.0.151 = 03/02/2021 13:53:42 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:53:42 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:53:42 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 13:53:43 INFO Checking backend latencies : 03/02/2021 13:53:43 INFO Network latency for backend 172.16.0.151 = 03/02/2021 13:53:43 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:53:43 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:53:45 INFO Checking backend latencies : 03/02/2021 13:53:45 INFO Network latency for backend 172.16.0.151 = 03/02/2021 13:53:45 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:53:45 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:53:45 ERROR Error on consul connection. 03/02/2021 13:53:45 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/iscsi_service.py", line 158, in start self.__current_lock_index = consul_api.current_index() File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 348, in current_index raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 345, in current_index index = consul_obj.get(self.root)[0] File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 13:53:45 INFO Checking backend latencies : 03/02/2021 13:53:45 INFO Network latency for backend 172.16.0.151 = 03/02/2021 13:53:45 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:53:45 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:53:45 ERROR Could not find key resource. 03/02/2021 13:53:45 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 496, in get_key_blocking index, data = consul_obj.get(key, lock_index, False, timeout) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 13:53:45 INFO ClusterLeader stop action 03/02/2021 13:53:52 ERROR Node Stats exception. 03/02/2021 13:53:52 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 4.33 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 36.81 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1064.96 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 40.96 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 194.56 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 0.0 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 4.33 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 36.81 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1064.96 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 40.96 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 194.56 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 0.0 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 13:53:55 INFO Checking backend latencies : 03/02/2021 13:53:55 INFO Network latency for backend 172.16.0.151 = 03/02/2021 13:53:55 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:53:55 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:53:55 ERROR object of type 'NoneType' has no len() Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/notification/plugins/check/replication.py", line 39, in run if len(failed_jobs) > 0: TypeError: object of type 'NoneType' has no len() 03/02/2021 13:53:55 ERROR An error occurred while ReplicationNotificationPlugin was running. 03/02/2021 13:53:55 INFO Checking backend latencies : 03/02/2021 13:53:55 INFO Network latency for backend 172.16.0.151 = 03/02/2021 13:53:55 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:53:55 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:53:55 ERROR error get down node list Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/notification/plugins/check/node.py", line 45, in __get_down_node_list node_list = con_api.get_node_list() File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 432, in get_node_list index, data = consul_obj.get(ConfigAPI().get_consul_nodes_path(), recurse=True) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 13:54:04 INFO Checking backend latencies : 03/02/2021 13:54:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 13:54:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:54:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:54:07 INFO Checking backend latencies : 03/02/2021 13:54:07 INFO Network latency for backend 172.16.0.151 = 03/02/2021 13:54:07 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:54:07 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:54:14 INFO Checking backend latencies : 03/02/2021 13:54:14 INFO Network latency for backend 172.16.0.151 = 03/02/2021 13:54:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:54:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:54:17 INFO Checking backend latencies : 03/02/2021 13:54:17 INFO Network latency for backend 172.16.0.151 = 03/02/2021 13:54:17 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:54:17 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:54:34 INFO Checking backend latencies : 03/02/2021 13:54:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 13:54:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:54:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:54:39 INFO Checking backend latencies : 03/02/2021 13:54:39 INFO Network latency for backend 172.16.0.151 = 03/02/2021 13:54:39 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:54:39 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:54:41 INFO Checking backend latencies : 03/02/2021 13:54:41 INFO Network latency for backend 172.16.0.151 = 03/02/2021 13:54:41 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:54:41 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:54:42 INFO Checking backend latencies : 03/02/2021 13:54:42 INFO Network latency for backend 172.16.0.151 = 03/02/2021 13:54:42 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:54:42 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:54:42 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 13:54:43 INFO Checking backend latencies : 03/02/2021 13:54:43 INFO Network latency for backend 172.16.0.151 = 03/02/2021 13:54:43 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:54:43 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:56:11 INFO Start settings IPs 03/02/2021 13:56:20 INFO GlusterFS mount attempt 03/02/2021 13:56:20 INFO str_start_command: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consul agent -raft-protocol 2 -config-dir /opt/petasan/config/etc/consul.d/server -bind 172.16.0.151 -retry-join 172.16.0.152 -retry-join 172.16.0.153 03/02/2021 13:56:23 INFO Starting cluster file sync service 03/02/2021 13:56:23 INFO Starting iSCSI Service 03/02/2021 13:56:23 INFO Starting Cluster Management application 03/02/2021 13:56:23 INFO Starting Node Stats Service 03/02/2021 13:56:23 INFO LeaderElectionBase dropping old sessions 03/02/2021 13:56:24 INFO Starting activating PetaSAN lvs 03/02/2021 13:56:24 INFO Running command: /sbin/vgs --noheadings --readonly --units=b --nosuffix --separator=";" -o vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_count,vg_extent_size 03/02/2021 13:56:24 INFO stdout ceph-847b13a2-67ce-4374-b26b-f42d81ceb154";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 13:56:24 INFO stdout ceph-d0153ed6-0b75-470c-b1c7-906759dee371";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 13:56:24 INFO stdout ceph-e72ec456-4943-4572-8b6f-f20fa4b85417";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 13:56:24 INFO stdout ceph-fe73888d-9945-41db-8d0d-5bb67360981c";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 13:56:24 INFO Starting activating PetaSAN lvs 03/02/2021 13:56:24 INFO Running command: /sbin/vgs --noheadings --readonly --units=b --nosuffix --separator=";" -o vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_count,vg_extent_size 03/02/2021 13:56:24 INFO stdout ceph-847b13a2-67ce-4374-b26b-f42d81ceb154";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 13:56:24 INFO stdout ceph-d0153ed6-0b75-470c-b1c7-906759dee371";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 13:56:24 INFO stdout ceph-e72ec456-4943-4572-8b6f-f20fa4b85417";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 13:56:24 INFO stdout ceph-fe73888d-9945-41db-8d0d-5bb67360981c";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 13:56:24 INFO checking uptime 03/02/2021 13:56:24 INFO checking uptime of node: peta1 ip:10.10.0.151 03/02/2021 13:56:24 INFO checking uptime of node: peta1 is starting 03/02/2021 13:56:24 INFO checking uptime of node: peta2 ip:10.10.0.152 03/02/2021 13:56:26 INFO Starting OSDs 03/02/2021 13:56:26 INFO Starting petasan tuning service 03/02/2021 13:56:26 INFO Starting Config Upload service 03/02/2021 13:56:26 INFO Starting petasan-qperf server 03/02/2021 13:56:42 INFO Checking backend latencies : 03/02/2021 13:56:42 INFO Network latency for backend 172.16.0.151 = 6.81 us 03/02/2021 13:56:42 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:56:42 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:56:44 ERROR checking uptime of node: peta2 ping error to ip: 10.10.0.152 03/02/2021 13:56:44 INFO checking uptime of node: peta3 ip:10.10.0.153 03/02/2021 13:56:44 INFO Checking backend latencies : 03/02/2021 13:56:44 INFO Network latency for backend 172.16.0.151 = 7.28 us 03/02/2021 13:56:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:56:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:56:53 INFO Checking backend latencies : 03/02/2021 13:56:53 INFO Network latency for backend 172.16.0.151 = 7.28 us 03/02/2021 13:56:53 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:56:53 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:56:55 INFO Checking backend latencies : 03/02/2021 13:56:55 INFO Network latency for backend 172.16.0.151 = 7.45 us 03/02/2021 13:56:55 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:56:55 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:56:57 INFO GlusterFS mount attempt 03/02/2021 13:57:04 INFO Checking backend latencies : 03/02/2021 13:57:04 INFO Network latency for backend 172.16.0.151 = 7.46 us 03/02/2021 13:57:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:57:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:57:04 ERROR checking uptime of node: peta3 ping error to ip: 10.10.0.153 03/02/2021 13:57:04 INFO Service is starting. 03/02/2021 13:57:04 INFO iSCSI Service dropping old sessions 03/02/2021 13:57:06 INFO Checking backend latencies : 03/02/2021 13:57:06 INFO Network latency for backend 172.16.0.151 = 6.82 us 03/02/2021 13:57:06 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:57:06 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:57:14 INFO Checking backend latencies : 03/02/2021 13:57:14 INFO Network latency for backend 172.16.0.151 = 6.73 us 03/02/2021 13:57:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:57:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:57:17 INFO Checking backend latencies : 03/02/2021 13:57:17 INFO Network latency for backend 172.16.0.151 = 6.73 us 03/02/2021 13:57:17 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:57:17 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:57:27 INFO Checking backend latencies : 03/02/2021 13:57:27 INFO Network latency for backend 172.16.0.151 = 6.19 us 03/02/2021 13:57:27 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:57:27 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:57:28 INFO GlusterFS mount attempt 03/02/2021 13:57:29 INFO Checking backend latencies : 03/02/2021 13:57:29 INFO Network latency for backend 172.16.0.151 = 6.15 us 03/02/2021 13:57:29 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:57:29 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:57:31 INFO Checking backend latencies : 03/02/2021 13:57:31 INFO Network latency for backend 172.16.0.151 = 7.0 us 03/02/2021 13:57:31 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:57:31 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:57:37 INFO Checking backend latencies : 03/02/2021 13:57:37 INFO Network latency for backend 172.16.0.151 = 6.76 us 03/02/2021 13:57:37 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:57:37 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:57:39 INFO Checking backend latencies : 03/02/2021 13:57:39 INFO Network latency for backend 172.16.0.151 = 7.06 us 03/02/2021 13:57:39 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:57:39 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:57:41 INFO Checking backend latencies : 03/02/2021 13:57:41 INFO Network latency for backend 172.16.0.151 = 6.13 us 03/02/2021 13:57:41 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:57:41 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:57:45 INFO Checking backend latencies : 03/02/2021 13:57:45 INFO Network latency for backend 172.16.0.151 = 7.33 us 03/02/2021 13:57:45 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:57:45 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:57:52 INFO Checking backend latencies : 03/02/2021 13:57:52 INFO Network latency for backend 172.16.0.151 = 6.81 us 03/02/2021 13:57:52 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:57:52 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:57:54 INFO Checking backend latencies : 03/02/2021 13:57:54 INFO Network latency for backend 172.16.0.151 = 6.25 us 03/02/2021 13:57:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:57:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:57:56 INFO Checking backend latencies : 03/02/2021 13:57:56 INFO Network latency for backend 172.16.0.151 = 6.1 us 03/02/2021 13:57:56 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:57:56 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:58:00 INFO Checking backend latencies : 03/02/2021 13:58:00 INFO Network latency for backend 172.16.0.151 = 6.97 us 03/02/2021 13:58:00 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:58:00 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:58:01 INFO GlusterFS mount attempt 03/02/2021 13:58:04 INFO Checking backend latencies : 03/02/2021 13:58:04 INFO Network latency for backend 172.16.0.151 = 6.9 us 03/02/2021 13:58:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:58:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:58:06 INFO Checking backend latencies : 03/02/2021 13:58:06 INFO Network latency for backend 172.16.0.151 = 6.18 us 03/02/2021 13:58:06 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:58:06 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:58:08 INFO Checking backend latencies : 03/02/2021 13:58:08 INFO Network latency for backend 172.16.0.151 = 6.12 us 03/02/2021 13:58:08 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:58:08 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:58:15 INFO Checking backend latencies : 03/02/2021 13:58:15 INFO Network latency for backend 172.16.0.151 = 6.19 us 03/02/2021 13:58:15 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:58:15 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:58:23 INFO Checking backend latencies : 03/02/2021 13:58:23 INFO Network latency for backend 172.16.0.151 = 6.99 us 03/02/2021 13:58:23 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:58:23 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:58:29 INFO Checking backend latencies : 03/02/2021 13:58:29 INFO Network latency for backend 172.16.0.151 = 6.56 us 03/02/2021 13:58:29 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:58:29 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:58:29 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 13:58:31 INFO Checking backend latencies : 03/02/2021 13:58:31 INFO Network latency for backend 172.16.0.151 = 6.1 us 03/02/2021 13:58:31 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:58:31 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:58:32 INFO GlusterFS mount attempt 03/02/2021 13:58:33 INFO Checking backend latencies : 03/02/2021 13:58:33 INFO Network latency for backend 172.16.0.151 = 6.19 us 03/02/2021 13:58:33 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:58:33 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:58:33 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader 03/02/2021 13:58:33 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 516, in get_key index, data = cons.get(key) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 13:58:39 INFO Checking backend latencies : 03/02/2021 13:58:39 INFO Network latency for backend 172.16.0.151 = 8.03 us 03/02/2021 13:58:39 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:58:39 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:58:39 ERROR LeaderElectionBase exception dropping old sessions 03/02/2021 13:58:42 INFO Checking backend latencies : 03/02/2021 13:58:42 INFO Network latency for backend 172.16.0.151 = 7.28 us 03/02/2021 13:58:42 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:58:42 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:58:44 INFO Checking backend latencies : 03/02/2021 13:58:44 INFO Network latency for backend 172.16.0.151 = 6.73 us 03/02/2021 13:58:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:58:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:58:44 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 13:58:50 INFO Checking backend latencies : 03/02/2021 13:58:50 INFO Network latency for backend 172.16.0.151 = 6.44 us 03/02/2021 13:58:50 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:58:50 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:58:52 INFO Checking backend latencies : 03/02/2021 13:58:52 INFO Network latency for backend 172.16.0.151 = 6.44 us 03/02/2021 13:58:52 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:58:52 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:58:58 INFO Checking backend latencies : 03/02/2021 13:58:58 INFO Network latency for backend 172.16.0.151 = 6.44 us 03/02/2021 13:58:58 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:58:58 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:59:05 INFO GlusterFS mount attempt 03/02/2021 13:59:07 INFO iSCSI Service successfully dropped old sessions 03/02/2021 13:59:07 INFO Cleaning unused configurations. 03/02/2021 13:59:07 INFO Cleaning all mapped disks 03/02/2021 13:59:07 INFO Cleaning unused rbd images. 03/02/2021 13:59:07 INFO Cleaning unused ips. 03/02/2021 13:59:09 INFO Checking backend latencies : 03/02/2021 13:59:09 INFO Network latency for backend 172.16.0.151 = 6.42 us 03/02/2021 13:59:09 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:59:09 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:59:11 INFO LeaderElectionBase successfully dropped old sessions 03/02/2021 13:59:13 INFO Checking backend latencies : 03/02/2021 13:59:13 INFO Network latency for backend 172.16.0.151 = 6.36 us 03/02/2021 13:59:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 13:59:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 13:59:36 INFO GlusterFS mount attempt 03/02/2021 13:59:42 INFO ClusterLeader start action 03/02/2021 13:59:42 INFO ClusterLeader starting services 03/02/2021 14:01:43 INFO Checking backend latencies : 03/02/2021 14:01:43 INFO Network latency for backend 172.16.0.151 = 7.34 us 03/02/2021 14:01:43 INFO Network latency for backend 172.16.0.152 = 03/02/2021 14:01:43 INFO Network latency for backend 172.16.0.153 = 03/02/2021 14:02:01 INFO Checking backend latencies : 03/02/2021 14:02:01 INFO Network latency for backend 172.16.0.151 = 7.55 us 03/02/2021 14:02:01 INFO Network latency for backend 172.16.0.152 = 03/02/2021 14:02:01 INFO Network latency for backend 172.16.0.153 = 03/02/2021 14:02:04 INFO Checking backend latencies : 03/02/2021 14:02:04 INFO Network latency for backend 172.16.0.151 = 7.44 us 03/02/2021 14:02:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 14:02:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 14:19:37 ERROR Node Stats exception. 03/02/2021 14:19:37 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 2.51 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 22.85 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1208.32 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 81.92 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.25 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 317071.36 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 190371.84 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.24 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 299857.92 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 2.51 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 22.85 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1208.32 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 81.92 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.25 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 317071.36 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 190371.84 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.24 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 299857.92 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 20:07:15 INFO Start settings IPs 03/02/2021 20:07:26 INFO GlusterFS mount attempt 03/02/2021 20:07:26 INFO str_start_command: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consul agent -raft-protocol 2 -config-dir /opt/petasan/config/etc/consul.d/server -bind 172.16.0.151 -retry-join 172.16.0.152 -retry-join 172.16.0.153 03/02/2021 20:07:29 INFO Starting cluster file sync service 03/02/2021 20:07:30 INFO Starting iSCSI Service 03/02/2021 20:07:30 INFO Starting Cluster Management application 03/02/2021 20:07:30 INFO Starting Node Stats Service 03/02/2021 20:07:30 INFO LeaderElectionBase dropping old sessions 03/02/2021 20:07:30 INFO Starting activating PetaSAN lvs 03/02/2021 20:07:30 INFO Running command: /sbin/vgs --noheadings --readonly --units=b --nosuffix --separator=";" -o vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_count,vg_extent_size 03/02/2021 20:07:30 INFO stdout ceph-847b13a2-67ce-4374-b26b-f42d81ceb154";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 20:07:30 INFO stdout ceph-d0153ed6-0b75-470c-b1c7-906759dee371";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 20:07:30 INFO stdout ceph-e72ec456-4943-4572-8b6f-f20fa4b85417";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 20:07:30 INFO stdout ceph-fe73888d-9945-41db-8d0d-5bb67360981c";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 20:07:30 INFO Starting activating PetaSAN lvs 03/02/2021 20:07:30 INFO Running command: /sbin/vgs --noheadings --readonly --units=b --nosuffix --separator=";" -o vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_count,vg_extent_size 03/02/2021 20:07:30 INFO stdout ceph-847b13a2-67ce-4374-b26b-f42d81ceb154";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 20:07:30 INFO stdout ceph-d0153ed6-0b75-470c-b1c7-906759dee371";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 20:07:30 INFO stdout ceph-e72ec456-4943-4572-8b6f-f20fa4b85417";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 20:07:30 INFO stdout ceph-fe73888d-9945-41db-8d0d-5bb67360981c";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 20:07:31 INFO checking uptime 03/02/2021 20:07:31 INFO checking uptime of node: peta1 ip:10.10.0.151 03/02/2021 20:07:31 INFO checking uptime of node: peta1 is starting 03/02/2021 20:07:31 INFO checking uptime of node: peta2 ip:10.10.0.152 03/02/2021 20:07:31 INFO checking uptime of node: peta3 ip:10.10.0.153 03/02/2021 20:07:31 INFO Service is starting. 03/02/2021 20:07:31 INFO iSCSI Service dropping old sessions 03/02/2021 20:07:32 INFO Starting OSDs 03/02/2021 20:07:32 INFO Starting petasan tuning service 03/02/2021 20:07:32 INFO Starting Config Upload service 03/02/2021 20:07:32 INFO Starting petasan-qperf server 03/02/2021 20:07:40 INFO LeaderElectionBase successfully dropped old sessions 03/02/2021 20:07:46 INFO iSCSI Service successfully dropped old sessions 03/02/2021 20:07:46 INFO Cleaning unused configurations. 03/02/2021 20:07:46 INFO Cleaning all mapped disks 03/02/2021 20:07:46 INFO Cleaning unused rbd images. 03/02/2021 20:07:46 INFO Cleaning unused ips. 03/02/2021 20:12:51 ERROR Node Stats exception. 03/02/2021 20:12:51 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 2.51 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 18.04 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1116.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 30.72 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.25 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 253122.56 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 318545.92 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.24 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 236800.0 `date +%s`" | nc -q0 10.10.0.152 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 2.51 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 18.04 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1116.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 30.72 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.25 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 253122.56 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 318545.92 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.24 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 236800.0 `date +%s`" | nc -q0 10.10.0.152 2003 03/02/2021 20:13:54 ERROR Node Stats exception. 03/02/2021 20:13:54 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.85 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 18.08 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1208.32 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 245.76 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.17 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 216391.68 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 200663.04 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 202977.28 `date +%s`" | nc -q0 10.10.0.152 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.85 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 18.08 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 1208.32 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 245.76 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.17 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 216391.68 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 200663.04 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 202977.28 `date +%s`" | nc -q0 10.10.0.152 2003 03/02/2021 20:14:58 ERROR Node Stats exception. 03/02/2021 20:14:58 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.68 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 18.1 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 880.64 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 61.44 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.18 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 172861.44 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 221235.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.17 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 161044.48 `date +%s`" | nc -q0 10.10.0.152 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.68 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 18.1 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 880.64 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 61.44 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.18 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 172861.44 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 221235.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.17 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 161044.48 `date +%s`" | nc -q0 10.10.0.152 2003 03/02/2021 20:16:01 ERROR Node Stats exception. 03/02/2021 20:16:01 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.17 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 18.11 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 819.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 30.72 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 167147.52 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 199372.8 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.15 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 155381.76 `date +%s`" | nc -q0 10.10.0.152 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.17 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 18.11 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 819.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 30.72 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.16 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 167147.52 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 199372.8 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.15 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 155381.76 `date +%s`" | nc -q0 10.10.0.152 2003 04/02/2021 03:48:01 ERROR Node Stats exception. 04/02/2021 03:48:01 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.17 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 23.72 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 901.12 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 20.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.19 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 217651.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 232099.84 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.18 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 202588.16 `date +%s`" | nc -q0 10.10.0.152 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta1.cpu_all.percent_util 1.17 `date +%s`" " PetaSAN.NodeStats.peta1.memory.percent_util 23.72 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_received 901.12 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth0_transmitted 20.48 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1 0.19 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_received 217651.2 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1_transmitted 232099.84 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.percent_util.eth1-100 0.18 `date +%s`" " PetaSAN.NodeStats.peta1.ifaces.throughput.eth1-100_received 202588.16 `date +%s`" | nc -q0 10.10.0.152 2003