01/02/2021 23:20:40 INFO Start settings IPs 01/02/2021 23:22:37 INFO Start settings IPs 01/02/2021 23:25:33 INFO Start settings IPs 01/02/2021 23:37:28 INFO Start settings IPs 01/02/2021 23:55:26 ERROR Config file error. The PetaSAN os maybe just installed. Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/cluster/deploy.py", line 69, in get_node_status node_name = config.get_node_info().name File "/usr/lib/python3/dist-packages/PetaSAN/core/cluster/configuration.py", line 99, in get_node_info with open(config.get_node_info_file_path(), 'r') as f: FileNotFoundError: [Errno 2] No such file or directory: '/opt/petasan/config/node_info.json' 01/02/2021 23:55:49 INFO Starting node join 01/02/2021 23:55:49 INFO Successfully copied public keys. 01/02/2021 23:55:49 INFO Successfully copied private keys. 01/02/2021 23:55:49 INFO password set successfully. 01/02/2021 23:55:50 INFO Start copying cluster info file. 01/02/2021 23:55:50 INFO Successfully copied cluster info file. 01/02/2021 23:55:50 INFO Start copying services interfaces file. 01/02/2021 23:55:50 INFO Successfully copied services interfaces file. 01/02/2021 23:55:50 INFO Joined cluster petasan 01/02/2021 23:56:27 ERROR 400 Bad Request: The browser (or proxy) sent a request that this server could not understand. 01/02/2021 23:56:27 ERROR 400 Bad Request: The browser (or proxy) sent a request that this server could not understand. 01/02/2021 23:56:27 ERROR 400 Bad Request: The browser (or proxy) sent a request that this server could not understand. 01/02/2021 23:56:27 ERROR 400 Bad Request: The browser (or proxy) sent a request that this server could not understand. 01/02/2021 23:56:27 INFO Set node role completed successfully. 01/02/2021 23:56:28 INFO Set node info completed successfully. 01/02/2021 23:57:26 INFO Stopping petasan services on all nodes. 01/02/2021 23:57:26 INFO Stopping all petasan services. 01/02/2021 23:57:26 INFO files_sync.py process is 1464 01/02/2021 23:57:26 INFO files_sync.py process stopped 01/02/2021 23:57:26 INFO iscsi_service.py process is 1466 01/02/2021 23:57:26 INFO iscsi_service.py process stopped 01/02/2021 23:57:26 INFO admin.py process is 1468 01/02/2021 23:57:26 INFO admin.py process stopped 01/02/2021 23:57:27 INFO Starting local clean_ceph. 01/02/2021 23:57:27 INFO Starting clean_ceph 01/02/2021 23:57:27 INFO Stopping ceph services 01/02/2021 23:57:27 INFO Start cleaning config files 01/02/2021 23:57:27 INFO Starting ceph services 01/02/2021 23:57:29 INFO Starting local clean_consul. 01/02/2021 23:57:29 INFO Trying to clean Consul on local node 01/02/2021 23:57:29 INFO delete /opt/petasan/config/etc/consul.d 01/02/2021 23:57:29 INFO delete /opt/petasan/config/var/consul 01/02/2021 23:57:29 INFO Trying to clean Consul on 10.10.0.151 01/02/2021 23:57:29 INFO Trying to clean Consul on 10.10.0.152 01/02/2021 23:57:30 INFO cluster_name: petasan 01/02/2021 23:57:30 INFO local_node_info.name: peta3 01/02/2021 23:57:31 INFO Start consul leaders remotely. 01/02/2021 11:58:40 INFO str_start_command: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consul agent -raft-protocol 2 -config-dir /opt/petasan/config/etc/consul.d/server -bind 172.16.0.153 -retry-join 172.16.0.151 -retry-join 172.16.0.152 01/02/2021 11:58:41 INFO Cluster Node {} joined the cluster and is alivepeta3 01/02/2021 11:58:41 INFO Cluster Node {} joined the cluster and is alivepeta1 01/02/2021 11:58:41 INFO Cluster Node {} joined the cluster and is alivepeta2 01/02/2021 11:58:41 INFO Consul leaders are ready 01/02/2021 11:58:51 INFO NFSServer : Changing NFS Settings in Consul 01/02/2021 11:58:51 INFO NFSServer : NFS Settings has been changed in Consul 01/02/2021 11:58:51 INFO Success saving application config 01/02/2021 11:58:51 INFO Success saving application config 01/02/2021 11:58:52 INFO First monitor started successfully 01/02/2021 11:58:52 INFO create_mgr() fresh install 01/02/2021 11:58:52 INFO create_mgr() started 01/02/2021 11:58:52 INFO create_mgr() cmd : mkdir -p /var/lib/ceph/mgr/ceph-peta3 01/02/2021 11:58:52 INFO create_mgr() cmd : ceph --cluster ceph auth get-or-create mgr.peta3 mon 'allow profile mgr' osd 'allow *' mds 'allow *' -o /var/lib/ceph/mgr/ceph-peta3/keyring 01/02/2021 11:59:41 INFO create_mgr() ended successfully 01/02/2021 11:59:41 INFO create_mds() started 01/02/2021 11:59:41 INFO create_mds() cmd : mkdir -p /var/lib/ceph/mds/ceph-peta3 01/02/2021 12:00:08 INFO create_mds() ended successfully 01/02/2021 12:00:08 INFO Starting to deploy remote monitors 01/02/2021 12:01:50 INFO Ceph monitors are ready. 01/02/2021 12:02:13 INFO Start deploying ceph OSDs. 01/02/2021 12:02:13 INFO Start cleaning disk : sdf 01/02/2021 12:02:17 INFO Executing : wipefs --all /dev/sdf1 01/02/2021 12:02:17 INFO Executing : dd if=/dev/zero of=/dev/sdf1 bs=1M count=20 oflag=direct,dsync >/dev/null 2>&1 01/02/2021 12:02:17 ERROR Error executing : dd if=/dev/zero of=/dev/sdf1 bs=1M count=20 oflag=direct,dsync >/dev/null 2>&1 01/02/2021 12:02:20 INFO Executing : wipefs --all /dev/sdf2 01/02/2021 12:02:20 INFO Executing : dd if=/dev/zero of=/dev/sdf2 bs=1M count=20 oflag=direct,dsync >/dev/null 2>&1 01/02/2021 12:02:20 INFO Executing : wipefs --all /dev/sdf 01/02/2021 12:02:20 INFO Executing : dd if=/dev/zero of=/dev/sdf bs=1M count=20 oflag=direct,dsync >/dev/null 2>&1 01/02/2021 12:02:20 INFO Executing : parted -s /dev/sdf mklabel gpt 01/02/2021 12:02:20 INFO Executing : partprobe /dev/sdf 01/02/2021 12:02:23 INFO Start cleaning disk : sdf 01/02/2021 12:02:24 INFO Executing : wipefs --all /dev/sdf 01/02/2021 12:02:24 INFO Executing : dd if=/dev/zero of=/dev/sdf bs=1M count=20 oflag=direct,dsync >/dev/null 2>&1 01/02/2021 12:02:24 INFO Executing : parted -s /dev/sdf mklabel gpt 01/02/2021 12:02:24 INFO Executing : partprobe /dev/sdf 01/02/2021 12:02:28 INFO Start cleaning disk : sda 01/02/2021 12:02:29 INFO Executing : wipefs --all /dev/sda 01/02/2021 12:02:29 INFO Executing : dd if=/dev/zero of=/dev/sda bs=1M count=20 oflag=direct,dsync >/dev/null 2>&1 01/02/2021 12:02:30 INFO Executing : parted -s /dev/sda mklabel gpt 01/02/2021 12:02:30 INFO Executing : partprobe /dev/sda 01/02/2021 12:02:35 INFO Start cleaning disk : sdf 01/02/2021 12:02:38 INFO Executing : wipefs --all /dev/sdf1 01/02/2021 12:02:38 INFO Executing : dd if=/dev/zero of=/dev/sdf1 bs=1M count=20 oflag=direct,dsync >/dev/null 2>&1 01/02/2021 12:02:38 INFO Executing : wipefs --all /dev/sdf 01/02/2021 12:02:38 INFO Executing : dd if=/dev/zero of=/dev/sdf bs=1M count=20 oflag=direct,dsync >/dev/null 2>&1 01/02/2021 12:02:38 INFO Executing : parted -s /dev/sdf mklabel gpt 01/02/2021 12:02:38 INFO Executing : partprobe /dev/sdf 01/02/2021 12:02:41 INFO Start prepare bluestore OSD : sda 01/02/2021 12:02:41 INFO Running command: /sbin/lvs --noheadings --readonly --separator=";" -a -o lv_tags,lv_path,lv_name,vg_name,lv_uuid,lv_size 01/02/2021 12:02:41 INFO Creating data partition num 1 size 0 on /dev/sda 01/02/2021 12:02:43 INFO Calling partprobe on sda device 01/02/2021 12:02:43 INFO Executing partprobe /dev/sda 01/02/2021 12:02:43 INFO Calling udevadm on sda device 01/02/2021 12:02:43 INFO Executing udevadm settle --timeout 30 01/02/2021 12:02:47 INFO Start remove osd.0 from crush map 01/02/2021 12:02:49 INFO osd.0 is removed from crush map 01/02/2021 12:02:49 INFO creating new journal partition. 01/02/2021 12:02:50 INFO Creating journal-db-0-c74c6e06 partition num 1 size 61440 on /dev/sdf 01/02/2021 12:02:51 INFO Calling partprobe on sdf device 01/02/2021 12:02:51 INFO Executing partprobe /dev/sdf 01/02/2021 12:02:51 INFO Calling udevadm on sdf device 01/02/2021 12:02:51 INFO Executing udevadm settle --timeout 30 01/02/2021 12:02:54 INFO Starting : ceph-volume --log-path /opt/petasan/log lvm prepare --bluestore --data /dev/sda1 --block.db /dev/sdf1 01/02/2021 12:03:00 INFO starting activate osd. 01/02/2021 12:03:00 INFO Starting : ceph-volume --log-path /opt/petasan/log lvm activate --bluestore 0 bdb04a29-703d-442b-acfb-05fb9fb920b7 01/02/2021 12:03:01 INFO Successfully executed ceph-volume prepare for sda 01/02/2021 12:03:01 INFO Start cleaning disk : sdb 01/02/2021 12:03:01 INFO Executing : wipefs --all /dev/sdb 01/02/2021 12:03:01 INFO Executing : dd if=/dev/zero of=/dev/sdb bs=1M count=20 oflag=direct,dsync >/dev/null 2>&1 01/02/2021 12:03:02 INFO Executing : parted -s /dev/sdb mklabel gpt 01/02/2021 12:03:02 INFO Executing : partprobe /dev/sdb 01/02/2021 12:03:07 INFO Start prepare bluestore OSD : sdb 01/02/2021 12:03:07 INFO Running command: /sbin/lvs --noheadings --readonly --separator=";" -a -o lv_tags,lv_path,lv_name,vg_name,lv_uuid,lv_size 01/02/2021 12:03:07 INFO stdout ceph.block_device=/dev/ceph-d4270832-4236-4724-a362-2dc9fb139f6f/osd-block-bdb04a29-703d-442b-acfb-05fb9fb920b7,ceph.block_uuid=peWdkc-SOzG-aq3c-xjRL-KAtL-8kWu-43je3o,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=c74c6e06-673f-4d0c-8235-563559e08858,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/sdf1,ceph.db_uuid=337d0154-26b8-4b50-b979-d1c6438137cf,ceph.encrypted=0,ceph.osd_fsid=bdb04a29-703d-442b-acfb-05fb9fb920b7,ceph.osd_id=0,ceph.osdspec_affinity=,ceph.type=block,ceph.vdo=0";"/dev/ceph-d4270832-4236-4724-a362-2dc9fb139f6f/osd-block-bdb04a29-703d-442b-acfb-05fb9fb920b7";"osd-block-bdb04a29-703d-442b-acfb-05fb9fb920b7";"ceph-d4270832-4236-4724-a362-2dc9fb139f6f";"peWdkc-SOzG-aq3c-xjRL-KAtL-8kWu-43je3o";"<1.82t 01/02/2021 12:03:07 INFO Creating data partition num 1 size 0 on /dev/sdb 01/02/2021 12:03:08 INFO Calling partprobe on sdb device 01/02/2021 12:03:08 INFO Executing partprobe /dev/sdb 01/02/2021 12:03:09 INFO Calling udevadm on sdb device 01/02/2021 12:03:09 INFO Executing udevadm settle --timeout 30 01/02/2021 12:03:13 INFO Start remove osd.1 from crush map 01/02/2021 12:03:15 INFO osd.1 is removed from crush map 01/02/2021 12:03:15 INFO creating new journal partition. 01/02/2021 12:03:16 INFO Creating journal-db-1-c74c6e06 partition num 2 size 61440 on /dev/sdf 01/02/2021 12:03:17 INFO Calling partprobe on sdf device 01/02/2021 12:03:17 INFO Executing partprobe /dev/sdf 01/02/2021 12:03:17 INFO Calling udevadm on sdf device 01/02/2021 12:03:17 INFO Executing udevadm settle --timeout 30 01/02/2021 12:03:20 INFO Starting : ceph-volume --log-path /opt/petasan/log lvm prepare --bluestore --data /dev/sdb1 --block.db /dev/sdf2 01/02/2021 12:03:26 INFO starting activate osd. 01/02/2021 12:03:27 INFO Starting : ceph-volume --log-path /opt/petasan/log lvm activate --bluestore 1 931f14e6-b378-4fd9-ab9a-f54fdae6fdd3 01/02/2021 12:03:27 INFO Successfully executed ceph-volume prepare for sdb 01/02/2021 12:03:27 INFO Start cleaning disk : sdc 01/02/2021 12:03:28 INFO Executing : wipefs --all /dev/sdc 01/02/2021 12:03:28 INFO Executing : dd if=/dev/zero of=/dev/sdc bs=1M count=20 oflag=direct,dsync >/dev/null 2>&1 01/02/2021 12:03:28 INFO Executing : parted -s /dev/sdc mklabel gpt 01/02/2021 12:03:28 INFO Executing : partprobe /dev/sdc 01/02/2021 12:03:33 INFO Start prepare bluestore OSD : sdc 01/02/2021 12:03:33 INFO Running command: /sbin/lvs --noheadings --readonly --separator=";" -a -o lv_tags,lv_path,lv_name,vg_name,lv_uuid,lv_size 01/02/2021 12:03:33 INFO stdout ceph.block_device=/dev/ceph-4ed4c220-9498-4351-8ce9-4e00f8b3a93b/osd-block-931f14e6-b378-4fd9-ab9a-f54fdae6fdd3,ceph.block_uuid=0DcZlj-lyyN-JoGs-Bu1Z-VHzP-HIMV-xjYFcq,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=c74c6e06-673f-4d0c-8235-563559e08858,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/sdf2,ceph.db_uuid=d986e67d-09f4-4b29-a43d-311338eca48a,ceph.encrypted=0,ceph.osd_fsid=931f14e6-b378-4fd9-ab9a-f54fdae6fdd3,ceph.osd_id=1,ceph.osdspec_affinity=,ceph.type=block,ceph.vdo=0";"/dev/ceph-4ed4c220-9498-4351-8ce9-4e00f8b3a93b/osd-block-931f14e6-b378-4fd9-ab9a-f54fdae6fdd3";"osd-block-931f14e6-b378-4fd9-ab9a-f54fdae6fdd3";"ceph-4ed4c220-9498-4351-8ce9-4e00f8b3a93b";"0DcZlj-lyyN-JoGs-Bu1Z-VHzP-HIMV-xjYFcq";"<1.82t 01/02/2021 12:03:33 INFO stdout ceph.block_device=/dev/ceph-d4270832-4236-4724-a362-2dc9fb139f6f/osd-block-bdb04a29-703d-442b-acfb-05fb9fb920b7,ceph.block_uuid=peWdkc-SOzG-aq3c-xjRL-KAtL-8kWu-43je3o,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=c74c6e06-673f-4d0c-8235-563559e08858,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/sdf1,ceph.db_uuid=337d0154-26b8-4b50-b979-d1c6438137cf,ceph.encrypted=0,ceph.osd_fsid=bdb04a29-703d-442b-acfb-05fb9fb920b7,ceph.osd_id=0,ceph.osdspec_affinity=,ceph.type=block,ceph.vdo=0";"/dev/ceph-d4270832-4236-4724-a362-2dc9fb139f6f/osd-block-bdb04a29-703d-442b-acfb-05fb9fb920b7";"osd-block-bdb04a29-703d-442b-acfb-05fb9fb920b7";"ceph-d4270832-4236-4724-a362-2dc9fb139f6f";"peWdkc-SOzG-aq3c-xjRL-KAtL-8kWu-43je3o";"<1.82t 01/02/2021 12:03:33 INFO Creating data partition num 1 size 0 on /dev/sdc 01/02/2021 12:03:34 INFO Calling partprobe on sdc device 01/02/2021 12:03:34 INFO Executing partprobe /dev/sdc 01/02/2021 12:03:34 INFO Calling udevadm on sdc device 01/02/2021 12:03:34 INFO Executing udevadm settle --timeout 30 01/02/2021 12:03:38 INFO Start remove osd.2 from crush map 01/02/2021 12:03:40 INFO osd.2 is removed from crush map 01/02/2021 12:03:40 INFO creating new journal partition. 01/02/2021 12:03:41 INFO Creating journal-db-2-c74c6e06 partition num 3 size 61440 on /dev/sdf 01/02/2021 12:03:42 INFO Calling partprobe on sdf device 01/02/2021 12:03:42 INFO Executing partprobe /dev/sdf 01/02/2021 12:03:42 INFO Calling udevadm on sdf device 01/02/2021 12:03:42 INFO Executing udevadm settle --timeout 30 01/02/2021 12:03:45 INFO Starting : ceph-volume --log-path /opt/petasan/log lvm prepare --bluestore --data /dev/sdc1 --block.db /dev/sdf3 01/02/2021 12:03:50 INFO starting activate osd. 01/02/2021 12:03:51 INFO Starting : ceph-volume --log-path /opt/petasan/log lvm activate --bluestore 2 68c4f029-cdb4-4c6b-909b-4b8981844f5a 01/02/2021 12:03:52 INFO Successfully executed ceph-volume prepare for sdc 01/02/2021 12:03:52 INFO Start cleaning disk : sdd 01/02/2021 12:03:52 INFO Executing : wipefs --all /dev/sdd 01/02/2021 12:03:52 INFO Executing : dd if=/dev/zero of=/dev/sdd bs=1M count=20 oflag=direct,dsync >/dev/null 2>&1 01/02/2021 12:03:53 INFO Executing : parted -s /dev/sdd mklabel gpt 01/02/2021 12:03:53 INFO Executing : partprobe /dev/sdd 01/02/2021 12:05:45 INFO 10.10.0.151 Setting name! partNum is 0 The operation has completed successfully. Setting name! partNum is 0 The operation has completed successfully. Setting name! partNum is 0 The operation has completed successfully. Setting name! partNum is 0 The operation has completed successfully. Setting name! partNum is 1 Warning: The kernel is still using the old partition table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8) The operation has completed successfully. Setting name! partNum is 0 The operation has completed successfully. Setting name! partNum is 2 Warning: The kernel is still using the old partition table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8) The operation has completed successfully. peta1 /report/{ "failed_tasks": [ "core_scripts_admin_add_osd_journal_err" ], "nod_num": "", "success": true } 01/02/2021 12:07:47 INFO 10.10.0.152 Setting name! partNum is 0 The operation has completed successfully. Setting name! partNum is 0 The operation has completed successfully. Setting name! partNum is 0 The operation has completed successfully. Setting name! partNum is 0 The operation has completed successfully. Setting name! partNum is 1 Warning: The kernel is still using the old partition table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8) The operation has completed successfully. Setting name! partNum is 0 The operation has completed successfully. Setting name! partNum is 2 Warning: The kernel is still using the old partition table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8) The operation has completed successfully. peta2 /report/{ "failed_tasks": [ "core_scripts_admin_add_osd_journal_err" ], "nod_num": "", "success": true } 01/02/2021 12:07:47 INFO Main core components deployed. 01/02/2021 12:07:47 INFO Starting all services. 01/02/2021 12:07:47 INFO Starting cluster file sync service 01/02/2021 12:07:48 INFO Starting Cluster Management application 01/02/2021 12:07:48 INFO Starting activating PetaSAN lvs 01/02/2021 12:07:48 INFO Running command: /sbin/vgs --noheadings --readonly --units=b --nosuffix --separator=";" -o vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_count,vg_extent_size 01/02/2021 12:07:48 INFO LeaderElectionBase dropping old sessions 01/02/2021 12:07:48 INFO stdout ceph-4ed4c220-9498-4351-8ce9-4e00f8b3a93b";"1";"1";"wz--n-";"476931";"0";"4194304 01/02/2021 12:07:48 INFO stdout ceph-d4270832-4236-4724-a362-2dc9fb139f6f";"1";"1";"wz--n-";"476931";"0";"4194304 01/02/2021 12:07:48 INFO stdout ceph-f3285994-1fed-4a78-a067-6141ffe2d08e";"1";"1";"wz--n-";"476931";"0";"4194304 01/02/2021 12:07:48 INFO Starting activating PetaSAN lvs 01/02/2021 12:07:48 INFO Running command: /sbin/vgs --noheadings --readonly --units=b --nosuffix --separator=";" -o vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_count,vg_extent_size 01/02/2021 12:07:48 INFO stdout ceph-4ed4c220-9498-4351-8ce9-4e00f8b3a93b";"1";"1";"wz--n-";"476931";"0";"4194304 01/02/2021 12:07:48 INFO stdout ceph-d4270832-4236-4724-a362-2dc9fb139f6f";"1";"1";"wz--n-";"476931";"0";"4194304 01/02/2021 12:07:48 INFO stdout ceph-f3285994-1fed-4a78-a067-6141ffe2d08e";"1";"1";"wz--n-";"476931";"0";"4194304 01/02/2021 12:07:49 INFO Starting Node Stats Service 01/02/2021 12:07:49 INFO Starting OSDs 01/02/2021 12:07:49 INFO Starting petasan tuning service 01/02/2021 12:07:49 INFO Starting petasan-qperf server 01/02/2021 12:07:58 INFO LeaderElectionBase successfully dropped old sessions 01/02/2021 12:08:03 INFO ClusterLeader start action 01/02/2021 12:08:10 INFO Begin commit_file, key: /etc/hosts 01/02/2021 12:08:10 INFO Begin reading key: /etc/hosts 01/02/2021 12:08:10 INFO Committed 01/02/2021 12:08:58 ERROR GlusterFS setup_cluster failed to disable nfs on volume gfs-vol 01/02/2021 12:08:58 INFO Creating Default Pools. 01/02/2021 12:08:58 INFO Creating default rbd pool. 01/02/2021 12:09:02 INFO Creating default cephfs pools. 01/02/2021 12:09:18 INFO GlusterFS mount attempt 01/02/2021 12:09:22 INFO all pools are active 01/02/2021 12:09:27 INFO Creating default cephfs . 01/02/2021 12:09:27 INFO Creating default cephfs filesystem. 01/02/2021 12:09:27 INFO ======= ceph fs dump output before cephfs ===========. 01/02/2021 12:09:28 INFO e5 enable_multiple, ever_enabled_multiple: 0,0 compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} legacy client fscid: -1 No filesystems configured Standby daemons: [mds.peta1{-1:14127} state up:standby seq 1 addr [v2:172.16.0.151:6800/1224330380,v1:172.16.0.151:6801/1224330380]] [mds.peta2{-1:14173} state up:standby seq 1 addr [v2:172.16.0.152:6800/3906864352,v1:172.16.0.152:6801/3906864352]] [mds.peta3{-1:14203} state up:standby seq 2 addr [v2:172.16.0.153:6802/4122103059,v1:172.16.0.153:6803/4122103059]] 01/02/2021 12:09:28 INFO ======= ==================== ===========. 01/02/2021 12:09:28 INFO ======= ceph fs dump output after cephfs ===========. 01/02/2021 12:09:29 INFO e6 enable_multiple, ever_enabled_multiple: 0,0 compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} legacy client fscid: 1 Filesystem 'cephfs' (1) fs_name cephfs epoch 6 flags 12 created 2021-02-01 12:09:28.533059 modified 2021-02-01 12:09:28.533138 tableserver 0 root 0 session_timeout 60 session_autoclose 300 max_file_size 1099511627776 min_compat_client -1 (unspecified) last_failure 0 last_failure_osd_epoch 0 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} max_mds 1 in 0 up {0=14127} failed damaged stopped data_pools [2] metadata_pool 3 inline_data disabled balancer standby_count_wanted 0 [mds.peta1{0:14127} state up:creating seq 1 addr [v2:172.16.0.151:6800/1224330380,v1:172.16.0.151:6801/1224330380]] Standby daemons: [mds.peta2{-1:14173} state up:standby seq 1 addr [v2:172.16.0.152:6800/3906864352,v1:172.16.0.152:6801/3906864352]] [mds.peta3{-1:14203} state up:standby seq 2 addr [v2:172.16.0.153:6802/4122103059,v1:172.16.0.153:6803/4122103059]] 01/02/2021 12:09:29 INFO ======= ==================== ===========. 01/02/2021 12:09:33 INFO ClusterLeader starting services 01/02/2021 12:09:44 INFO attempt 1 mounting cephfs. 01/02/2021 12:09:44 INFO successfully mounted default cephfs . 01/02/2021 12:09:44 INFO Creating EC Profiles. 01/02/2021 12:09:49 INFO Waiting for ceph to reach active and clean status. 01/02/2021 12:09:51 INFO all pools are active 01/02/2021 12:09:51 INFO Generate PetaSAN Certificate. 01/02/2021 12:09:53 INFO certificates has been created successfully. 01/02/2021 12:09:54 INFO Generate Nginx Configuration. 01/02/2021 12:09:56 INFO Run post deploy script. 01/02/2021 12:10:01 INFO console.py process is 707 01/02/2021 12:10:01 INFO Trying to stop console.py process 01/02/2021 12:10:04 INFO console.py process stopped 01/02/2021 12:10:04 INFO console.py process is 712 01/02/2021 12:10:04 INFO console.py process stopped 01/02/2021 12:10:04 INFO console.py process is 713 01/02/2021 12:10:04 INFO console.py process stopped 01/02/2021 12:10:04 INFO console.py process is 10201 01/02/2021 12:10:04 INFO Trying to stop console.py process 01/02/2021 12:10:07 INFO console.py process is 10202 01/02/2021 12:10:14 INFO Node 3 added and cluster is now ready. 01/02/2021 12:14:21 INFO script 01/02/2021 12:14:21 INFO /opt/petasan/scripts/admin/node_manage_disks.py add-osd 01/02/2021 12:14:21 INFO params 01/02/2021 12:14:21 INFO -disk_name sdd 01/02/2021 12:14:21 INFO Start add osd job for disk sdd. 01/02/2021 12:14:24 INFO Start cleaning disk : sdd 01/02/2021 12:14:24 INFO Executing : wipefs --all /dev/sdd 01/02/2021 12:14:24 INFO Executing : dd if=/dev/zero of=/dev/sdd bs=1M count=20 oflag=direct,dsync >/dev/null 2>&1 01/02/2021 12:14:25 INFO Executing : parted -s /dev/sdd mklabel gpt 01/02/2021 12:14:25 INFO Executing : partprobe /dev/sdd 01/02/2021 12:14:28 INFO User didn't select a journal for disk sdd, so the journal will be on the same disk. 01/02/2021 12:14:28 INFO User didn't select a cache for disk sdd. 01/02/2021 12:14:28 INFO User selected cache disk for disk sdd. 01/02/2021 12:14:28 INFO Start prepare bluestore OSD : sdd 01/02/2021 12:14:28 INFO Running command: /sbin/lvs --noheadings --readonly --separator=";" -a -o lv_tags,lv_path,lv_name,vg_name,lv_uuid,lv_size 01/02/2021 12:14:28 INFO stdout ceph.block_device=/dev/ceph-4ed4c220-9498-4351-8ce9-4e00f8b3a93b/osd-block-931f14e6-b378-4fd9-ab9a-f54fdae6fdd3,ceph.block_uuid=0DcZlj-lyyN-JoGs-Bu1Z-VHzP-HIMV-xjYFcq,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=c74c6e06-673f-4d0c-8235-563559e08858,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/sdf2,ceph.db_uuid=d986e67d-09f4-4b29-a43d-311338eca48a,ceph.encrypted=0,ceph.osd_fsid=931f14e6-b378-4fd9-ab9a-f54fdae6fdd3,ceph.osd_id=1,ceph.osdspec_affinity=,ceph.type=block,ceph.vdo=0";"/dev/ceph-4ed4c220-9498-4351-8ce9-4e00f8b3a93b/osd-block-931f14e6-b378-4fd9-ab9a-f54fdae6fdd3";"osd-block-931f14e6-b378-4fd9-ab9a-f54fdae6fdd3";"ceph-4ed4c220-9498-4351-8ce9-4e00f8b3a93b";"0DcZlj-lyyN-JoGs-Bu1Z-VHzP-HIMV-xjYFcq";"<1.82t 01/02/2021 12:14:28 INFO stdout ceph.block_device=/dev/ceph-d4270832-4236-4724-a362-2dc9fb139f6f/osd-block-bdb04a29-703d-442b-acfb-05fb9fb920b7,ceph.block_uuid=peWdkc-SOzG-aq3c-xjRL-KAtL-8kWu-43je3o,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=c74c6e06-673f-4d0c-8235-563559e08858,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/sdf1,ceph.db_uuid=337d0154-26b8-4b50-b979-d1c6438137cf,ceph.encrypted=0,ceph.osd_fsid=bdb04a29-703d-442b-acfb-05fb9fb920b7,ceph.osd_id=0,ceph.osdspec_affinity=,ceph.type=block,ceph.vdo=0";"/dev/ceph-d4270832-4236-4724-a362-2dc9fb139f6f/osd-block-bdb04a29-703d-442b-acfb-05fb9fb920b7";"osd-block-bdb04a29-703d-442b-acfb-05fb9fb920b7";"ceph-d4270832-4236-4724-a362-2dc9fb139f6f";"peWdkc-SOzG-aq3c-xjRL-KAtL-8kWu-43je3o";"<1.82t 01/02/2021 12:14:28 INFO stdout ceph.block_device=/dev/ceph-f3285994-1fed-4a78-a067-6141ffe2d08e/osd-block-68c4f029-cdb4-4c6b-909b-4b8981844f5a,ceph.block_uuid=1ZEWl9-GttB-i6bQ-791F-FsdN-KhLD-a8suPw,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=c74c6e06-673f-4d0c-8235-563559e08858,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/sdf3,ceph.db_uuid=64f1e865-618c-4fcd-a8a9-c3ee651d24bc,ceph.encrypted=0,ceph.osd_fsid=68c4f029-cdb4-4c6b-909b-4b8981844f5a,ceph.osd_id=2,ceph.osdspec_affinity=,ceph.type=block,ceph.vdo=0";"/dev/ceph-f3285994-1fed-4a78-a067-6141ffe2d08e/osd-block-68c4f029-cdb4-4c6b-909b-4b8981844f5a";"osd-block-68c4f029-cdb4-4c6b-909b-4b8981844f5a";"ceph-f3285994-1fed-4a78-a067-6141ffe2d08e";"1ZEWl9-GttB-i6bQ-791F-FsdN-KhLD-a8suPw";"<1.82t 01/02/2021 12:14:28 INFO Creating data partition num 1 size 0 on /dev/sdd 01/02/2021 12:14:30 INFO Calling partprobe on sdd device 01/02/2021 12:14:30 INFO Executing partprobe /dev/sdd 01/02/2021 12:14:30 INFO Calling udevadm on sdd device 01/02/2021 12:14:30 INFO Executing udevadm settle --timeout 30 01/02/2021 12:14:33 INFO Starting : ceph-volume --log-path /opt/petasan/log lvm prepare --bluestore --data /dev/sdd1 01/02/2021 12:14:39 INFO starting activate osd. 01/02/2021 12:14:40 INFO Starting : ceph-volume --log-path /opt/petasan/log lvm activate --bluestore 11 ccf76a5c-e076-43a1-835d-1d50049ab75c 01/02/2021 12:21:47 INFO script 01/02/2021 12:21:47 INFO /opt/petasan/scripts/util/benchmark.py storage 01/02/2021 12:21:47 INFO params 01/02/2021 12:21:47 INFO -d 15 01/02/2021 12:21:47 INFO Start storage load job for 'sar' 01/02/2021 12:21:48 INFO Benchmark storage cmd. 01/02/2021 12:22:21 INFO script 01/02/2021 12:22:21 INFO /opt/petasan/scripts/util/benchmark.py storage 01/02/2021 12:22:21 INFO params 01/02/2021 12:22:21 INFO -d 15 01/02/2021 12:22:21 INFO Start storage load job for 'sar' 01/02/2021 12:22:22 INFO Benchmark storage cmd. 01/02/2021 12:59:27 INFO Start settings IPs 01/02/2021 12:59:33 INFO GlusterFS mount attempt 01/02/2021 12:59:41 INFO str_start_command: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consul agent -raft-protocol 2 -config-dir /opt/petasan/config/etc/consul.d/server -bind 172.16.0.153 -retry-join 172.16.0.151 -retry-join 172.16.0.152 01/02/2021 12:59:44 INFO Starting cluster file sync service 01/02/2021 12:59:45 INFO LeaderElectionBase dropping old sessions 01/02/2021 12:59:45 INFO Starting Cluster Management application 01/02/2021 12:59:45 INFO Starting Node Stats Service 01/02/2021 12:59:45 INFO Starting activating PetaSAN lvs 01/02/2021 12:59:45 INFO Running command: /sbin/vgs --noheadings --readonly --units=b --nosuffix --separator=";" -o vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_count,vg_extent_size 01/02/2021 12:59:45 INFO stdout ceph-09209f5b-5f36-4ad4-82ff-17c0bfb7f60d";"1";"1";"wz--n-";"476931";"0";"4194304 01/02/2021 12:59:45 INFO stdout ceph-4ed4c220-9498-4351-8ce9-4e00f8b3a93b";"1";"1";"wz--n-";"476931";"0";"4194304 01/02/2021 12:59:45 INFO stdout ceph-d4270832-4236-4724-a362-2dc9fb139f6f";"1";"1";"wz--n-";"476931";"0";"4194304 01/02/2021 12:59:45 INFO stdout ceph-f3285994-1fed-4a78-a067-6141ffe2d08e";"1";"1";"wz--n-";"476931";"0";"4194304 01/02/2021 12:59:45 INFO Starting activating PetaSAN lvs 01/02/2021 12:59:45 INFO Running command: /sbin/vgs --noheadings --readonly --units=b --nosuffix --separator=";" -o vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_count,vg_extent_size 01/02/2021 12:59:45 INFO stdout ceph-09209f5b-5f36-4ad4-82ff-17c0bfb7f60d";"1";"1";"wz--n-";"476931";"0";"4194304 01/02/2021 12:59:45 INFO stdout ceph-4ed4c220-9498-4351-8ce9-4e00f8b3a93b";"1";"1";"wz--n-";"476931";"0";"4194304 01/02/2021 12:59:45 INFO stdout ceph-d4270832-4236-4724-a362-2dc9fb139f6f";"1";"1";"wz--n-";"476931";"0";"4194304 01/02/2021 12:59:45 INFO stdout ceph-f3285994-1fed-4a78-a067-6141ffe2d08e";"1";"1";"wz--n-";"476931";"0";"4194304 01/02/2021 12:59:47 INFO Starting OSDs 01/02/2021 12:59:47 INFO Starting petasan tuning service 01/02/2021 12:59:47 INFO Starting Config Upload service 01/02/2021 12:59:47 INFO Starting petasan-qperf server 01/02/2021 13:00:00 INFO Checking backend latencies : 01/02/2021 13:00:00 INFO Network latency for backend 172.16.0.151 = 89.8 us 01/02/2021 13:00:00 INFO Network latency for backend 172.16.0.152 = 87.6 us 01/02/2021 13:00:00 INFO Network latency for backend 172.16.0.153 = 27.5 us 01/02/2021 13:00:02 INFO Checking backend latencies : 01/02/2021 13:00:02 INFO Network latency for backend 172.16.0.151 = 86.2 us 01/02/2021 13:00:02 INFO Network latency for backend 172.16.0.152 = 91.0 us 01/02/2021 13:00:02 INFO Network latency for backend 172.16.0.153 = 12.6 us 01/02/2021 13:00:08 INFO Checking backend latencies : 01/02/2021 13:00:08 INFO Network latency for backend 172.16.0.151 = 67.3 us 01/02/2021 13:00:08 INFO Network latency for backend 172.16.0.152 = 64.0 us 01/02/2021 13:00:08 INFO Network latency for backend 172.16.0.153 = 7.76 us 01/02/2021 13:00:10 INFO GlusterFS mount attempt 01/02/2021 13:00:15 INFO Checking backend latencies : 01/02/2021 13:00:15 INFO Network latency for backend 172.16.0.151 = 53.5 us 01/02/2021 13:00:15 INFO Network latency for backend 172.16.0.152 = 50.3 us 01/02/2021 13:00:15 INFO Network latency for backend 172.16.0.153 = 8.25 us 01/02/2021 13:00:16 INFO LeaderElectionBase successfully dropped old sessions 01/02/2021 13:00:17 INFO Checking backend latencies : 01/02/2021 13:00:17 INFO Network latency for backend 172.16.0.151 = 50.2 us 01/02/2021 13:00:17 INFO Network latency for backend 172.16.0.152 = 51.5 us 01/02/2021 13:00:17 INFO Network latency for backend 172.16.0.153 = 7.47 us 01/02/2021 13:01:38 INFO ClusterLeader start action 01/02/2021 13:01:38 INFO ClusterLeader starting services 01/02/2021 13:01:46 ERROR Node Stats exception. 01/02/2021 13:01:46 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.5 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 10.49 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1341.44 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 71.68 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.18 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 88217.6 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 220549.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.17 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 81315.84 `date +%s`" | nc -q0 10.10.0.152 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.5 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 10.49 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1341.44 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 71.68 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.18 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 88217.6 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 220549.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.17 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 81315.84 `date +%s`" | nc -q0 10.10.0.152 2003 01/02/2021 13:03:25 INFO Checking backend latencies : 01/02/2021 13:03:25 INFO Network latency for backend 172.16.0.151 = 01/02/2021 13:03:25 INFO Network latency for backend 172.16.0.152 = 45.6 us 01/02/2021 13:03:25 INFO Network latency for backend 172.16.0.153 = 8.29 us 01/02/2021 13:03:27 INFO Checking backend latencies : 01/02/2021 13:03:27 INFO Network latency for backend 172.16.0.151 = 01/02/2021 13:03:27 INFO Network latency for backend 172.16.0.152 = 50.6 us 01/02/2021 13:03:27 INFO Network latency for backend 172.16.0.153 = 7.48 us 01/02/2021 13:03:27 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 01/02/2021 16:08:47 ERROR Node Stats exception. 01/02/2021 16:08:47 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 3.7 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 17.43 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1955.84 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 194.56 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.17 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 213104.64 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 132986.88 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 200765.44 `date +%s`" | nc -q0 10.10.0.153 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 3.7 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 17.43 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1955.84 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 194.56 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.17 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 213104.64 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 132986.88 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 200765.44 `date +%s`" | nc -q0 10.10.0.153 2003 01/02/2021 16:26:49 INFO Start settings IPs 01/02/2021 16:26:55 INFO GlusterFS mount attempt 01/02/2021 16:27:04 INFO str_start_command: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consul agent -raft-protocol 2 -config-dir /opt/petasan/config/etc/consul.d/server -bind 172.16.0.153 -retry-join 172.16.0.151 -retry-join 172.16.0.152 01/02/2021 16:27:07 INFO Starting cluster file sync service 01/02/2021 16:27:08 INFO LeaderElectionBase dropping old sessions 01/02/2021 16:27:08 INFO Starting Cluster Management application 01/02/2021 16:27:08 INFO Starting Node Stats Service 01/02/2021 16:27:08 INFO Starting activating PetaSAN lvs 01/02/2021 16:27:08 INFO Running command: /sbin/vgs --noheadings --readonly --units=b --nosuffix --separator=";" -o vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_count,vg_extent_size 01/02/2021 16:27:08 INFO stdout ceph-09209f5b-5f36-4ad4-82ff-17c0bfb7f60d";"1";"1";"wz--n-";"476931";"0";"4194304 01/02/2021 16:27:08 INFO stdout ceph-4ed4c220-9498-4351-8ce9-4e00f8b3a93b";"1";"1";"wz--n-";"476931";"0";"4194304 01/02/2021 16:27:08 INFO stdout ceph-d4270832-4236-4724-a362-2dc9fb139f6f";"1";"1";"wz--n-";"476931";"0";"4194304 01/02/2021 16:27:08 INFO stdout ceph-f3285994-1fed-4a78-a067-6141ffe2d08e";"1";"1";"wz--n-";"476931";"0";"4194304 01/02/2021 16:27:08 INFO Starting activating PetaSAN lvs 01/02/2021 16:27:08 INFO Running command: /sbin/vgs --noheadings --readonly --units=b --nosuffix --separator=";" -o vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_count,vg_extent_size 01/02/2021 16:27:08 INFO stdout ceph-09209f5b-5f36-4ad4-82ff-17c0bfb7f60d";"1";"1";"wz--n-";"476931";"0";"4194304 01/02/2021 16:27:08 INFO stdout ceph-4ed4c220-9498-4351-8ce9-4e00f8b3a93b";"1";"1";"wz--n-";"476931";"0";"4194304 01/02/2021 16:27:08 INFO stdout ceph-d4270832-4236-4724-a362-2dc9fb139f6f";"1";"1";"wz--n-";"476931";"0";"4194304 01/02/2021 16:27:08 INFO stdout ceph-f3285994-1fed-4a78-a067-6141ffe2d08e";"1";"1";"wz--n-";"476931";"0";"4194304 01/02/2021 16:27:10 INFO Starting OSDs 01/02/2021 16:27:10 INFO Starting petasan tuning service 01/02/2021 16:27:10 INFO Starting Config Upload service 01/02/2021 16:27:10 INFO Starting petasan-qperf server 01/02/2021 16:27:22 INFO Checking backend latencies : 01/02/2021 16:27:22 INFO Network latency for backend 172.16.0.151 = 90.5 us 01/02/2021 16:27:22 INFO Network latency for backend 172.16.0.152 = 94.6 us 01/02/2021 16:27:22 INFO Network latency for backend 172.16.0.153 = 14.7 us 01/02/2021 16:27:24 INFO Checking backend latencies : 01/02/2021 16:27:24 INFO Network latency for backend 172.16.0.151 = 101.0 us 01/02/2021 16:27:24 INFO Network latency for backend 172.16.0.152 = 66.7 us 01/02/2021 16:27:24 INFO Network latency for backend 172.16.0.153 = 12.0 us 01/02/2021 16:27:31 INFO Checking backend latencies : 01/02/2021 16:27:31 INFO Network latency for backend 172.16.0.151 = 80.1 us 01/02/2021 16:27:31 INFO Network latency for backend 172.16.0.152 = 60.7 us 01/02/2021 16:27:31 INFO Network latency for backend 172.16.0.153 = 7.48 us 01/02/2021 16:27:32 INFO GlusterFS mount attempt 01/02/2021 16:27:37 INFO LeaderElectionBase successfully dropped old sessions 01/02/2021 16:27:37 INFO Checking backend latencies : 01/02/2021 16:27:37 INFO Network latency for backend 172.16.0.151 = 54.1 us 01/02/2021 16:27:37 INFO Network latency for backend 172.16.0.152 = 50.7 us 01/02/2021 16:27:37 INFO Network latency for backend 172.16.0.153 = 8.33 us 01/02/2021 16:27:39 INFO Checking backend latencies : 01/02/2021 16:27:39 INFO Network latency for backend 172.16.0.151 = 51.3 us 01/02/2021 16:27:39 INFO Network latency for backend 172.16.0.152 = 51.5 us 01/02/2021 16:27:39 INFO Network latency for backend 172.16.0.153 = 7.37 us 01/02/2021 16:31:23 INFO Checking backend latencies : 01/02/2021 16:31:23 INFO Network latency for backend 172.16.0.151 = 01/02/2021 16:31:23 INFO Network latency for backend 172.16.0.152 = 48.7 us 01/02/2021 16:31:23 INFO Network latency for backend 172.16.0.153 = 7.79 us 01/02/2021 16:31:23 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 01/02/2021 16:31:25 INFO Checking backend latencies : 01/02/2021 16:31:25 INFO Network latency for backend 172.16.0.151 = 01/02/2021 16:31:25 INFO Network latency for backend 172.16.0.152 = 55.3 us 01/02/2021 16:31:25 INFO Network latency for backend 172.16.0.153 = 7.72 us 01/02/2021 16:31:52 INFO ClusterLeader start action 01/02/2021 16:31:52 INFO ClusterLeader starting services 01/02/2021 16:32:15 ERROR Node Stats exception. 01/02/2021 16:32:15 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 92.31 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 12.15 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 5406.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 4976.64 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.46 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 578652.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 363048.96 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.44 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 545464.32 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 92.31 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 12.15 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 5406.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 4976.64 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.46 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 578652.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 363048.96 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.44 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 545464.32 `date +%s`" | nc -q0 10.10.0.151 2003 01/02/2021 19:55:28 INFO GlusterFS mount attempt 01/02/2021 19:55:31 ERROR Node Stats exception. 01/02/2021 19:55:31 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 77.96 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 10.18 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1320.96 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.38 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 475432.96 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 309923.84 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.36 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 449198.08 `date +%s`" | nc -q0 10.10.0.153 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 77.96 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 10.18 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1320.96 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.38 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 475432.96 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 309923.84 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.36 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 449198.08 `date +%s`" | nc -q0 10.10.0.153 2003 02/02/2021 03:53:44 INFO Start settings IPs 02/02/2021 03:53:50 INFO GlusterFS mount attempt 02/02/2021 03:53:59 INFO str_start_command: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consul agent -raft-protocol 2 -config-dir /opt/petasan/config/etc/consul.d/server -bind 172.16.0.153 -retry-join 172.16.0.151 -retry-join 172.16.0.152 02/02/2021 03:54:02 INFO Starting cluster file sync service 02/02/2021 03:54:02 INFO Starting Cluster Management application 02/02/2021 03:54:03 INFO Starting Node Stats Service 02/02/2021 03:54:03 INFO LeaderElectionBase dropping old sessions 02/02/2021 03:54:03 INFO Starting activating PetaSAN lvs 02/02/2021 03:54:03 INFO Running command: /sbin/vgs --noheadings --readonly --units=b --nosuffix --separator=";" -o vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_count,vg_extent_size 02/02/2021 03:54:03 INFO stdout ceph-09209f5b-5f36-4ad4-82ff-17c0bfb7f60d";"1";"1";"wz--n-";"476931";"0";"4194304 02/02/2021 03:54:03 INFO stdout ceph-4ed4c220-9498-4351-8ce9-4e00f8b3a93b";"1";"1";"wz--n-";"476931";"0";"4194304 02/02/2021 03:54:03 INFO stdout ceph-d4270832-4236-4724-a362-2dc9fb139f6f";"1";"1";"wz--n-";"476931";"0";"4194304 02/02/2021 03:54:03 INFO stdout ceph-f3285994-1fed-4a78-a067-6141ffe2d08e";"1";"1";"wz--n-";"476931";"0";"4194304 02/02/2021 03:54:03 INFO Starting activating PetaSAN lvs 02/02/2021 03:54:03 INFO Running command: /sbin/vgs --noheadings --readonly --units=b --nosuffix --separator=";" -o vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_count,vg_extent_size 02/02/2021 03:54:03 INFO stdout ceph-09209f5b-5f36-4ad4-82ff-17c0bfb7f60d";"1";"1";"wz--n-";"476931";"0";"4194304 02/02/2021 03:54:03 INFO stdout ceph-4ed4c220-9498-4351-8ce9-4e00f8b3a93b";"1";"1";"wz--n-";"476931";"0";"4194304 02/02/2021 03:54:03 INFO stdout ceph-d4270832-4236-4724-a362-2dc9fb139f6f";"1";"1";"wz--n-";"476931";"0";"4194304 02/02/2021 03:54:03 INFO stdout ceph-f3285994-1fed-4a78-a067-6141ffe2d08e";"1";"1";"wz--n-";"476931";"0";"4194304 02/02/2021 03:54:04 INFO Starting OSDs 02/02/2021 03:54:05 INFO Starting petasan tuning service 02/02/2021 03:54:05 INFO Starting Config Upload service 02/02/2021 03:54:05 INFO Starting petasan-qperf server 02/02/2021 03:54:18 INFO Checking backend latencies : 02/02/2021 03:54:18 INFO Network latency for backend 172.16.0.151 = 83.0 us 02/02/2021 03:54:18 INFO Network latency for backend 172.16.0.152 = 83.1 us 02/02/2021 03:54:18 INFO Network latency for backend 172.16.0.153 = 12.9 us 02/02/2021 03:54:20 INFO Checking backend latencies : 02/02/2021 03:54:20 INFO Network latency for backend 172.16.0.151 = 81.8 us 02/02/2021 03:54:20 INFO Network latency for backend 172.16.0.152 = 67.4 us 02/02/2021 03:54:20 INFO Network latency for backend 172.16.0.153 = 9.7 us 02/02/2021 03:54:26 INFO Checking backend latencies : 02/02/2021 03:54:26 INFO Network latency for backend 172.16.0.151 = 85.1 us 02/02/2021 03:54:26 INFO Network latency for backend 172.16.0.152 = 67.7 us 02/02/2021 03:54:26 INFO Network latency for backend 172.16.0.153 = 7.76 us 02/02/2021 03:54:27 INFO GlusterFS mount attempt 02/02/2021 03:54:28 INFO LeaderElectionBase successfully dropped old sessions 02/02/2021 03:58:19 INFO Checking backend latencies : 02/02/2021 03:58:19 INFO Network latency for backend 172.16.0.151 = 54.0 us 02/02/2021 03:58:19 INFO Network latency for backend 172.16.0.152 = 02/02/2021 03:58:19 INFO Network latency for backend 172.16.0.153 = 6.96 us 02/02/2021 03:58:21 INFO Checking backend latencies : 02/02/2021 03:58:21 INFO Network latency for backend 172.16.0.151 = 50.9 us 02/02/2021 03:58:21 INFO Network latency for backend 172.16.0.152 = 02/02/2021 03:58:21 INFO Network latency for backend 172.16.0.153 = 7.4 us 02/02/2021 03:58:21 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 02/02/2021 04:02:06 ERROR Node Stats exception. 02/02/2021 04:02:06 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.84 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 9.82 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1269.76 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 399.36 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.05 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 59484.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 58798.08 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.04 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 55408.64 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.84 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 9.82 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1269.76 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 399.36 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.05 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 59484.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 58798.08 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.04 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 55408.64 `date +%s`" | nc -q0 10.10.0.151 2003 02/02/2021 06:25:06 INFO GlusterFS mount attempt 02/02/2021 06:51:52 INFO PetaSAN certificate has been synced 02/02/2021 07:21:42 ERROR Node Stats exception. 02/02/2021 07:21:42 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 3.49 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.61 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1075.2 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 92.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.22 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 279326.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 191252.48 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.21 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 264140.8 `date +%s`" | nc -q0 10.10.0.152 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 3.49 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.61 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1075.2 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 92.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.22 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 279326.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 191252.48 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.21 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 264140.8 `date +%s`" | nc -q0 10.10.0.152 2003 02/02/2021 08:15:34 ERROR Node Stats exception. 02/02/2021 08:15:34 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.84 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 16.03 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 993.28 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.19 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 238950.4 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 179425.28 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.18 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 225177.6 `date +%s`" | nc -q0 10.10.0.152 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.84 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 16.03 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 993.28 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.19 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 238950.4 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 179425.28 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.18 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 225177.6 `date +%s`" | nc -q0 10.10.0.152 2003 02/02/2021 09:28:26 ERROR Node Stats exception. 02/02/2021 09:28:26 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 2.01 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 16.87 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 870.4 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.23 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 221491.2 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 283617.28 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.22 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 207851.52 `date +%s`" | nc -q0 10.10.0.152 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 2.01 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 16.87 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 870.4 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.23 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 221491.2 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 283617.28 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.22 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 207851.52 `date +%s`" | nc -q0 10.10.0.152 2003 02/02/2021 09:29:29 ERROR Node Stats exception. 02/02/2021 09:29:29 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 2.0 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 16.88 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1064.96 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.2 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 251904.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 200335.36 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.19 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 237721.6 `date +%s`" | nc -q0 10.10.0.152 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 2.0 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 16.88 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1064.96 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.2 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 251904.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 200335.36 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.19 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 237721.6 `date +%s`" | nc -q0 10.10.0.152 2003 02/02/2021 09:30:32 ERROR Node Stats exception. 02/02/2021 09:30:32 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 0.84 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 16.88 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1064.96 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.24 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 304435.2 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 174827.52 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.23 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 288286.72 `date +%s`" | nc -q0 10.10.0.152 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 0.84 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 16.88 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1064.96 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.24 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 304435.2 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 174827.52 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.23 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 288286.72 `date +%s`" | nc -q0 10.10.0.152 2003 02/02/2021 09:31:37 ERROR Node Stats exception. 02/02/2021 09:31:37 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.disks.smart.sda.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sda.power_on_hours 84250 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sda.temperature_celsius 32 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.power_on_hours 84251 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.temperature_celsius 29 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.power_on_hours 20415 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.temperature_celsius 28 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdd.reallocated_sector_ct 0 `date +%s`" | nc -q0 10.10.0.152 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.disks.smart.sda.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sda.power_on_hours 84250 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sda.temperature_celsius 32 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.power_on_hours 84251 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.temperature_celsius 29 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.power_on_hours 20415 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.temperature_celsius 28 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdd.reallocated_sector_ct 0 `date +%s`" | nc -q0 10.10.0.152 2003 02/02/2021 09:32:40 ERROR Node Stats exception. 02/02/2021 09:32:40 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.0 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 16.96 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1290.24 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.17 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 212387.84 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 131328.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 200355.84 `date +%s`" | nc -q0 10.10.0.152 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.0 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 16.96 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1290.24 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.17 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 212387.84 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 131328.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 200355.84 `date +%s`" | nc -q0 10.10.0.152 2003 02/02/2021 09:33:43 ERROR Node Stats exception. 02/02/2021 09:33:43 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 2.69 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 16.96 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1904.64 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.25 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 241827.84 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 313815.04 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.24 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 227112.96 `date +%s`" | nc -q0 10.10.0.152 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 2.69 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 16.96 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1904.64 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.25 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 241827.84 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 313815.04 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.24 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 227112.96 `date +%s`" | nc -q0 10.10.0.152 2003 02/02/2021 09:34:46 ERROR Node Stats exception. 02/02/2021 09:34:46 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.33 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 17.01 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1126.4 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 200632.32 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 119265.28 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.15 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 189491.2 `date +%s`" | nc -q0 10.10.0.152 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.33 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 17.01 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1126.4 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 200632.32 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 119265.28 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.15 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 189491.2 `date +%s`" | nc -q0 10.10.0.152 2003 02/02/2021 09:35:50 ERROR Node Stats exception. 02/02/2021 09:35:50 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.18 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 17.04 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 890.88 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.17 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 214773.76 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 133406.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 202383.36 `date +%s`" | nc -q0 10.10.0.152 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.18 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 17.04 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 890.88 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.17 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 214773.76 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 133406.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 202383.36 `date +%s`" | nc -q0 10.10.0.152 2003 02/02/2021 10:41:18 ERROR Node Stats exception. 02/02/2021 10:41:18 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 2.17 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 18.04 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 901.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.24 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 304384.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 196106.24 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.23 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 287754.24 `date +%s`" | nc -q0 10.10.0.152 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 2.17 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 18.04 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 901.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.24 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 304384.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 196106.24 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.23 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 287754.24 `date +%s`" | nc -q0 10.10.0.152 2003 02/02/2021 10:42:21 ERROR Node Stats exception. 02/02/2021 10:42:21 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.85 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 18.05 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 972.8 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.2 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 250091.52 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 213596.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.19 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 235038.72 `date +%s`" | nc -q0 10.10.0.152 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.85 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 18.05 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 972.8 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.2 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 250091.52 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 213596.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.19 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 235038.72 `date +%s`" | nc -q0 10.10.0.152 2003 02/02/2021 16:36:08 ERROR Node Stats exception. 02/02/2021 16:36:08 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 2.33 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 23.32 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1177.6 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.26 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 324894.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 151132.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.25 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 308060.16 `date +%s`" | nc -q0 10.10.0.152 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 2.33 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 23.32 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1177.6 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.26 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 324894.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 151132.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.25 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 308060.16 `date +%s`" | nc -q0 10.10.0.152 2003 03/02/2021 06:25:03 INFO GlusterFS mount attempt 03/02/2021 06:44:12 INFO Update roles. 03/02/2021 06:44:12 INFO Update node iscsi role to true 03/02/2021 06:44:12 INFO Starting PetaSAN-iscsi service 03/02/2021 06:44:13 INFO checking uptime 03/02/2021 06:44:13 INFO checking uptime of node: peta1 ip:10.10.0.151 03/02/2021 06:44:13 INFO checking uptime of node: peta2 ip:10.10.0.152 03/02/2021 06:44:13 INFO checking uptime of node: peta3 ip:10.10.0.153 03/02/2021 06:44:13 INFO Service is starting. 03/02/2021 06:44:13 INFO iSCSI Service dropping old sessions 03/02/2021 06:44:13 INFO sync_replication_node called on non-backup node 03/02/2021 06:44:28 INFO iSCSI Service successfully dropped old sessions 03/02/2021 06:44:28 INFO Cleaning unused configurations. 03/02/2021 06:44:28 INFO Cleaning all mapped disks 03/02/2021 06:44:28 INFO Cleaning unused rbd images. 03/02/2021 06:44:28 INFO Cleaning unused ips. 03/02/2021 06:45:28 INFO Found pool:rbd for disk:00001 via consul 03/02/2021 06:45:28 INFO Image image-00001 mapped successfully. 03/02/2021 06:45:31 INFO LIO add_target() disk wwn is c74c6e0600001 03/02/2021 06:45:31 INFO Path 00001/3 acquired successfully 03/02/2021 06:47:11 INFO Cleaned disk path 00001/3. 03/02/2021 06:47:11 INFO PetaSAN cleaned local paths not locked by this node in consul. 03/02/2021 06:47:11 INFO Stopping disk 00001 03/02/2021 06:47:11 ERROR Could not find ips for image-00001 03/02/2021 06:47:11 INFO LIO deleted backstore image image-00001 03/02/2021 06:47:11 INFO LIO deleted Target iqn.2016-05.com.petasan:00001 03/02/2021 06:47:11 INFO Image image-00001 unmapped successfully. 03/02/2021 06:47:13 INFO Found pool:rbd for disk:00001 via consul 03/02/2021 06:47:13 INFO PetaSAN removed key of stopped disk 00001 from consul. 03/02/2021 06:50:47 INFO Found pool:rbd for disk:00001 via consul 03/02/2021 06:50:47 INFO Image image-00001 mapped successfully. 03/02/2021 06:50:50 INFO LIO add_target() disk wwn is c74c6e0600001 03/02/2021 06:50:50 INFO Path 00001/2 acquired successfully 03/02/2021 07:01:46 INFO Checking backend latencies : 03/02/2021 07:01:46 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:01:46 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:01:46 INFO Network latency for backend 172.16.0.153 = 7.47 us 03/02/2021 07:01:50 INFO Checking backend latencies : 03/02/2021 07:01:50 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:01:50 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:01:50 INFO Network latency for backend 172.16.0.153 = 7.47 us 03/02/2021 07:02:07 INFO Checking backend latencies : 03/02/2021 07:02:07 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:02:07 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:02:07 INFO Network latency for backend 172.16.0.153 = 7.51 us 03/02/2021 07:02:11 INFO Checking backend latencies : 03/02/2021 07:02:11 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:02:11 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:02:11 INFO Network latency for backend 172.16.0.153 = 7.47 us 03/02/2021 07:02:31 INFO Checking backend latencies : 03/02/2021 07:02:31 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:02:31 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:02:31 INFO Network latency for backend 172.16.0.153 = 7.48 us 03/02/2021 07:02:34 INFO Checking backend latencies : 03/02/2021 07:02:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:02:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:02:34 INFO Network latency for backend 172.16.0.153 = 7.54 us 03/02/2021 07:02:58 INFO Checking backend latencies : 03/02/2021 07:02:58 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:02:58 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:02:58 INFO Network latency for backend 172.16.0.153 = 7.5 us 03/02/2021 07:03:01 INFO Checking backend latencies : 03/02/2021 07:03:01 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:03:01 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:03:01 INFO Network latency for backend 172.16.0.153 = 7.5 us 03/02/2021 07:03:33 INFO Checking backend latencies : 03/02/2021 07:03:33 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:03:33 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:03:33 INFO Network latency for backend 172.16.0.153 = 7.49 us 03/02/2021 07:03:33 ERROR Could not find key resource. 03/02/2021 07:03:33 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 496, in get_key_blocking index, data = consul_obj.get(key, lock_index, False, timeout) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:03:36 INFO Checking backend latencies : 03/02/2021 07:03:36 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:03:36 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:03:36 INFO Network latency for backend 172.16.0.153 = 7.46 us 03/02/2021 07:03:36 ERROR Error on consul connection. 03/02/2021 07:03:36 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/iscsi_service.py", line 158, in start self.__current_lock_index = consul_api.current_index() File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 348, in current_index raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 345, in current_index index = consul_obj.get(self.root)[0] File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:03:52 INFO Checking backend latencies : 03/02/2021 07:03:52 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:03:52 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:03:52 INFO Network latency for backend 172.16.0.153 = 7.52 us 03/02/2021 07:04:05 INFO Checking backend latencies : 03/02/2021 07:04:05 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:05 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:05 INFO Network latency for backend 172.16.0.153 = 7.52 us 03/02/2021 07:04:13 ERROR Node Stats exception. 03/02/2021 07:04:13 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 24.12 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 25.4 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1300.48 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 102.4 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 0.0 `date +%s`" | nc -q0 10.10.0.152 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 24.12 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 25.4 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1300.48 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 102.4 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 0.0 `date +%s`" | nc -q0 10.10.0.152 2003 03/02/2021 07:04:13 INFO Checking backend latencies : 03/02/2021 07:04:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:13 INFO Network latency for backend 172.16.0.153 = 7.53 us 03/02/2021 07:04:27 INFO Checking backend latencies : 03/02/2021 07:04:27 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:27 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:27 INFO Network latency for backend 172.16.0.153 = 7.51 us 03/02/2021 07:04:32 INFO Checking backend latencies : 03/02/2021 07:04:32 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:32 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:32 INFO Network latency for backend 172.16.0.153 = 7.56 us 03/02/2021 07:04:37 INFO Checking backend latencies : 03/02/2021 07:04:37 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:37 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:37 INFO Network latency for backend 172.16.0.153 = 7.53 us 03/02/2021 07:04:50 INFO Checking backend latencies : 03/02/2021 07:04:50 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:50 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:50 INFO Network latency for backend 172.16.0.153 = 7.53 us 03/02/2021 07:04:53 INFO Checking backend latencies : 03/02/2021 07:04:53 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:04:53 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:04:53 INFO Network latency for backend 172.16.0.153 = 7.51 us 03/02/2021 07:05:04 INFO Checking backend latencies : 03/02/2021 07:05:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:05:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:05:04 INFO Network latency for backend 172.16.0.153 = 7.58 us 03/02/2021 07:05:09 INFO Checking backend latencies : 03/02/2021 07:05:09 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:05:09 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:05:09 INFO Network latency for backend 172.16.0.153 = 7.43 us 03/02/2021 07:05:16 INFO Checking backend latencies : 03/02/2021 07:05:16 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:05:16 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:05:16 INFO Network latency for backend 172.16.0.153 = 7.44 us 03/02/2021 07:05:18 INFO Checking backend latencies : 03/02/2021 07:05:18 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:05:18 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:05:18 INFO Network latency for backend 172.16.0.153 = 7.57 us 03/02/2021 07:05:30 INFO Checking backend latencies : 03/02/2021 07:05:30 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:05:30 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:05:30 INFO Network latency for backend 172.16.0.153 = 7.46 us 03/02/2021 07:05:39 INFO Checking backend latencies : 03/02/2021 07:05:39 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:05:39 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:05:39 INFO Network latency for backend 172.16.0.153 = 7.48 us 03/02/2021 07:05:39 ERROR LeaderElectionBase exception 03/02/2021 07:05:39 ERROR 03/02/2021 07:05:43 INFO Checking backend latencies : 03/02/2021 07:05:43 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:05:43 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:05:43 INFO Network latency for backend 172.16.0.153 = 7.56 us 03/02/2021 07:05:53 INFO Checking backend latencies : 03/02/2021 07:05:53 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:05:53 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:05:53 INFO Network latency for backend 172.16.0.153 = 7.61 us 03/02/2021 07:05:53 ERROR Error on consul connection. 03/02/2021 07:05:53 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/iscsi_service.py", line 158, in start self.__current_lock_index = consul_api.current_index() File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 348, in current_index raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 345, in current_index index = consul_obj.get(self.root)[0] File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:05:55 INFO Checking backend latencies : 03/02/2021 07:05:55 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:05:55 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:05:55 INFO Network latency for backend 172.16.0.153 = 7.41 us 03/02/2021 07:06:08 INFO Checking backend latencies : 03/02/2021 07:06:08 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:08 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:08 INFO Network latency for backend 172.16.0.153 = 7.52 us 03/02/2021 07:06:18 INFO Checking backend latencies : 03/02/2021 07:06:18 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:18 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:18 INFO Network latency for backend 172.16.0.153 = 7.4 us 03/02/2021 07:06:18 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader 03/02/2021 07:06:18 ERROR Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 24.12 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 25.4 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1300.48 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 102.4 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 0.0 `date +%s`" | nc -q0 10.10.0.152 2003 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 516, in get_key index, data = cons.get(key) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:06:22 INFO Checking backend latencies : 03/02/2021 07:06:22 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:22 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:22 INFO Network latency for backend 172.16.0.153 = 7.52 us 03/02/2021 07:06:29 INFO Checking backend latencies : 03/02/2021 07:06:29 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:29 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:29 INFO Network latency for backend 172.16.0.153 = 7.52 us 03/02/2021 07:06:32 INFO Checking backend latencies : 03/02/2021 07:06:32 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:32 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:32 INFO Network latency for backend 172.16.0.153 = 7.46 us 03/02/2021 07:06:52 INFO Checking backend latencies : 03/02/2021 07:06:52 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:52 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:52 INFO Network latency for backend 172.16.0.153 = 7.55 us 03/02/2021 07:06:54 INFO Checking backend latencies : 03/02/2021 07:06:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:54 INFO Network latency for backend 172.16.0.153 = 7.61 us 03/02/2021 07:06:58 INFO Checking backend latencies : 03/02/2021 07:06:58 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:06:58 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:06:58 INFO Network latency for backend 172.16.0.153 = 7.53 us 03/02/2021 07:06:58 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:07:18 INFO Checking backend latencies : 03/02/2021 07:07:18 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:07:18 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:07:18 INFO Network latency for backend 172.16.0.153 = 7.58 us 03/02/2021 07:07:20 INFO Checking backend latencies : 03/02/2021 07:07:20 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:07:20 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:07:20 INFO Network latency for backend 172.16.0.153 = 7.51 us 03/02/2021 07:07:22 INFO Checking backend latencies : 03/02/2021 07:07:22 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:07:22 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:07:22 INFO Network latency for backend 172.16.0.153 = 7.44 us 03/02/2021 07:07:37 INFO Checking backend latencies : 03/02/2021 07:07:37 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:07:37 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:07:37 INFO Network latency for backend 172.16.0.153 = 7.45 us 03/02/2021 07:07:41 INFO Checking backend latencies : 03/02/2021 07:07:41 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:07:41 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:07:41 INFO Network latency for backend 172.16.0.153 = 7.48 us 03/02/2021 07:07:45 INFO Checking backend latencies : 03/02/2021 07:07:45 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:07:45 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:07:45 INFO Network latency for backend 172.16.0.153 = 7.47 us 03/02/2021 07:07:57 INFO Checking backend latencies : 03/02/2021 07:07:57 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:07:57 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:07:57 INFO Network latency for backend 172.16.0.153 = 7.59 us 03/02/2021 07:07:57 ERROR Could not find key resource. 03/02/2021 07:07:57 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 496, in get_key_blocking index, data = consul_obj.get(key, lock_index, False, timeout) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:07:59 INFO Checking backend latencies : 03/02/2021 07:07:59 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:07:59 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:07:59 INFO Network latency for backend 172.16.0.153 = 7.53 us 03/02/2021 07:08:04 INFO Checking backend latencies : 03/02/2021 07:08:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:04 INFO Network latency for backend 172.16.0.153 = 7.54 us 03/02/2021 07:08:16 INFO Checking backend latencies : 03/02/2021 07:08:16 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:16 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:16 INFO Network latency for backend 172.16.0.153 = 7.49 us 03/02/2021 07:08:20 INFO Checking backend latencies : 03/02/2021 07:08:20 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:20 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:20 INFO Network latency for backend 172.16.0.153 = 7.42 us 03/02/2021 07:08:20 ERROR Error on consul connection. 03/02/2021 07:08:20 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/iscsi_service.py", line 158, in start self.__current_lock_index = consul_api.current_index() File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 348, in current_index raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 345, in current_index index = consul_obj.get(self.root)[0] File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:08:22 INFO Checking backend latencies : 03/02/2021 07:08:22 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:22 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:22 INFO Network latency for backend 172.16.0.153 = 7.58 us 03/02/2021 07:08:31 INFO Checking backend latencies : 03/02/2021 07:08:31 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:31 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:31 INFO Network latency for backend 172.16.0.153 = 7.5 us 03/02/2021 07:08:35 WARNING PetaSAN could not complete process, there are too many exceptions. 03/02/2021 07:08:37 INFO Checking backend latencies : 03/02/2021 07:08:37 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:37 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:37 INFO Network latency for backend 172.16.0.153 = 7.54 us 03/02/2021 07:08:49 INFO Checking backend latencies : 03/02/2021 07:08:49 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:49 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:49 INFO Network latency for backend 172.16.0.153 = 7.48 us 03/02/2021 07:08:56 INFO Checking backend latencies : 03/02/2021 07:08:56 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:08:56 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:08:56 INFO Network latency for backend 172.16.0.153 = 7.54 us 03/02/2021 07:09:00 INFO Checking backend latencies : 03/02/2021 07:09:00 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:09:00 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:09:00 INFO Network latency for backend 172.16.0.153 = 7.53 us 03/02/2021 07:09:07 INFO Checking backend latencies : 03/02/2021 07:09:07 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:09:07 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:09:07 INFO Network latency for backend 172.16.0.153 = 7.48 us 03/02/2021 07:09:07 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:09:17 INFO Checking backend latencies : 03/02/2021 07:09:17 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:09:17 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:09:17 INFO Network latency for backend 172.16.0.153 = 7.55 us 03/02/2021 07:09:25 INFO Checking backend latencies : 03/02/2021 07:09:25 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:09:25 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:09:25 INFO Network latency for backend 172.16.0.153 = 7.4 us 03/02/2021 07:09:25 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader 03/02/2021 07:09:25 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 516, in get_key index, data = cons.get(key) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:09:27 INFO Checking backend latencies : 03/02/2021 07:09:27 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:09:27 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:09:27 INFO Network latency for backend 172.16.0.153 = 7.54 us 03/02/2021 07:09:29 INFO Checking backend latencies : 03/02/2021 07:09:29 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:09:29 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:09:29 INFO Network latency for backend 172.16.0.153 = 7.51 us 03/02/2021 07:09:40 INFO Checking backend latencies : 03/02/2021 07:09:40 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:09:40 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:09:40 INFO Network latency for backend 172.16.0.153 = 7.47 us 03/02/2021 07:09:51 INFO Checking backend latencies : 03/02/2021 07:09:51 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:09:51 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:09:51 INFO Network latency for backend 172.16.0.153 = 7.46 us 03/02/2021 07:10:05 INFO Checking backend latencies : 03/02/2021 07:10:05 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:10:05 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:10:05 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:10:05 ERROR LeaderElectionBase exception 03/02/2021 07:10:05 ERROR 03/02/2021 07:10:10 INFO Checking backend latencies : 03/02/2021 07:10:10 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:10:10 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:10:10 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:10:15 INFO ClusterLeader start action 03/02/2021 07:10:15 INFO ClusterLeader starting services 03/02/2021 07:10:36 INFO Found pool:rbd for disk:00001 via consul 03/02/2021 07:10:36 INFO The path 00001/1 was locked by peta2. 03/02/2021 07:10:37 INFO This node will stop node peta2/10.10.0.152. 03/02/2021 07:10:52 INFO Path 00001/1 acquired successfully 03/02/2021 07:12:47 INFO Checking backend latencies : 03/02/2021 07:12:47 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:12:47 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:12:47 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:12:48 INFO Checking backend latencies : 03/02/2021 07:12:48 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:12:48 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:12:48 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:12:54 INFO Checking backend latencies : 03/02/2021 07:12:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:12:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:12:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:12:54 INFO Checking backend latencies : 03/02/2021 07:12:54 INFO Checking backend latencies : 03/02/2021 07:12:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:12:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:12:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:12:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:12:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:12:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:12:54 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:13:12 INFO Checking backend latencies : 03/02/2021 07:13:12 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:13:12 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:13:12 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:13:12 INFO Checking backend latencies : 03/02/2021 07:13:12 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:13:12 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:13:12 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:13:18 INFO Checking backend latencies : 03/02/2021 07:13:18 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:13:18 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:13:18 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:13:18 INFO Checking backend latencies : 03/02/2021 07:13:18 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:13:18 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:13:18 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:13:18 INFO Checking backend latencies : 03/02/2021 07:13:18 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:13:18 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:13:18 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:13:38 INFO Checking backend latencies : 03/02/2021 07:13:38 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:13:38 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:13:38 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:13:38 INFO Checking backend latencies : 03/02/2021 07:13:38 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:13:38 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:13:38 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:13:43 INFO Checking backend latencies : 03/02/2021 07:13:43 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:13:43 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:13:43 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:13:44 INFO Checking backend latencies : 03/02/2021 07:13:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:13:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:13:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:13:44 INFO Checking backend latencies : 03/02/2021 07:13:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:13:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:13:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:14:08 INFO Checking backend latencies : 03/02/2021 07:14:08 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:14:08 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:14:08 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:14:08 INFO Checking backend latencies : 03/02/2021 07:14:08 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:14:08 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:14:08 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:14:09 INFO Checking backend latencies : 03/02/2021 07:14:09 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:14:09 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:14:09 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:14:14 INFO Checking backend latencies : 03/02/2021 07:14:14 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:14:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:14:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:14:14 ERROR Could not start watch. 03/02/2021 07:14:14 INFO Checking backend latencies : 03/02/2021 07:14:14 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:14:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:14:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:14:38 INFO Checking backend latencies : 03/02/2021 07:14:38 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:14:38 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:14:38 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:14:39 INFO Checking backend latencies : 03/02/2021 07:14:39 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:14:39 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:14:39 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:14:46 INFO Checking backend latencies : 03/02/2021 07:14:46 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:14:46 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:14:46 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:14:46 ERROR object of type 'NoneType' has no len() Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/notification/plugins/check/replication.py", line 39, in run if len(failed_jobs) > 0: TypeError: object of type 'NoneType' has no len() 03/02/2021 07:14:46 ERROR An error occurred while ReplicationNotificationPlugin was running. 03/02/2021 07:14:46 INFO Checking backend latencies : 03/02/2021 07:14:46 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:14:46 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:14:46 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:14:46 ERROR error get down node list Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/notification/plugins/check/node.py", line 45, in __get_down_node_list node_list = con_api.get_node_list() File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 432, in get_node_list index, data = consul_obj.get(ConfigAPI().get_consul_nodes_path(), recurse=True) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:14:53 INFO Checking backend latencies : 03/02/2021 07:14:53 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:14:53 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:14:53 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:14:53 ERROR Could not find key resource. 03/02/2021 07:14:53 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 496, in get_key_blocking index, data = consul_obj.get(key, lock_index, False, timeout) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:14:53 INFO ClusterLeader stop action 03/02/2021 07:15:03 INFO Checking backend latencies : 03/02/2021 07:15:03 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:15:03 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:15:03 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:15:17 INFO Checking backend latencies : 03/02/2021 07:15:17 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:15:17 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:15:17 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:15:17 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:15:29 INFO Checking backend latencies : 03/02/2021 07:15:29 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:15:29 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:15:29 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:15:42 INFO Checking backend latencies : 03/02/2021 07:15:42 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:15:42 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:15:42 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:15:46 ERROR Node Stats exception. 03/02/2021 07:15:46 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.disks.smart.sda.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sda.power_on_hours 84272 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sda.temperature_celsius 31 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.power_on_hours 84273 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.temperature_celsius 28 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.power_on_hours 20436 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.temperature_celsius 27 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdd.reallocated_sector_ct 0 `date +%s`" | nc -q0 10.10.0.153 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.disks.smart.sda.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sda.power_on_hours 84272 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sda.temperature_celsius 31 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.power_on_hours 84273 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.temperature_celsius 28 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.power_on_hours 20436 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.temperature_celsius 27 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdd.reallocated_sector_ct 0 `date +%s`" | nc -q0 10.10.0.153 2003 03/02/2021 07:15:59 INFO Checking backend latencies : 03/02/2021 07:15:59 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:15:59 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:15:59 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:06 INFO Checking backend latencies : 03/02/2021 07:16:06 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:06 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:06 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:08 INFO Checking backend latencies : 03/02/2021 07:16:08 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:08 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:08 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:32 INFO Checking backend latencies : 03/02/2021 07:16:32 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:32 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:32 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:33 INFO Checking backend latencies : 03/02/2021 07:16:33 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:33 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:33 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:37 INFO Checking backend latencies : 03/02/2021 07:16:37 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:37 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:37 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:37 ERROR Error on consul connection. 03/02/2021 07:16:37 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/iscsi_service.py", line 158, in start self.__current_lock_index = consul_api.current_index() File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 348, in current_index raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 345, in current_index index = consul_obj.get(self.root)[0] File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:16:50 INFO Checking backend latencies : 03/02/2021 07:16:50 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:50 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:50 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:16:59 INFO Checking backend latencies : 03/02/2021 07:16:59 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:16:59 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:16:59 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:02 INFO Checking backend latencies : 03/02/2021 07:17:02 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:02 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:02 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:09 INFO Checking backend latencies : 03/02/2021 07:17:09 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:09 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:09 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:15 INFO Checking backend latencies : 03/02/2021 07:17:15 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:15 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:15 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:29 INFO Checking backend latencies : 03/02/2021 07:17:29 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:29 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:29 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:33 INFO Checking backend latencies : 03/02/2021 07:17:33 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:33 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:33 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:40 INFO Checking backend latencies : 03/02/2021 07:17:40 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:40 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:40 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:40 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:17:41 INFO Checking backend latencies : 03/02/2021 07:17:41 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:41 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:41 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:46 INFO Checking backend latencies : 03/02/2021 07:17:46 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:46 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:46 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:17:59 INFO Checking backend latencies : 03/02/2021 07:17:59 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:17:59 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:17:59 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:18:04 INFO Checking backend latencies : 03/02/2021 07:18:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:18:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:18:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:18:07 INFO Checking backend latencies : 03/02/2021 07:18:07 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:18:07 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:18:07 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:18:07 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader 03/02/2021 07:18:07 ERROR Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.disks.smart.sda.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sda.power_on_hours 84272 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sda.temperature_celsius 31 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.power_on_hours 84273 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.temperature_celsius 28 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.power_on_hours 20436 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.temperature_celsius 27 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdd.reallocated_sector_ct 0 `date +%s`" | nc -q0 10.10.0.153 2003 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 516, in get_key index, data = cons.get(key) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:18:10 INFO Checking backend latencies : 03/02/2021 07:18:10 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:18:10 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:18:10 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:18:11 INFO Checking backend latencies : 03/02/2021 07:18:11 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:18:11 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:18:11 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:18:28 INFO Checking backend latencies : 03/02/2021 07:18:28 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:18:28 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:18:28 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:18:30 INFO Checking backend latencies : 03/02/2021 07:18:30 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:18:30 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:18:30 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:18:36 INFO Checking backend latencies : 03/02/2021 07:18:36 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:18:36 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:18:36 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:18:49 INFO Checking backend latencies : 03/02/2021 07:18:49 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:18:49 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:18:49 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:18:49 ERROR Could not find key resource. 03/02/2021 07:18:49 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 496, in get_key_blocking index, data = consul_obj.get(key, lock_index, False, timeout) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:18:55 INFO Checking backend latencies : 03/02/2021 07:18:55 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:18:55 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:18:55 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:19:06 INFO Checking backend latencies : 03/02/2021 07:19:06 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:19:06 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:19:06 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:19:08 INFO Checking backend latencies : 03/02/2021 07:19:08 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:19:08 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:19:08 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:19:08 ERROR Error on consul connection. 03/02/2021 07:19:08 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/iscsi_service.py", line 158, in start self.__current_lock_index = consul_api.current_index() File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 348, in current_index raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 345, in current_index index = consul_obj.get(self.root)[0] File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:19:11 INFO Checking backend latencies : 03/02/2021 07:19:11 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:19:11 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:19:11 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:19:25 INFO Checking backend latencies : 03/02/2021 07:19:25 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:19:25 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:19:25 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:19:29 INFO Checking backend latencies : 03/02/2021 07:19:29 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:19:29 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:19:29 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:19:36 INFO Checking backend latencies : 03/02/2021 07:19:36 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:19:36 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:19:36 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:19:44 INFO Checking backend latencies : 03/02/2021 07:19:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:19:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:19:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:19:44 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:19:50 INFO Checking backend latencies : 03/02/2021 07:19:50 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:19:50 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:19:50 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:19:54 INFO Checking backend latencies : 03/02/2021 07:19:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:19:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:19:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:20:02 INFO Checking backend latencies : 03/02/2021 07:20:02 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:20:02 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:20:02 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:20:03 INFO Checking backend latencies : 03/02/2021 07:20:03 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:20:03 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:20:03 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:20:03 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:20:14 INFO Checking backend latencies : 03/02/2021 07:20:14 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:20:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:20:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:20:20 INFO Checking backend latencies : 03/02/2021 07:20:20 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:20:20 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:20:20 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:20:27 INFO Checking backend latencies : 03/02/2021 07:20:27 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:20:27 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:20:27 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:20:32 INFO Checking backend latencies : 03/02/2021 07:20:32 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:20:32 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:20:32 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:20:41 INFO Checking backend latencies : 03/02/2021 07:20:41 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:20:41 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:20:41 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:20:50 INFO Checking backend latencies : 03/02/2021 07:20:50 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:20:50 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:20:50 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:20:51 INFO Checking backend latencies : 03/02/2021 07:20:51 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:20:51 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:20:51 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:21:10 INFO Checking backend latencies : 03/02/2021 07:21:10 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:21:10 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:21:10 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:21:10 ERROR LeaderElectionBase exception 03/02/2021 07:21:10 ERROR 03/02/2021 07:21:11 INFO Checking backend latencies : 03/02/2021 07:21:11 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:21:11 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:21:11 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:21:18 INFO Checking backend latencies : 03/02/2021 07:21:18 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:21:18 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:21:18 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:21:28 INFO Checking backend latencies : 03/02/2021 07:21:28 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:21:28 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:21:28 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:21:28 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader 03/02/2021 07:21:28 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 516, in get_key index, data = cons.get(key) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:21:43 INFO Checking backend latencies : 03/02/2021 07:21:43 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:21:43 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:21:43 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:21:48 INFO Checking backend latencies : 03/02/2021 07:21:48 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:21:48 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:21:48 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:21:49 INFO Checking backend latencies : 03/02/2021 07:21:49 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:21:49 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:21:49 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:21:49 ERROR Error on consul connection. 03/02/2021 07:21:49 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/iscsi_service.py", line 158, in start self.__current_lock_index = consul_api.current_index() File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 348, in current_index raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 345, in current_index index = consul_obj.get(self.root)[0] File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:22:04 WARNING PetaSAN could not complete process, there are too many exceptions. 03/02/2021 07:22:07 INFO Checking backend latencies : 03/02/2021 07:22:07 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:22:07 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:22:07 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:22:26 INFO Checking backend latencies : 03/02/2021 07:22:26 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:22:26 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:22:26 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:22:26 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:22:27 INFO Checking backend latencies : 03/02/2021 07:22:27 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:22:27 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:22:27 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:22:33 INFO Checking backend latencies : 03/02/2021 07:22:33 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:22:33 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:22:33 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:22:50 INFO Checking backend latencies : 03/02/2021 07:22:50 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:22:50 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:22:50 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:22:50 INFO Checking backend latencies : 03/02/2021 07:22:50 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:22:50 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:22:50 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:22:51 INFO Checking backend latencies : 03/02/2021 07:22:51 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:22:51 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:22:51 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:23:03 INFO Checking backend latencies : 03/02/2021 07:23:03 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:23:03 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:23:03 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:23:14 INFO Checking backend latencies : 03/02/2021 07:23:14 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:23:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:23:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:23:14 INFO Checking backend latencies : 03/02/2021 07:23:14 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:23:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:23:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:23:18 INFO Checking backend latencies : 03/02/2021 07:23:18 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:23:18 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:23:18 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:23:40 INFO Checking backend latencies : 03/02/2021 07:23:40 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:23:40 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:23:40 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:23:40 INFO Checking backend latencies : 03/02/2021 07:23:40 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:23:40 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:23:40 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:23:41 INFO Checking backend latencies : 03/02/2021 07:23:41 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:23:41 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:23:41 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:23:41 ERROR Could not find key resource. 03/02/2021 07:23:41 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 496, in get_key_blocking index, data = consul_obj.get(key, lock_index, False, timeout) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:23:48 INFO Checking backend latencies : 03/02/2021 07:23:48 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:23:48 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:23:48 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:24:03 INFO Checking backend latencies : 03/02/2021 07:24:03 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:24:03 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:24:03 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:24:10 INFO Checking backend latencies : 03/02/2021 07:24:10 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:24:10 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:24:10 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:24:11 INFO Checking backend latencies : 03/02/2021 07:24:11 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:24:11 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:24:11 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:24:26 INFO Checking backend latencies : 03/02/2021 07:24:26 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:24:26 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:24:26 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:24:26 ERROR Error on consul connection. 03/02/2021 07:24:26 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/iscsi_service.py", line 158, in start self.__current_lock_index = consul_api.current_index() File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 348, in current_index raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 345, in current_index index = consul_obj.get(self.root)[0] File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:24:27 INFO Checking backend latencies : 03/02/2021 07:24:27 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:24:27 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:24:27 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:24:38 WARNING There are too many exceptions.Service will clean this node. 03/02/2021 07:24:38 INFO Cleaning unused configurations. 03/02/2021 07:24:38 INFO Cleaning all mapped disks 03/02/2021 07:24:48 INFO Checking backend latencies : 03/02/2021 07:24:48 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:24:48 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:24:48 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:24:48 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:24:49 INFO Checking backend latencies : 03/02/2021 07:24:49 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:24:49 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:24:49 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:24:49 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader 03/02/2021 07:24:49 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 516, in get_key index, data = cons.get(key) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:24:53 INFO Checking backend latencies : 03/02/2021 07:24:53 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:24:53 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:24:53 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:25:00 INFO Checking backend latencies : 03/02/2021 07:25:00 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:25:00 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:25:00 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:25:13 INFO Checking backend latencies : 03/02/2021 07:25:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:25:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:25:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:25:24 INFO Checking backend latencies : 03/02/2021 07:25:24 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:25:24 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:25:24 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:25:24 INFO Checking backend latencies : 03/02/2021 07:25:24 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:25:24 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:25:24 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:25:37 INFO Checking backend latencies : 03/02/2021 07:25:37 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:25:37 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:25:37 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:25:50 INFO Checking backend latencies : 03/02/2021 07:25:50 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:25:50 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:25:50 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:26:02 INFO Checking backend latencies : 03/02/2021 07:26:02 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:26:02 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:26:02 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:26:02 ERROR LeaderElectionBase exception 03/02/2021 07:26:02 ERROR 03/02/2021 07:26:03 INFO Checking backend latencies : 03/02/2021 07:26:03 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:26:03 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:26:03 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:26:11 INFO Checking backend latencies : 03/02/2021 07:26:11 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:26:11 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:26:11 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:26:21 INFO Checking backend latencies : 03/02/2021 07:26:21 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:26:21 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:26:21 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:26:33 INFO Checking backend latencies : 03/02/2021 07:26:33 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:26:33 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:26:33 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:26:34 INFO Checking backend latencies : 03/02/2021 07:26:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:26:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:26:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:26:35 INFO Checking backend latencies : 03/02/2021 07:26:35 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:26:35 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:26:35 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:26:58 INFO Checking backend latencies : 03/02/2021 07:26:58 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:26:58 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:26:58 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:26:59 INFO Checking backend latencies : 03/02/2021 07:26:59 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:26:59 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:26:59 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:26:59 ERROR Error reading application config 03/02/2021 07:27:01 INFO Checking backend latencies : 03/02/2021 07:27:01 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:27:01 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:27:01 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:27:02 INFO checking uptime 03/02/2021 07:27:02 INFO checking uptime of node: peta1 ip:10.10.0.151 03/02/2021 07:27:03 INFO checking uptime of node: peta2 ip:10.10.0.152 03/02/2021 07:27:11 INFO Checking backend latencies : 03/02/2021 07:27:11 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:27:11 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:27:11 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:27:11 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:27:23 ERROR checking uptime of node: peta2 ping error to ip: 10.10.0.152 03/02/2021 07:27:23 INFO checking uptime of node: peta3 ip:10.10.0.153 03/02/2021 07:27:23 INFO Service is starting. 03/02/2021 07:27:23 INFO iSCSI Service dropping old sessions 03/02/2021 07:27:24 INFO Checking backend latencies : 03/02/2021 07:27:24 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:27:24 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:27:24 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:27:31 INFO Checking backend latencies : 03/02/2021 07:27:31 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:27:31 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:27:31 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:27:35 INFO Checking backend latencies : 03/02/2021 07:27:35 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:27:35 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:27:35 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:27:54 INFO Checking backend latencies : 03/02/2021 07:27:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:27:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:27:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:27:59 INFO Checking backend latencies : 03/02/2021 07:27:59 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:27:59 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:27:59 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:28:00 INFO Checking backend latencies : 03/02/2021 07:28:00 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:28:00 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:28:00 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:28:10 INFO Checking backend latencies : 03/02/2021 07:28:10 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:28:10 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:28:10 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:28:10 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader 03/02/2021 07:28:10 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 516, in get_key index, data = cons.get(key) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:28:24 INFO Checking backend latencies : 03/02/2021 07:28:24 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:28:24 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:28:24 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:28:26 INFO Checking backend latencies : 03/02/2021 07:28:26 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:28:26 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:28:26 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:28:33 INFO Checking backend latencies : 03/02/2021 07:28:33 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:28:33 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:28:33 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:28:33 ERROR Could not find key resource. 03/02/2021 07:28:33 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 496, in get_key_blocking index, data = consul_obj.get(key, lock_index, False, timeout) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:28:50 INFO Checking backend latencies : 03/02/2021 07:28:50 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:28:50 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:28:50 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:28:55 INFO Checking backend latencies : 03/02/2021 07:28:55 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:28:55 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:28:55 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:28:56 INFO Checking backend latencies : 03/02/2021 07:28:56 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:28:56 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:28:56 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:29:19 INFO Checking backend latencies : 03/02/2021 07:29:19 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:29:19 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:29:19 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:29:20 INFO Checking backend latencies : 03/02/2021 07:29:20 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:29:20 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:29:20 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:29:32 INFO Checking backend latencies : 03/02/2021 07:29:32 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:29:32 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:29:32 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:29:34 INFO Checking backend latencies : 03/02/2021 07:29:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:29:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:29:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:29:34 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:29:45 INFO Checking backend latencies : 03/02/2021 07:29:45 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:29:45 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:29:45 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:29:56 INFO Checking backend latencies : 03/02/2021 07:29:56 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:29:56 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:29:56 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:29:58 INFO Checking backend latencies : 03/02/2021 07:29:58 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:29:58 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:29:58 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:29:59 INFO Checking backend latencies : 03/02/2021 07:29:59 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:29:59 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:29:59 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:29:59 ERROR iSCSI Service exception dropping old sessions 03/02/2021 07:30:15 INFO Checking backend latencies : 03/02/2021 07:30:15 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:30:15 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:30:15 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:30:22 INFO Checking backend latencies : 03/02/2021 07:30:22 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:30:22 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:30:22 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:30:23 INFO Checking backend latencies : 03/02/2021 07:30:23 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:30:23 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:30:23 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:30:36 INFO Checking backend latencies : 03/02/2021 07:30:36 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:30:36 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:30:36 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:30:49 INFO Checking backend latencies : 03/02/2021 07:30:49 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:30:49 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:30:49 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:30:53 INFO Checking backend latencies : 03/02/2021 07:30:53 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:30:53 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:30:53 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:30:53 INFO Checking backend latencies : 03/02/2021 07:30:53 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:30:53 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:30:53 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:30:53 ERROR LeaderElectionBase exception 03/02/2021 07:30:53 ERROR 03/02/2021 07:31:00 INFO Checking backend latencies : 03/02/2021 07:31:00 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:31:00 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:31:00 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:31:19 INFO Checking backend latencies : 03/02/2021 07:31:19 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:31:19 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:31:19 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:31:25 INFO Checking backend latencies : 03/02/2021 07:31:25 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:31:25 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:31:25 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:31:26 INFO Checking backend latencies : 03/02/2021 07:31:26 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:31:26 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:31:26 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:31:31 INFO Checking backend latencies : 03/02/2021 07:31:31 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:31:31 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:31:31 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:31:31 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader 03/02/2021 07:31:31 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 516, in get_key index, data = cons.get(key) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:31:49 INFO Checking backend latencies : 03/02/2021 07:31:49 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:31:49 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:31:49 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:31:56 INFO Checking backend latencies : 03/02/2021 07:31:56 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:31:56 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:31:56 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:31:57 INFO Checking backend latencies : 03/02/2021 07:31:57 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:31:57 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:31:57 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:31:57 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:32:16 INFO Checking backend latencies : 03/02/2021 07:32:16 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:32:16 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:32:16 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:32:21 INFO Checking backend latencies : 03/02/2021 07:32:21 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:32:21 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:32:21 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:32:34 INFO Checking backend latencies : 03/02/2021 07:32:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:32:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:32:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:32:34 ERROR iSCSI Service exception dropping old sessions 03/02/2021 07:32:45 INFO Checking backend latencies : 03/02/2021 07:32:45 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:32:45 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:32:45 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:32:46 INFO Checking backend latencies : 03/02/2021 07:32:46 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:32:46 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:32:46 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:32:53 INFO Checking backend latencies : 03/02/2021 07:32:53 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:32:53 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:32:53 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:33:11 INFO Checking backend latencies : 03/02/2021 07:33:11 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:33:11 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:33:11 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:33:11 INFO Checking backend latencies : 03/02/2021 07:33:11 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:33:11 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:33:11 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:33:17 INFO Checking backend latencies : 03/02/2021 07:33:17 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:33:17 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:33:17 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:33:24 INFO Checking backend latencies : 03/02/2021 07:33:24 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:33:24 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:33:24 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:33:24 ERROR Could not find key resource. 03/02/2021 07:33:24 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 496, in get_key_blocking index, data = consul_obj.get(key, lock_index, False, timeout) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:33:35 INFO Checking backend latencies : 03/02/2021 07:33:35 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:33:35 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:33:35 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:33:41 INFO Checking backend latencies : 03/02/2021 07:33:41 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:33:41 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:33:41 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:33:43 INFO Checking backend latencies : 03/02/2021 07:33:43 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:33:43 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:33:43 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:33:46 INFO Checking backend latencies : 03/02/2021 07:33:46 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:33:46 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:33:46 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:34:01 INFO Checking backend latencies : 03/02/2021 07:34:01 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:34:01 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:34:01 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:34:10 INFO Checking backend latencies : 03/02/2021 07:34:10 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:34:10 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:34:10 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:34:13 INFO Checking backend latencies : 03/02/2021 07:34:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:34:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:34:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:34:20 INFO Checking backend latencies : 03/02/2021 07:34:20 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:34:20 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:34:20 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:34:20 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:34:31 INFO Checking backend latencies : 03/02/2021 07:34:31 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:34:31 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:34:31 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:34:36 INFO Checking backend latencies : 03/02/2021 07:34:36 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:34:36 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:34:36 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:34:44 INFO Checking backend latencies : 03/02/2021 07:34:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:34:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:34:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:34:52 INFO Checking backend latencies : 03/02/2021 07:34:52 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:34:52 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:34:52 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:34:52 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader 03/02/2021 07:34:52 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 516, in get_key index, data = cons.get(key) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:35:07 INFO Checking backend latencies : 03/02/2021 07:35:07 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:35:07 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:35:07 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:35:08 INFO Checking backend latencies : 03/02/2021 07:35:08 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:35:08 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:35:08 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:35:10 INFO Checking backend latencies : 03/02/2021 07:35:10 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:35:10 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:35:10 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:35:10 ERROR iSCSI Service exception dropping old sessions 03/02/2021 07:35:34 INFO Checking backend latencies : 03/02/2021 07:35:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:35:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:35:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:35:45 INFO Checking backend latencies : 03/02/2021 07:35:45 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:35:45 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:35:45 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:35:45 ERROR LeaderElectionBase exception 03/02/2021 07:35:45 ERROR 03/02/2021 07:35:47 INFO Checking backend latencies : 03/02/2021 07:35:47 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:35:47 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:35:47 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:36:04 INFO Checking backend latencies : 03/02/2021 07:36:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:36:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:36:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:36:11 INFO Checking backend latencies : 03/02/2021 07:36:11 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:36:11 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:36:11 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:36:14 INFO Checking backend latencies : 03/02/2021 07:36:14 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:36:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:36:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:36:17 INFO Checking backend latencies : 03/02/2021 07:36:17 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:36:17 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:36:17 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:36:37 INFO Checking backend latencies : 03/02/2021 07:36:37 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:36:37 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:36:37 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:36:38 INFO Checking backend latencies : 03/02/2021 07:36:38 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:36:38 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:36:38 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:36:41 INFO Checking backend latencies : 03/02/2021 07:36:41 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:36:41 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:36:41 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:36:43 INFO Checking backend latencies : 03/02/2021 07:36:43 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:36:43 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:36:43 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:36:43 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:37:04 INFO Checking backend latencies : 03/02/2021 07:37:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:37:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:37:04 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:37:07 INFO Checking backend latencies : 03/02/2021 07:37:07 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:37:07 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:37:07 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:37:07 INFO Checking backend latencies : 03/02/2021 07:37:07 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:37:07 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:37:07 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:37:07 INFO Checking backend latencies : 03/02/2021 07:37:07 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:37:07 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:37:07 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:37:18 INFO Checking backend latencies : 03/02/2021 07:37:18 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:37:18 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:37:18 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:37:31 INFO Checking backend latencies : 03/02/2021 07:37:31 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:37:31 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:37:31 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:37:34 INFO Checking backend latencies : 03/02/2021 07:37:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:37:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:37:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:37:37 INFO Checking backend latencies : 03/02/2021 07:37:37 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:37:37 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:37:37 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:37:42 INFO Checking backend latencies : 03/02/2021 07:37:42 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:37:42 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:37:42 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:37:45 INFO Checking backend latencies : 03/02/2021 07:37:45 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:37:45 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:37:45 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:37:45 ERROR iSCSI Service exception dropping old sessions 03/02/2021 07:37:57 INFO Checking backend latencies : 03/02/2021 07:37:57 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:37:57 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:37:57 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:38:08 INFO Checking backend latencies : 03/02/2021 07:38:08 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:38:08 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:38:08 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:38:12 INFO Checking backend latencies : 03/02/2021 07:38:12 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:38:12 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:38:12 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:38:12 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader 03/02/2021 07:38:12 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 516, in get_key index, data = cons.get(key) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:38:16 INFO Checking backend latencies : 03/02/2021 07:38:16 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:38:16 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:38:16 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:38:16 ERROR Could not find key resource. 03/02/2021 07:38:16 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 496, in get_key_blocking index, data = consul_obj.get(key, lock_index, False, timeout) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:38:22 INFO Checking backend latencies : 03/02/2021 07:38:22 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:38:22 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:38:22 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:38:27 INFO Checking backend latencies : 03/02/2021 07:38:27 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:38:27 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:38:27 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:38:38 INFO Checking backend latencies : 03/02/2021 07:38:38 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:38:38 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:38:38 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:38:38 INFO Checking backend latencies : 03/02/2021 07:38:38 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:38:38 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:38:38 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:38:47 INFO Checking backend latencies : 03/02/2021 07:38:47 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:38:47 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:38:47 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:39:02 INFO Checking backend latencies : 03/02/2021 07:39:02 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:39:02 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:39:02 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:39:05 INFO Checking backend latencies : 03/02/2021 07:39:05 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:39:05 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:39:05 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:39:05 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:39:13 INFO Checking backend latencies : 03/02/2021 07:39:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:39:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:39:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:39:17 INFO Checking backend latencies : 03/02/2021 07:39:17 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:39:17 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:39:17 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:39:17 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:39:28 INFO Checking backend latencies : 03/02/2021 07:39:28 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:39:28 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:39:28 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:39:29 INFO Checking backend latencies : 03/02/2021 07:39:29 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:39:29 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:39:29 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:39:35 INFO Checking backend latencies : 03/02/2021 07:39:35 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:39:35 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:39:35 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:39:43 INFO Checking backend latencies : 03/02/2021 07:39:43 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:39:43 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:39:43 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:39:54 INFO Checking backend latencies : 03/02/2021 07:39:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:39:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:39:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:39:59 INFO Checking backend latencies : 03/02/2021 07:39:59 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:39:59 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:39:59 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:39:59 INFO Checking backend latencies : 03/02/2021 07:39:59 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:39:59 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:39:59 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:40:19 INFO Checking backend latencies : 03/02/2021 07:40:19 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:40:19 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:40:19 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:40:20 INFO Checking backend latencies : 03/02/2021 07:40:20 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:40:20 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:40:20 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:40:21 INFO Checking backend latencies : 03/02/2021 07:40:21 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:40:21 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:40:21 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:40:21 ERROR iSCSI Service exception dropping old sessions 03/02/2021 07:40:25 INFO Checking backend latencies : 03/02/2021 07:40:25 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:40:25 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:40:25 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:40:37 INFO Checking backend latencies : 03/02/2021 07:40:37 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:40:37 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:40:37 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:40:37 ERROR LeaderElectionBase exception 03/02/2021 07:40:37 ERROR 03/02/2021 07:40:43 INFO Checking backend latencies : 03/02/2021 07:40:43 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:40:43 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:40:43 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:40:50 INFO Checking backend latencies : 03/02/2021 07:40:50 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:40:50 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:40:50 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:40:55 INFO Checking backend latencies : 03/02/2021 07:40:55 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:40:55 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:40:55 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:40:58 INFO Checking backend latencies : 03/02/2021 07:40:58 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:40:58 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:40:58 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:41:09 INFO Checking backend latencies : 03/02/2021 07:41:09 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:41:09 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:41:09 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:41:10 INFO Checking backend latencies : 03/02/2021 07:41:10 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:41:10 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:41:10 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:41:22 INFO Checking backend latencies : 03/02/2021 07:41:22 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:41:22 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:41:22 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:41:29 INFO Checking backend latencies : 03/02/2021 07:41:29 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:41:29 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:41:29 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:41:29 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:41:33 INFO Checking backend latencies : 03/02/2021 07:41:33 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:41:33 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:41:33 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:41:33 INFO Checking backend latencies : 03/02/2021 07:41:33 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:41:33 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:41:33 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:41:33 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader 03/02/2021 07:41:33 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 516, in get_key index, data = cons.get(key) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:41:40 INFO Checking backend latencies : 03/02/2021 07:41:40 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:41:40 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:41:40 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:41:49 INFO Checking backend latencies : 03/02/2021 07:41:49 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:41:49 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:41:49 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:41:53 INFO Checking backend latencies : 03/02/2021 07:41:53 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:41:53 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:41:53 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:41:59 INFO Checking backend latencies : 03/02/2021 07:41:59 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:41:59 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:41:59 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:42:17 INFO Checking backend latencies : 03/02/2021 07:42:17 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:42:17 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:42:17 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:42:18 INFO Checking backend latencies : 03/02/2021 07:42:18 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:42:18 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:42:18 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:42:18 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:42:19 INFO Checking backend latencies : 03/02/2021 07:42:19 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:42:19 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:42:19 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:42:29 INFO Checking backend latencies : 03/02/2021 07:42:29 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:42:29 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:42:29 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:42:43 INFO Checking backend latencies : 03/02/2021 07:42:43 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:42:43 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:42:43 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:42:55 INFO Checking backend latencies : 03/02/2021 07:42:55 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:42:55 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:42:55 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:42:57 INFO Checking backend latencies : 03/02/2021 07:42:57 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:42:57 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:42:57 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:42:57 ERROR iSCSI Service exception dropping old sessions 03/02/2021 07:43:07 INFO Checking backend latencies : 03/02/2021 07:43:07 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:43:07 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:43:07 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:43:07 ERROR Could not find key resource. 03/02/2021 07:43:07 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 496, in get_key_blocking index, data = consul_obj.get(key, lock_index, False, timeout) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:43:13 INFO Checking backend latencies : 03/02/2021 07:43:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:43:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:43:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:43:20 INFO Checking backend latencies : 03/02/2021 07:43:20 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:43:20 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:43:20 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:43:22 INFO Checking backend latencies : 03/02/2021 07:43:22 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:43:22 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:43:22 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:43:30 INFO Checking backend latencies : 03/02/2021 07:43:30 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:43:30 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:43:30 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:43:34 INFO Checking backend latencies : 03/02/2021 07:43:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:43:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:43:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:43:46 INFO Checking backend latencies : 03/02/2021 07:43:46 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:43:46 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:43:46 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:43:46 INFO Checking backend latencies : 03/02/2021 07:43:46 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:43:46 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:43:46 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:43:51 INFO Checking backend latencies : 03/02/2021 07:43:51 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:43:51 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:43:51 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:43:51 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:43:54 INFO Checking backend latencies : 03/02/2021 07:43:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:43:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:43:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:43:58 INFO Checking backend latencies : 03/02/2021 07:43:58 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:43:58 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:43:58 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:44:12 INFO Checking backend latencies : 03/02/2021 07:44:12 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:44:12 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:44:12 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:44:16 INFO Checking backend latencies : 03/02/2021 07:44:16 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:44:16 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:44:16 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:44:16 INFO Checking backend latencies : 03/02/2021 07:44:16 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:44:16 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:44:16 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:44:20 INFO Checking backend latencies : 03/02/2021 07:44:20 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:44:20 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:44:20 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:44:25 INFO Checking backend latencies : 03/02/2021 07:44:25 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:44:25 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:44:25 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:44:40 INFO Checking backend latencies : 03/02/2021 07:44:40 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:44:40 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:44:40 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:44:42 INFO Checking backend latencies : 03/02/2021 07:44:42 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:44:42 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:44:42 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:44:50 INFO Checking backend latencies : 03/02/2021 07:44:50 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:44:50 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:44:50 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:44:54 INFO Checking backend latencies : 03/02/2021 07:44:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:44:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:44:54 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:44:54 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader 03/02/2021 07:44:54 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 516, in get_key index, data = cons.get(key) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:44:55 INFO Checking backend latencies : 03/02/2021 07:44:55 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:44:55 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:44:55 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:45:06 INFO Checking backend latencies : 03/02/2021 07:45:06 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:45:06 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:45:06 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:45:20 INFO Checking backend latencies : 03/02/2021 07:45:20 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:45:20 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:45:20 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:45:20 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:45:28 INFO Checking backend latencies : 03/02/2021 07:45:28 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:45:28 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:45:28 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:45:28 ERROR LeaderElectionBase exception 03/02/2021 07:45:28 ERROR 03/02/2021 07:45:33 INFO Checking backend latencies : 03/02/2021 07:45:33 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:45:33 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:45:33 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:45:33 ERROR iSCSI Service exception dropping old sessions 03/02/2021 07:45:36 INFO Checking backend latencies : 03/02/2021 07:45:36 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:45:36 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:45:36 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:46:00 INFO Checking backend latencies : 03/02/2021 07:46:00 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:46:00 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:46:00 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:46:10 INFO Checking backend latencies : 03/02/2021 07:46:10 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:46:10 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:46:10 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:46:14 INFO Checking backend latencies : 03/02/2021 07:46:14 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:46:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:46:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:46:14 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:46:16 INFO Checking backend latencies : 03/02/2021 07:46:16 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:46:16 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:46:16 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:46:25 INFO Checking backend latencies : 03/02/2021 07:46:25 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:46:25 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:46:25 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:46:34 INFO Checking backend latencies : 03/02/2021 07:46:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:46:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:46:34 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:46:38 INFO Checking backend latencies : 03/02/2021 07:46:38 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:46:38 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:46:38 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:46:40 INFO Checking backend latencies : 03/02/2021 07:46:40 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:46:40 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:46:40 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:46:51 INFO Checking backend latencies : 03/02/2021 07:46:51 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:46:51 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:46:51 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:47:01 INFO Checking backend latencies : 03/02/2021 07:47:01 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:47:01 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:47:01 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:47:02 INFO Checking backend latencies : 03/02/2021 07:47:02 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:47:02 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:47:02 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:47:06 INFO Checking backend latencies : 03/02/2021 07:47:06 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:47:06 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:47:06 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:47:21 INFO Checking backend latencies : 03/02/2021 07:47:21 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:47:21 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:47:21 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:47:28 INFO Checking backend latencies : 03/02/2021 07:47:28 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:47:28 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:47:28 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:47:31 INFO Checking backend latencies : 03/02/2021 07:47:31 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:47:31 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:47:31 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:47:36 INFO Checking backend latencies : 03/02/2021 07:47:36 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:47:36 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:47:36 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:47:58 INFO Checking backend latencies : 03/02/2021 07:47:58 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:47:58 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:47:58 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:47:59 INFO Checking backend latencies : 03/02/2021 07:47:59 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:47:59 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:47:59 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:47:59 ERROR Could not find key resource. 03/02/2021 07:47:59 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 496, in get_key_blocking index, data = consul_obj.get(key, lock_index, False, timeout) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:48:09 INFO Checking backend latencies : 03/02/2021 07:48:09 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:48:09 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:48:09 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:48:09 ERROR iSCSI Service exception dropping old sessions 03/02/2021 07:48:15 INFO Checking backend latencies : 03/02/2021 07:48:15 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:48:15 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:48:15 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:48:15 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader 03/02/2021 07:48:15 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 516, in get_key index, data = cons.get(key) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:48:21 INFO Checking backend latencies : 03/02/2021 07:48:21 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:48:21 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:48:21 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:48:36 INFO Checking backend latencies : 03/02/2021 07:48:36 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:48:36 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:48:36 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:48:36 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:48:46 INFO Checking backend latencies : 03/02/2021 07:48:46 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:48:46 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:48:46 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:48:46 INFO Checking backend latencies : 03/02/2021 07:48:46 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:48:46 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:48:46 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:49:00 INFO Checking backend latencies : 03/02/2021 07:49:00 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:49:00 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:49:00 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:49:10 INFO Checking backend latencies : 03/02/2021 07:49:10 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:49:10 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:49:10 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:49:12 INFO Checking backend latencies : 03/02/2021 07:49:12 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:49:12 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:49:12 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:49:25 INFO Checking backend latencies : 03/02/2021 07:49:25 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:49:25 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:49:25 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:49:36 INFO Checking backend latencies : 03/02/2021 07:49:36 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:49:36 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:49:36 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:49:37 INFO Checking backend latencies : 03/02/2021 07:49:37 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:49:37 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:49:37 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:49:42 INFO Checking backend latencies : 03/02/2021 07:49:42 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:49:42 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:49:42 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:49:51 INFO Checking backend latencies : 03/02/2021 07:49:51 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:49:51 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:49:51 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:50:01 INFO Checking backend latencies : 03/02/2021 07:50:01 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:50:01 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:50:01 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:50:07 INFO Checking backend latencies : 03/02/2021 07:50:07 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:50:07 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:50:07 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:50:20 INFO Checking backend latencies : 03/02/2021 07:50:20 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:50:20 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:50:20 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:50:20 ERROR LeaderElectionBase exception 03/02/2021 07:50:20 ERROR 03/02/2021 07:50:21 INFO Checking backend latencies : 03/02/2021 07:50:21 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:50:21 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:50:21 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:50:27 INFO Checking backend latencies : 03/02/2021 07:50:27 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:50:27 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:50:27 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:50:45 INFO Checking backend latencies : 03/02/2021 07:50:45 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:50:45 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:50:45 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:50:45 ERROR iSCSI Service exception dropping old sessions 03/02/2021 07:50:53 INFO Checking backend latencies : 03/02/2021 07:50:53 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:50:53 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:50:53 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:50:57 INFO Checking backend latencies : 03/02/2021 07:50:57 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:50:57 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:50:57 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:50:59 INFO Checking backend latencies : 03/02/2021 07:50:59 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:50:59 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:50:59 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:50:59 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:51:17 INFO Checking backend latencies : 03/02/2021 07:51:17 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:51:17 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:51:17 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:51:22 INFO Checking backend latencies : 03/02/2021 07:51:22 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:51:22 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:51:22 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:51:23 INFO Checking backend latencies : 03/02/2021 07:51:23 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:51:23 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:51:23 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:51:36 INFO Checking backend latencies : 03/02/2021 07:51:36 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:51:36 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:51:36 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:51:36 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader 03/02/2021 07:51:36 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 516, in get_key index, data = cons.get(key) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:51:43 INFO Checking backend latencies : 03/02/2021 07:51:43 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:51:43 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:51:43 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:51:46 INFO Checking backend latencies : 03/02/2021 07:51:46 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:51:46 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:51:46 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:51:48 INFO Checking backend latencies : 03/02/2021 07:51:48 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:51:48 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:51:48 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:52:13 INFO Checking backend latencies : 03/02/2021 07:52:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:52:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:52:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:52:13 INFO Checking backend latencies : 03/02/2021 07:52:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:52:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:52:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:52:14 INFO Checking backend latencies : 03/02/2021 07:52:14 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:52:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:52:14 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:52:43 INFO Checking backend latencies : 03/02/2021 07:52:43 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:52:43 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:52:43 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:52:44 INFO Checking backend latencies : 03/02/2021 07:52:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:52:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:52:44 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:52:51 INFO Checking backend latencies : 03/02/2021 07:52:51 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:52:51 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:52:51 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:52:51 ERROR Could not find key resource. 03/02/2021 07:52:51 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 496, in get_key_blocking index, data = consul_obj.get(key, lock_index, False, timeout) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:52:58 INFO Checking backend latencies : 03/02/2021 07:52:58 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:52:58 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:52:58 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:53:13 INFO Checking backend latencies : 03/02/2021 07:53:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:53:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:53:13 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:53:21 INFO Checking backend latencies : 03/02/2021 07:53:21 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:53:21 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:53:21 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:53:21 ERROR iSCSI Service exception dropping old sessions 03/02/2021 07:53:22 INFO Checking backend latencies : 03/02/2021 07:53:22 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:53:22 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:53:22 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:53:22 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:53:22 INFO Checking backend latencies : 03/02/2021 07:53:22 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:53:22 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:53:22 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:53:37 INFO Checking backend latencies : 03/02/2021 07:53:37 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:53:37 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:53:37 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:53:47 INFO Checking backend latencies : 03/02/2021 07:53:47 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:53:47 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:53:47 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:53:49 INFO Checking backend latencies : 03/02/2021 07:53:49 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:53:49 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:53:49 INFO Network latency for backend 172.16.0.153 = 03/02/2021 07:54:54 INFO Start settings IPs 03/02/2021 07:55:02 INFO GlusterFS mount attempt 03/02/2021 07:55:11 INFO str_start_command: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consul agent -raft-protocol 2 -config-dir /opt/petasan/config/etc/consul.d/server -bind 172.16.0.153 -retry-join 172.16.0.151 -retry-join 172.16.0.152 03/02/2021 07:55:14 INFO Starting cluster file sync service 03/02/2021 07:55:15 INFO LeaderElectionBase dropping old sessions 03/02/2021 07:55:15 INFO Starting iSCSI Service 03/02/2021 07:55:15 INFO Starting Cluster Management application 03/02/2021 07:55:15 INFO Starting Node Stats Service 03/02/2021 07:55:15 INFO Starting activating PetaSAN lvs 03/02/2021 07:55:15 INFO Running command: /sbin/vgs --noheadings --readonly --units=b --nosuffix --separator=";" -o vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_count,vg_extent_size 03/02/2021 07:55:15 INFO stdout ceph-09209f5b-5f36-4ad4-82ff-17c0bfb7f60d";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 07:55:15 INFO stdout ceph-4ed4c220-9498-4351-8ce9-4e00f8b3a93b";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 07:55:15 INFO stdout ceph-d4270832-4236-4724-a362-2dc9fb139f6f";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 07:55:15 INFO stdout ceph-f3285994-1fed-4a78-a067-6141ffe2d08e";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 07:55:15 INFO Starting activating PetaSAN lvs 03/02/2021 07:55:15 INFO Running command: /sbin/vgs --noheadings --readonly --units=b --nosuffix --separator=";" -o vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_count,vg_extent_size 03/02/2021 07:55:15 INFO stdout ceph-09209f5b-5f36-4ad4-82ff-17c0bfb7f60d";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 07:55:15 INFO stdout ceph-4ed4c220-9498-4351-8ce9-4e00f8b3a93b";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 07:55:15 INFO stdout ceph-d4270832-4236-4724-a362-2dc9fb139f6f";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 07:55:15 INFO stdout ceph-f3285994-1fed-4a78-a067-6141ffe2d08e";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 07:55:15 INFO checking uptime 03/02/2021 07:55:15 INFO checking uptime of node: peta1 ip:10.10.0.151 03/02/2021 07:55:15 INFO checking uptime of node: peta2 ip:10.10.0.152 03/02/2021 07:55:17 INFO Starting OSDs 03/02/2021 07:55:17 INFO Starting petasan tuning service 03/02/2021 07:55:17 INFO Starting Config Upload service 03/02/2021 07:55:17 INFO Starting petasan-qperf server 03/02/2021 07:55:34 INFO Checking backend latencies : 03/02/2021 07:55:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:55:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:55:34 INFO Network latency for backend 172.16.0.153 = 7.5 us 03/02/2021 07:55:35 ERROR checking uptime of node: peta2 ping error to ip: 10.10.0.152 03/02/2021 07:55:35 INFO checking uptime of node: peta3 ip:10.10.0.153 03/02/2021 07:55:36 INFO checking uptime of node: peta3 is starting 03/02/2021 07:55:36 INFO Service is starting. 03/02/2021 07:55:36 INFO iSCSI Service dropping old sessions 03/02/2021 07:55:36 INFO Checking backend latencies : 03/02/2021 07:55:36 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:55:36 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:55:36 INFO Network latency for backend 172.16.0.153 = 7.34 us 03/02/2021 07:55:39 INFO GlusterFS mount attempt 03/02/2021 07:55:44 INFO Checking backend latencies : 03/02/2021 07:55:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:55:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:55:44 INFO Network latency for backend 172.16.0.153 = 7.29 us 03/02/2021 07:55:55 INFO Checking backend latencies : 03/02/2021 07:55:55 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:55:55 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:55:55 INFO Network latency for backend 172.16.0.153 = 6.57 us 03/02/2021 07:55:57 INFO Checking backend latencies : 03/02/2021 07:55:57 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:55:57 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:55:57 INFO Network latency for backend 172.16.0.153 = 6.71 us 03/02/2021 07:56:05 INFO Checking backend latencies : 03/02/2021 07:56:05 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:56:05 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:56:05 INFO Network latency for backend 172.16.0.153 = 7.13 us 03/02/2021 07:56:10 INFO Checking backend latencies : 03/02/2021 07:56:10 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:56:10 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:56:10 INFO Network latency for backend 172.16.0.153 = 7.25 us 03/02/2021 07:56:15 INFO GlusterFS mount attempt 03/02/2021 07:56:18 INFO Checking backend latencies : 03/02/2021 07:56:18 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:56:18 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:56:18 INFO Network latency for backend 172.16.0.153 = 6.97 us 03/02/2021 07:56:20 INFO Checking backend latencies : 03/02/2021 07:56:20 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:56:20 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:56:20 INFO Network latency for backend 172.16.0.153 = 7.12 us 03/02/2021 07:56:29 INFO Checking backend latencies : 03/02/2021 07:56:29 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:56:29 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:56:29 INFO Network latency for backend 172.16.0.153 = 6.66 us 03/02/2021 07:56:31 INFO Checking backend latencies : 03/02/2021 07:56:31 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:56:31 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:56:31 INFO Network latency for backend 172.16.0.153 = 7.18 us 03/02/2021 07:56:45 INFO Checking backend latencies : 03/02/2021 07:56:45 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:56:45 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:56:45 INFO Network latency for backend 172.16.0.153 = 6.73 us 03/02/2021 07:56:48 INFO Checking backend latencies : 03/02/2021 07:56:48 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:56:48 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:56:48 INFO Network latency for backend 172.16.0.153 = 6.32 us 03/02/2021 07:56:52 INFO GlusterFS mount attempt 03/02/2021 07:56:54 INFO Checking backend latencies : 03/02/2021 07:56:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:56:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:56:54 INFO Network latency for backend 172.16.0.153 = 6.96 us 03/02/2021 07:56:56 INFO Checking backend latencies : 03/02/2021 07:56:56 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:56:56 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:56:56 INFO Network latency for backend 172.16.0.153 = 6.48 us 03/02/2021 07:57:20 INFO Checking backend latencies : 03/02/2021 07:57:20 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:57:20 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:57:20 INFO Network latency for backend 172.16.0.153 = 8.15 us 03/02/2021 07:57:20 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:57:22 INFO Checking backend latencies : 03/02/2021 07:57:22 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:57:22 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:57:22 INFO Network latency for backend 172.16.0.153 = 7.86 us 03/02/2021 07:57:24 INFO Checking backend latencies : 03/02/2021 07:57:24 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:57:24 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:57:24 INFO Network latency for backend 172.16.0.153 = 6.86 us 03/02/2021 07:57:24 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader 03/02/2021 07:57:24 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 516, in get_key index, data = cons.get(key) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:57:29 INFO GlusterFS mount attempt 03/02/2021 07:57:31 INFO Checking backend latencies : 03/02/2021 07:57:31 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:57:31 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:57:31 INFO Network latency for backend 172.16.0.153 = 6.8 us 03/02/2021 07:57:31 ERROR LeaderElectionBase exception dropping old sessions 03/02/2021 07:57:41 INFO Checking backend latencies : 03/02/2021 07:57:41 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:57:41 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:57:41 INFO Network latency for backend 172.16.0.153 = 7.08 us 03/02/2021 07:57:58 INFO Checking backend latencies : 03/02/2021 07:57:58 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:57:58 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:57:58 INFO Network latency for backend 172.16.0.153 = 6.68 us 03/02/2021 07:57:58 ERROR iSCSI Service exception dropping old sessions 03/02/2021 07:58:00 INFO Checking backend latencies : 03/02/2021 07:58:00 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:58:00 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:58:00 INFO Network latency for backend 172.16.0.153 = 6.83 us 03/02/2021 07:58:02 INFO Checking backend latencies : 03/02/2021 07:58:02 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:58:02 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:58:02 INFO Network latency for backend 172.16.0.153 = 7.05 us 03/02/2021 07:58:06 INFO GlusterFS mount attempt 03/02/2021 07:58:21 INFO Checking backend latencies : 03/02/2021 07:58:21 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:58:21 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:58:21 INFO Network latency for backend 172.16.0.153 = 7.09 us 03/02/2021 07:58:26 INFO Checking backend latencies : 03/02/2021 07:58:26 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:58:26 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:58:26 INFO Network latency for backend 172.16.0.153 = 6.96 us 03/02/2021 07:58:32 INFO Checking backend latencies : 03/02/2021 07:58:32 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:58:32 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:58:32 INFO Network latency for backend 172.16.0.153 = 6.85 us 03/02/2021 07:58:42 INFO GlusterFS mount attempt 03/02/2021 07:58:43 INFO Checking backend latencies : 03/02/2021 07:58:43 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:58:43 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:58:43 INFO Network latency for backend 172.16.0.153 = 6.7 us 03/02/2021 07:58:45 INFO Checking backend latencies : 03/02/2021 07:58:45 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:58:45 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:58:45 INFO Network latency for backend 172.16.0.153 = 6.7 us 03/02/2021 07:58:53 INFO Checking backend latencies : 03/02/2021 07:58:53 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:58:53 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:58:53 INFO Network latency for backend 172.16.0.153 = 7.11 us 03/02/2021 07:58:55 INFO Checking backend latencies : 03/02/2021 07:58:55 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:58:55 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:58:55 INFO Network latency for backend 172.16.0.153 = 6.16 us 03/02/2021 07:59:05 INFO Checking backend latencies : 03/02/2021 07:59:05 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:59:05 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:59:05 INFO Network latency for backend 172.16.0.153 = 6.36 us 03/02/2021 07:59:13 INFO Checking backend latencies : 03/02/2021 07:59:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:59:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:59:13 INFO Network latency for backend 172.16.0.153 = 6.78 us 03/02/2021 07:59:16 INFO GlusterFS mount attempt 03/02/2021 07:59:18 INFO Checking backend latencies : 03/02/2021 07:59:18 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:59:18 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:59:18 INFO Network latency for backend 172.16.0.153 = 7.21 us 03/02/2021 07:59:28 INFO Checking backend latencies : 03/02/2021 07:59:28 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:59:28 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:59:28 INFO Network latency for backend 172.16.0.153 = 6.62 us 03/02/2021 07:59:30 INFO Checking backend latencies : 03/02/2021 07:59:30 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:59:30 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:59:30 INFO Network latency for backend 172.16.0.153 = 6.66 us 03/02/2021 07:59:30 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 07:59:45 INFO Checking backend latencies : 03/02/2021 07:59:45 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:59:45 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:59:45 INFO Network latency for backend 172.16.0.153 = 6.48 us 03/02/2021 07:59:48 INFO Checking backend latencies : 03/02/2021 07:59:48 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:59:48 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:59:48 INFO Network latency for backend 172.16.0.153 = 6.9 us 03/02/2021 07:59:48 ERROR LeaderElectionBase exception dropping old sessions 03/02/2021 07:59:50 INFO GlusterFS mount attempt 03/02/2021 07:59:51 INFO Checking backend latencies : 03/02/2021 07:59:51 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:59:51 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:59:51 INFO Network latency for backend 172.16.0.153 = 6.63 us 03/02/2021 07:59:55 INFO Checking backend latencies : 03/02/2021 07:59:55 INFO Network latency for backend 172.16.0.151 = 03/02/2021 07:59:55 INFO Network latency for backend 172.16.0.152 = 03/02/2021 07:59:55 INFO Network latency for backend 172.16.0.153 = 7.12 us 03/02/2021 08:00:12 INFO Checking backend latencies : 03/02/2021 08:00:12 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:00:12 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:00:12 INFO Network latency for backend 172.16.0.153 = 6.82 us 03/02/2021 08:00:17 INFO Checking backend latencies : 03/02/2021 08:00:17 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:00:17 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:00:17 INFO Network latency for backend 172.16.0.153 = 6.73 us 03/02/2021 08:00:20 INFO Checking backend latencies : 03/02/2021 08:00:20 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:00:20 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:00:20 INFO Network latency for backend 172.16.0.153 = 6.85 us 03/02/2021 08:00:20 ERROR iSCSI Service exception dropping old sessions 03/02/2021 08:00:24 INFO GlusterFS mount attempt 03/02/2021 08:00:30 INFO Checking backend latencies : 03/02/2021 08:00:30 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:00:30 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:00:30 INFO Network latency for backend 172.16.0.153 = 6.2 us 03/02/2021 08:00:30 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader 03/02/2021 08:00:30 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 516, in get_key index, data = cons.get(key) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:00:35 INFO Checking backend latencies : 03/02/2021 08:00:35 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:00:35 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:00:35 INFO Network latency for backend 172.16.0.153 = 6.72 us 03/02/2021 08:00:38 INFO Checking backend latencies : 03/02/2021 08:00:38 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:00:38 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:00:38 INFO Network latency for backend 172.16.0.153 = 6.59 us 03/02/2021 08:00:49 INFO Checking backend latencies : 03/02/2021 08:00:49 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:00:49 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:00:49 INFO Network latency for backend 172.16.0.153 = 6.17 us 03/02/2021 08:00:54 INFO Checking backend latencies : 03/02/2021 08:00:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:00:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:00:54 INFO Network latency for backend 172.16.0.153 = 6.32 us 03/02/2021 08:01:01 INFO GlusterFS mount attempt 03/02/2021 08:01:01 INFO Checking backend latencies : 03/02/2021 08:01:01 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:01:01 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:01:01 INFO Network latency for backend 172.16.0.153 = 6.88 us 03/02/2021 08:01:03 INFO Checking backend latencies : 03/02/2021 08:01:03 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:01:03 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:01:03 INFO Network latency for backend 172.16.0.153 = 6.24 us 03/02/2021 08:01:11 INFO Checking backend latencies : 03/02/2021 08:01:11 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:01:11 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:01:11 INFO Network latency for backend 172.16.0.153 = 6.25 us 03/02/2021 08:01:15 INFO Checking backend latencies : 03/02/2021 08:01:15 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:01:15 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:01:15 INFO Network latency for backend 172.16.0.153 = 6.05 us 03/02/2021 08:01:29 INFO Checking backend latencies : 03/02/2021 08:01:29 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:01:29 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:01:29 INFO Network latency for backend 172.16.0.153 = 6.79 us 03/02/2021 08:01:34 INFO Checking backend latencies : 03/02/2021 08:01:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:01:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:01:34 INFO Network latency for backend 172.16.0.153 = 6.4 us 03/02/2021 08:01:34 INFO GlusterFS mount attempt 03/02/2021 08:01:39 INFO Checking backend latencies : 03/02/2021 08:01:39 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:01:39 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:01:39 INFO Network latency for backend 172.16.0.153 = 6.79 us 03/02/2021 08:01:41 INFO Checking backend latencies : 03/02/2021 08:01:41 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:01:41 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:01:41 INFO Network latency for backend 172.16.0.153 = 6.65 us 03/02/2021 08:01:41 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:01:49 INFO Checking backend latencies : 03/02/2021 08:01:49 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:01:49 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:01:49 INFO Network latency for backend 172.16.0.153 = 6.8 us 03/02/2021 08:02:01 INFO Checking backend latencies : 03/02/2021 08:02:01 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:02:01 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:02:01 INFO Network latency for backend 172.16.0.153 = 7.16 us 03/02/2021 08:02:03 INFO Checking backend latencies : 03/02/2021 08:02:03 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:02:03 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:02:03 INFO Network latency for backend 172.16.0.153 = 6.77 us 03/02/2021 08:02:05 INFO Checking backend latencies : 03/02/2021 08:02:05 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:02:05 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:02:05 INFO Network latency for backend 172.16.0.153 = 6.58 us 03/02/2021 08:02:05 ERROR LeaderElectionBase exception dropping old sessions 03/02/2021 08:02:07 INFO Checking backend latencies : 03/02/2021 08:02:07 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:02:07 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:02:07 INFO Network latency for backend 172.16.0.153 = 6.35 us 03/02/2021 08:02:10 INFO Checking backend latencies : 03/02/2021 08:02:10 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:02:10 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:02:10 INFO Network latency for backend 172.16.0.153 = 6.42 us 03/02/2021 08:02:11 INFO GlusterFS mount attempt 03/02/2021 08:02:24 INFO Checking backend latencies : 03/02/2021 08:02:24 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:02:24 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:02:24 INFO Network latency for backend 172.16.0.153 = 6.89 us 03/02/2021 08:02:34 INFO Checking backend latencies : 03/02/2021 08:02:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:02:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:02:34 INFO Network latency for backend 172.16.0.153 = 6.9 us 03/02/2021 08:02:36 INFO Checking backend latencies : 03/02/2021 08:02:36 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:02:36 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:02:36 INFO Network latency for backend 172.16.0.153 = 6.85 us 03/02/2021 08:02:38 INFO Checking backend latencies : 03/02/2021 08:02:38 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:02:38 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:02:38 INFO Network latency for backend 172.16.0.153 = 6.13 us 03/02/2021 08:02:38 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:02:42 INFO Checking backend latencies : 03/02/2021 08:02:42 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:02:42 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:02:42 INFO Network latency for backend 172.16.0.153 = 7.2 us 03/02/2021 08:02:42 ERROR iSCSI Service exception dropping old sessions 03/02/2021 08:02:47 INFO Checking backend latencies : 03/02/2021 08:02:47 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:02:47 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:02:47 INFO Network latency for backend 172.16.0.153 = 6.89 us 03/02/2021 08:02:48 INFO GlusterFS mount attempt 03/02/2021 08:02:57 INFO Checking backend latencies : 03/02/2021 08:02:57 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:02:57 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:02:57 INFO Network latency for backend 172.16.0.153 = 6.57 us 03/02/2021 08:03:01 INFO Checking backend latencies : 03/02/2021 08:03:01 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:03:01 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:03:01 INFO Network latency for backend 172.16.0.153 = 6.97 us 03/02/2021 08:03:14 INFO Checking backend latencies : 03/02/2021 08:03:14 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:03:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:03:14 INFO Network latency for backend 172.16.0.153 = 6.66 us 03/02/2021 08:03:16 INFO Checking backend latencies : 03/02/2021 08:03:16 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:03:16 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:03:16 INFO Network latency for backend 172.16.0.153 = 6.7 us 03/02/2021 08:03:20 INFO Checking backend latencies : 03/02/2021 08:03:20 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:03:20 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:03:20 INFO Network latency for backend 172.16.0.153 = 6.74 us 03/02/2021 08:03:24 INFO GlusterFS mount attempt 03/02/2021 08:03:36 INFO Checking backend latencies : 03/02/2021 08:03:36 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:03:36 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:03:36 INFO Network latency for backend 172.16.0.153 = 7.09 us 03/02/2021 08:03:36 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader 03/02/2021 08:03:36 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 516, in get_key index, data = cons.get(key) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:03:38 INFO Checking backend latencies : 03/02/2021 08:03:38 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:03:38 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:03:38 INFO Network latency for backend 172.16.0.153 = 6.63 us 03/02/2021 08:03:47 INFO Checking backend latencies : 03/02/2021 08:03:47 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:03:47 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:03:47 INFO Network latency for backend 172.16.0.153 = 6.9 us 03/02/2021 08:03:50 INFO Checking backend latencies : 03/02/2021 08:03:50 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:03:50 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:03:50 INFO Network latency for backend 172.16.0.153 = 6.89 us 03/02/2021 08:03:50 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:03:59 INFO GlusterFS mount attempt 03/02/2021 08:04:01 INFO Checking backend latencies : 03/02/2021 08:04:01 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:04:01 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:04:01 INFO Network latency for backend 172.16.0.153 = 6.56 us 03/02/2021 08:04:11 INFO Checking backend latencies : 03/02/2021 08:04:11 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:04:11 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:04:11 INFO Network latency for backend 172.16.0.153 = 6.84 us 03/02/2021 08:04:22 INFO Checking backend latencies : 03/02/2021 08:04:22 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:04:22 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:04:22 INFO Network latency for backend 172.16.0.153 = 7.06 us 03/02/2021 08:04:22 ERROR LeaderElectionBase exception dropping old sessions 03/02/2021 08:04:28 INFO Checking backend latencies : 03/02/2021 08:04:28 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:04:28 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:04:28 INFO Network latency for backend 172.16.0.153 = 7.1 us 03/02/2021 08:04:32 INFO Checking backend latencies : 03/02/2021 08:04:32 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:04:32 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:04:32 INFO Network latency for backend 172.16.0.153 = 6.91 us 03/02/2021 08:04:32 INFO GlusterFS mount attempt 03/02/2021 08:04:51 INFO Checking backend latencies : 03/02/2021 08:04:51 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:04:51 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:04:51 INFO Network latency for backend 172.16.0.153 = 6.99 us 03/02/2021 08:04:55 INFO Checking backend latencies : 03/02/2021 08:04:55 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:04:55 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:04:55 INFO Network latency for backend 172.16.0.153 = 6.85 us 03/02/2021 08:04:57 INFO Checking backend latencies : 03/02/2021 08:04:57 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:04:57 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:04:57 INFO Network latency for backend 172.16.0.153 = 6.81 us 03/02/2021 08:05:03 INFO Checking backend latencies : 03/02/2021 08:05:03 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:05:03 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:05:03 INFO Network latency for backend 172.16.0.153 = 6.81 us 03/02/2021 08:05:03 ERROR iSCSI Service exception dropping old sessions 03/02/2021 08:05:08 INFO GlusterFS mount attempt 03/02/2021 08:05:13 INFO Checking backend latencies : 03/02/2021 08:05:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:05:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:05:13 INFO Network latency for backend 172.16.0.153 = 7.14 us 03/02/2021 08:05:16 INFO Checking backend latencies : 03/02/2021 08:05:16 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:05:16 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:05:16 INFO Network latency for backend 172.16.0.153 = 7.08 us 03/02/2021 08:05:24 INFO Checking backend latencies : 03/02/2021 08:05:24 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:05:24 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:05:24 INFO Network latency for backend 172.16.0.153 = 6.95 us 03/02/2021 08:05:36 INFO Checking backend latencies : 03/02/2021 08:05:36 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:05:36 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:05:36 INFO Network latency for backend 172.16.0.153 = 6.77 us 03/02/2021 08:05:38 INFO Checking backend latencies : 03/02/2021 08:05:38 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:05:38 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:05:38 INFO Network latency for backend 172.16.0.153 = 6.92 us 03/02/2021 08:05:40 INFO Checking backend latencies : 03/02/2021 08:05:40 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:05:40 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:05:40 INFO Network latency for backend 172.16.0.153 = 6.38 us 03/02/2021 08:05:41 INFO GlusterFS mount attempt 03/02/2021 08:05:59 INFO Checking backend latencies : 03/02/2021 08:05:59 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:05:59 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:05:59 INFO Network latency for backend 172.16.0.153 = 6.93 us 03/02/2021 08:06:01 INFO Checking backend latencies : 03/02/2021 08:06:01 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:06:01 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:06:01 INFO Network latency for backend 172.16.0.153 = 6.84 us 03/02/2021 08:06:01 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:06:03 INFO Checking backend latencies : 03/02/2021 08:06:03 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:06:03 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:06:03 INFO Network latency for backend 172.16.0.153 = 6.48 us 03/02/2021 08:06:07 INFO Checking backend latencies : 03/02/2021 08:06:07 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:06:07 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:06:07 INFO Network latency for backend 172.16.0.153 = 6.89 us 03/02/2021 08:06:15 INFO GlusterFS mount attempt 03/02/2021 08:06:22 INFO Checking backend latencies : 03/02/2021 08:06:22 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:06:22 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:06:22 INFO Network latency for backend 172.16.0.153 = 6.72 us 03/02/2021 08:06:24 INFO Checking backend latencies : 03/02/2021 08:06:24 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:06:24 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:06:24 INFO Network latency for backend 172.16.0.153 = 6.75 us 03/02/2021 08:06:38 INFO Checking backend latencies : 03/02/2021 08:06:38 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:06:38 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:06:38 INFO Network latency for backend 172.16.0.153 = 6.9 us 03/02/2021 08:06:38 ERROR LeaderElectionBase exception dropping old sessions 03/02/2021 08:06:42 INFO Checking backend latencies : 03/02/2021 08:06:42 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:06:42 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:06:42 INFO Network latency for backend 172.16.0.153 = 6.69 us 03/02/2021 08:06:42 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader 03/02/2021 08:06:42 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 516, in get_key index, data = cons.get(key) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:06:45 INFO Checking backend latencies : 03/02/2021 08:06:45 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:06:45 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:06:45 INFO Network latency for backend 172.16.0.153 = 7.2 us 03/02/2021 08:06:49 INFO Checking backend latencies : 03/02/2021 08:06:49 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:06:49 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:06:49 INFO Network latency for backend 172.16.0.153 = 6.59 us 03/02/2021 08:06:52 INFO GlusterFS mount attempt 03/02/2021 08:07:07 INFO Checking backend latencies : 03/02/2021 08:07:07 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:07:07 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:07:07 INFO Network latency for backend 172.16.0.153 = 6.71 us 03/02/2021 08:07:09 INFO Checking backend latencies : 03/02/2021 08:07:09 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:07:09 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:07:09 INFO Network latency for backend 172.16.0.153 = 6.55 us 03/02/2021 08:07:24 INFO Checking backend latencies : 03/02/2021 08:07:24 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:07:24 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:07:24 INFO Network latency for backend 172.16.0.153 = 7.15 us 03/02/2021 08:07:24 ERROR iSCSI Service exception dropping old sessions 03/02/2021 08:07:26 INFO GlusterFS mount attempt 03/02/2021 08:07:28 INFO Checking backend latencies : 03/02/2021 08:07:28 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:07:28 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:07:28 INFO Network latency for backend 172.16.0.153 = 6.6 us 03/02/2021 08:07:36 INFO Checking backend latencies : 03/02/2021 08:07:36 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:07:36 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:07:36 INFO Network latency for backend 172.16.0.153 = 6.23 us 03/02/2021 08:07:51 INFO Checking backend latencies : 03/02/2021 08:07:51 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:07:51 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:07:51 INFO Network latency for backend 172.16.0.153 = 6.52 us 03/02/2021 08:07:59 INFO Checking backend latencies : 03/02/2021 08:07:59 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:07:59 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:07:59 INFO Network latency for backend 172.16.0.153 = 7.05 us 03/02/2021 08:08:01 INFO Checking backend latencies : 03/02/2021 08:08:01 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:08:01 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:08:01 INFO Network latency for backend 172.16.0.153 = 6.81 us 03/02/2021 08:08:02 INFO GlusterFS mount attempt 03/02/2021 08:08:11 INFO Checking backend latencies : 03/02/2021 08:08:11 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:08:11 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:08:11 INFO Network latency for backend 172.16.0.153 = 7.02 us 03/02/2021 08:08:11 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:08:18 INFO Checking backend latencies : 03/02/2021 08:08:18 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:08:18 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:08:18 INFO Network latency for backend 172.16.0.153 = 6.87 us 03/02/2021 08:08:20 INFO Checking backend latencies : 03/02/2021 08:08:20 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:08:20 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:08:20 INFO Network latency for backend 172.16.0.153 = 6.69 us 03/02/2021 08:08:22 INFO Checking backend latencies : 03/02/2021 08:08:22 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:08:22 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:08:22 INFO Network latency for backend 172.16.0.153 = 6.49 us 03/02/2021 08:08:33 INFO Checking backend latencies : 03/02/2021 08:08:33 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:08:33 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:08:33 INFO Network latency for backend 172.16.0.153 = 6.86 us 03/02/2021 08:08:37 INFO GlusterFS mount attempt 03/02/2021 08:08:44 INFO Checking backend latencies : 03/02/2021 08:08:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:08:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:08:44 INFO Network latency for backend 172.16.0.153 = 7.07 us 03/02/2021 08:08:46 INFO Checking backend latencies : 03/02/2021 08:08:46 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:08:46 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:08:46 INFO Network latency for backend 172.16.0.153 = 6.39 us 03/02/2021 08:08:54 INFO Checking backend latencies : 03/02/2021 08:08:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:08:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:08:54 INFO Network latency for backend 172.16.0.153 = 7.14 us 03/02/2021 08:08:56 INFO Checking backend latencies : 03/02/2021 08:08:56 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:08:56 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:08:56 INFO Network latency for backend 172.16.0.153 = 7.22 us 03/02/2021 08:08:56 ERROR LeaderElectionBase exception dropping old sessions 03/02/2021 08:09:11 INFO Checking backend latencies : 03/02/2021 08:09:11 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:09:11 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:09:11 INFO Network latency for backend 172.16.0.153 = 6.52 us 03/02/2021 08:09:11 INFO GlusterFS mount attempt 03/02/2021 08:09:13 INFO Checking backend latencies : 03/02/2021 08:09:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:09:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:09:13 INFO Network latency for backend 172.16.0.153 = 7.11 us 03/02/2021 08:09:17 INFO Checking backend latencies : 03/02/2021 08:09:17 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:09:17 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:09:17 INFO Network latency for backend 172.16.0.153 = 6.98 us 03/02/2021 08:09:25 INFO Checking backend latencies : 03/02/2021 08:09:25 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:09:25 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:09:25 INFO Network latency for backend 172.16.0.153 = 7.1 us 03/02/2021 08:09:44 INFO Checking backend latencies : 03/02/2021 08:09:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:09:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:09:44 INFO Network latency for backend 172.16.0.153 = 7.25 us 03/02/2021 08:09:45 INFO GlusterFS mount attempt 03/02/2021 08:09:46 INFO Checking backend latencies : 03/02/2021 08:09:46 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:09:46 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:09:46 INFO Network latency for backend 172.16.0.153 = 7.13 us 03/02/2021 08:09:46 ERROR iSCSI Service exception dropping old sessions 03/02/2021 08:09:48 INFO Checking backend latencies : 03/02/2021 08:09:48 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:09:48 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:09:48 INFO Network latency for backend 172.16.0.153 = 7.31 us 03/02/2021 08:09:50 INFO Checking backend latencies : 03/02/2021 08:09:50 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:09:50 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:09:50 INFO Network latency for backend 172.16.0.153 = 6.9 us 03/02/2021 08:09:50 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader 03/02/2021 08:09:50 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 516, in get_key index, data = cons.get(key) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:10:11 INFO Checking backend latencies : 03/02/2021 08:10:11 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:10:11 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:10:11 INFO Network latency for backend 172.16.0.153 = 6.95 us 03/02/2021 08:10:18 INFO GlusterFS mount attempt 03/02/2021 08:10:19 INFO Checking backend latencies : 03/02/2021 08:10:19 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:10:19 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:10:19 INFO Network latency for backend 172.16.0.153 = 7.48 us 03/02/2021 08:10:19 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:10:21 INFO Checking backend latencies : 03/02/2021 08:10:21 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:10:21 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:10:21 INFO Network latency for backend 172.16.0.153 = 7.18 us 03/02/2021 08:10:38 INFO Checking backend latencies : 03/02/2021 08:10:38 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:10:38 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:10:38 INFO Network latency for backend 172.16.0.153 = 6.02 us 03/02/2021 08:10:40 INFO Checking backend latencies : 03/02/2021 08:10:40 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:10:40 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:10:40 INFO Network latency for backend 172.16.0.153 = 6.93 us 03/02/2021 08:10:43 INFO Checking backend latencies : 03/02/2021 08:10:43 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:10:43 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:10:43 INFO Network latency for backend 172.16.0.153 = 6.98 us 03/02/2021 08:10:51 INFO GlusterFS mount attempt 03/02/2021 08:11:02 INFO Checking backend latencies : 03/02/2021 08:11:02 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:11:02 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:11:02 INFO Network latency for backend 172.16.0.153 = 7.31 us 03/02/2021 08:11:06 INFO Checking backend latencies : 03/02/2021 08:11:06 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:11:06 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:11:06 INFO Network latency for backend 172.16.0.153 = 7.22 us 03/02/2021 08:11:09 INFO Checking backend latencies : 03/02/2021 08:11:09 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:11:09 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:11:09 INFO Network latency for backend 172.16.0.153 = 6.74 us 03/02/2021 08:11:13 INFO Checking backend latencies : 03/02/2021 08:11:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:11:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:11:13 INFO Network latency for backend 172.16.0.153 = 7.05 us 03/02/2021 08:11:13 ERROR LeaderElectionBase exception dropping old sessions 03/02/2021 08:11:25 INFO Checking backend latencies : 03/02/2021 08:11:25 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:11:25 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:11:25 INFO Network latency for backend 172.16.0.153 = 7.08 us 03/02/2021 08:11:28 INFO GlusterFS mount attempt 03/02/2021 08:11:30 INFO Checking backend latencies : 03/02/2021 08:11:30 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:11:30 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:11:30 INFO Network latency for backend 172.16.0.153 = 6.53 us 03/02/2021 08:11:33 INFO Checking backend latencies : 03/02/2021 08:11:33 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:11:33 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:11:33 INFO Network latency for backend 172.16.0.153 = 7.37 us 03/02/2021 08:11:43 INFO Checking backend latencies : 03/02/2021 08:11:43 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:11:43 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:11:43 INFO Network latency for backend 172.16.0.153 = 7.09 us 03/02/2021 08:11:52 INFO Checking backend latencies : 03/02/2021 08:11:52 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:11:52 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:11:52 INFO Network latency for backend 172.16.0.153 = 7.1 us 03/02/2021 08:11:54 INFO Checking backend latencies : 03/02/2021 08:11:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:11:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:11:54 INFO Network latency for backend 172.16.0.153 = 6.95 us 03/02/2021 08:12:04 INFO Checking backend latencies : 03/02/2021 08:12:04 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:12:04 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:12:04 INFO Network latency for backend 172.16.0.153 = 7.15 us 03/02/2021 08:12:05 INFO GlusterFS mount attempt 03/02/2021 08:12:08 INFO Checking backend latencies : 03/02/2021 08:12:08 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:12:08 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:12:08 INFO Network latency for backend 172.16.0.153 = 7.17 us 03/02/2021 08:12:08 ERROR iSCSI Service exception dropping old sessions 03/02/2021 08:12:21 INFO Checking backend latencies : 03/02/2021 08:12:21 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:12:21 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:12:21 INFO Network latency for backend 172.16.0.153 = 6.88 us 03/02/2021 08:12:27 INFO Checking backend latencies : 03/02/2021 08:12:27 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:12:27 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:12:27 INFO Network latency for backend 172.16.0.153 = 7.25 us 03/02/2021 08:12:29 INFO Checking backend latencies : 03/02/2021 08:12:29 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:12:29 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:12:29 INFO Network latency for backend 172.16.0.153 = 7.38 us 03/02/2021 08:12:29 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:12:42 INFO GlusterFS mount attempt 03/02/2021 08:12:42 INFO Checking backend latencies : 03/02/2021 08:12:42 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:12:42 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:12:42 INFO Network latency for backend 172.16.0.153 = 6.92 us 03/02/2021 08:12:50 INFO Checking backend latencies : 03/02/2021 08:12:50 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:12:50 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:12:50 INFO Network latency for backend 172.16.0.153 = 7.15 us 03/02/2021 08:12:54 INFO Checking backend latencies : 03/02/2021 08:12:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:12:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:12:54 INFO Network latency for backend 172.16.0.153 = 7.04 us 03/02/2021 08:12:56 INFO Checking backend latencies : 03/02/2021 08:12:56 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:12:56 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:12:56 INFO Network latency for backend 172.16.0.153 = 7.09 us 03/02/2021 08:12:56 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader 03/02/2021 08:12:56 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 516, in get_key index, data = cons.get(key) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:13:03 INFO Checking backend latencies : 03/02/2021 08:13:03 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:13:03 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:13:03 INFO Network latency for backend 172.16.0.153 = 7.17 us 03/02/2021 08:13:11 INFO Checking backend latencies : 03/02/2021 08:13:11 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:13:11 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:13:11 INFO Network latency for backend 172.16.0.153 = 7.15 us 03/02/2021 08:13:16 INFO GlusterFS mount attempt 03/02/2021 08:13:27 INFO Checking backend latencies : 03/02/2021 08:13:27 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:13:27 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:13:27 INFO Network latency for backend 172.16.0.153 = 6.99 us 03/02/2021 08:13:29 INFO Checking backend latencies : 03/02/2021 08:13:29 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:13:29 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:13:29 INFO Network latency for backend 172.16.0.153 = 7.21 us 03/02/2021 08:13:29 ERROR LeaderElectionBase exception dropping old sessions 03/02/2021 08:13:34 INFO Checking backend latencies : 03/02/2021 08:13:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:13:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:13:34 INFO Network latency for backend 172.16.0.153 = 7.11 us 03/02/2021 08:13:50 INFO GlusterFS mount attempt 03/02/2021 08:13:54 INFO Checking backend latencies : 03/02/2021 08:13:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:13:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:13:54 INFO Network latency for backend 172.16.0.153 = 7.21 us 03/02/2021 08:13:58 INFO Checking backend latencies : 03/02/2021 08:13:58 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:13:58 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:13:58 INFO Network latency for backend 172.16.0.153 = 7.25 us 03/02/2021 08:14:01 INFO Checking backend latencies : 03/02/2021 08:14:01 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:14:01 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:14:01 INFO Network latency for backend 172.16.0.153 = 7.19 us 03/02/2021 08:14:15 INFO Checking backend latencies : 03/02/2021 08:14:15 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:14:15 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:14:15 INFO Network latency for backend 172.16.0.153 = 7.03 us 03/02/2021 08:14:18 INFO Checking backend latencies : 03/02/2021 08:14:18 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:14:18 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:14:18 INFO Network latency for backend 172.16.0.153 = 6.21 us 03/02/2021 08:14:20 INFO Checking backend latencies : 03/02/2021 08:14:20 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:14:20 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:14:20 INFO Network latency for backend 172.16.0.153 = 6.69 us 03/02/2021 08:14:27 INFO GlusterFS mount attempt 03/02/2021 08:14:29 INFO Checking backend latencies : 03/02/2021 08:14:29 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:14:29 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:14:29 INFO Network latency for backend 172.16.0.153 = 7.19 us 03/02/2021 08:14:29 ERROR iSCSI Service exception dropping old sessions 03/02/2021 08:14:36 INFO Checking backend latencies : 03/02/2021 08:14:36 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:14:36 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:14:36 INFO Network latency for backend 172.16.0.153 = 7.21 us 03/02/2021 08:14:38 INFO Checking backend latencies : 03/02/2021 08:14:38 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:14:38 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:14:38 INFO Network latency for backend 172.16.0.153 = 7.02 us 03/02/2021 08:14:38 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:14:40 INFO Checking backend latencies : 03/02/2021 08:14:40 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:14:40 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:14:40 INFO Network latency for backend 172.16.0.153 = 7.11 us 03/02/2021 08:14:43 INFO Checking backend latencies : 03/02/2021 08:14:43 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:14:43 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:14:43 INFO Network latency for backend 172.16.0.153 = 7.15 us 03/02/2021 08:14:59 INFO Checking backend latencies : 03/02/2021 08:14:59 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:14:59 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:14:59 INFO Network latency for backend 172.16.0.153 = 6.75 us 03/02/2021 08:15:01 INFO GlusterFS mount attempt 03/02/2021 08:15:01 INFO Checking backend latencies : 03/02/2021 08:15:01 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:15:01 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:15:01 INFO Network latency for backend 172.16.0.153 = 7.26 us 03/02/2021 08:15:03 INFO Checking backend latencies : 03/02/2021 08:15:03 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:15:03 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:15:03 INFO Network latency for backend 172.16.0.153 = 7.26 us 03/02/2021 08:15:05 INFO Checking backend latencies : 03/02/2021 08:15:05 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:15:05 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:15:05 INFO Network latency for backend 172.16.0.153 = 7.24 us 03/02/2021 08:15:10 INFO Checking backend latencies : 03/02/2021 08:15:10 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:15:10 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:15:10 INFO Network latency for backend 172.16.0.153 = 7.19 us 03/02/2021 08:15:22 INFO Checking backend latencies : 03/02/2021 08:15:22 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:15:22 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:15:22 INFO Network latency for backend 172.16.0.153 = 7.1 us 03/02/2021 08:15:25 INFO Checking backend latencies : 03/02/2021 08:15:25 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:15:25 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:15:25 INFO Network latency for backend 172.16.0.153 = 7.09 us 03/02/2021 08:15:27 INFO Checking backend latencies : 03/02/2021 08:15:27 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:15:27 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:15:27 INFO Network latency for backend 172.16.0.153 = 7.17 us 03/02/2021 08:15:32 INFO Checking backend latencies : 03/02/2021 08:15:32 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:15:32 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:15:32 INFO Network latency for backend 172.16.0.153 = 7.33 us 03/02/2021 08:15:35 INFO GlusterFS mount attempt 03/02/2021 08:15:46 INFO Checking backend latencies : 03/02/2021 08:15:46 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:15:46 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:15:46 INFO Network latency for backend 172.16.0.153 = 6.65 us 03/02/2021 08:15:46 ERROR LeaderElectionBase exception dropping old sessions 03/02/2021 08:15:48 INFO Checking backend latencies : 03/02/2021 08:15:48 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:15:48 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:15:48 INFO Network latency for backend 172.16.0.153 = 7.19 us 03/02/2021 08:15:50 INFO Checking backend latencies : 03/02/2021 08:15:50 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:15:50 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:15:50 INFO Network latency for backend 172.16.0.153 = 7.17 us 03/02/2021 08:16:02 INFO Checking backend latencies : 03/02/2021 08:16:02 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:16:02 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:16:02 INFO Network latency for backend 172.16.0.153 = 7.12 us 03/02/2021 08:16:02 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader 03/02/2021 08:16:02 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 516, in get_key index, data = cons.get(key) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:16:07 INFO Checking backend latencies : 03/02/2021 08:16:07 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:16:07 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:16:07 INFO Network latency for backend 172.16.0.153 = 6.94 us 03/02/2021 08:16:07 ERROR Error on request: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 270, in run_wsgi execute(self.server.app) File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 258, in execute application_iter = app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/usr/lib/python3/dist-packages/flask/app.py", line 1978, in wsgi_app ctx.push() File "/usr/lib/python3/dist-packages/flask/ctx.py", line 332, in push self.session = self.app.open_session(self.request) File "/usr/lib/python3/dist-packages/flask/app.py", line 912, in open_session return self.session_interface.open_session(self, request) File "/usr/lib/python3/dist-packages/PetaSAN/core/security/session.py", line 127, in open_session _exp = consul_base_api.read_value("".join( [consul_session_key,sid,"/_exp"])) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 33, in read_value index, data = cons.kv.get(key) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:16:09 INFO GlusterFS mount attempt 03/02/2021 08:16:15 INFO Checking backend latencies : 03/02/2021 08:16:15 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:16:15 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:16:15 INFO Network latency for backend 172.16.0.153 = 7.37 us 03/02/2021 08:16:17 INFO Checking backend latencies : 03/02/2021 08:16:17 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:16:17 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:16:17 INFO Network latency for backend 172.16.0.153 = 7.06 us 03/02/2021 08:16:19 INFO Checking backend latencies : 03/02/2021 08:16:19 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:16:19 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:16:19 INFO Network latency for backend 172.16.0.153 = 7.23 us 03/02/2021 08:16:36 INFO Checking backend latencies : 03/02/2021 08:16:36 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:16:36 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:16:36 INFO Network latency for backend 172.16.0.153 = 7.28 us 03/02/2021 08:16:43 INFO GlusterFS mount attempt 03/02/2021 08:16:52 INFO Checking backend latencies : 03/02/2021 08:16:52 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:16:52 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:16:52 INFO Network latency for backend 172.16.0.153 = 7.02 us 03/02/2021 08:16:52 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:16:54 INFO Checking backend latencies : 03/02/2021 08:16:54 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:16:54 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:16:54 INFO Network latency for backend 172.16.0.153 = 7.28 us 03/02/2021 08:16:54 ERROR iSCSI Service exception dropping old sessions 03/02/2021 08:16:59 INFO Checking backend latencies : 03/02/2021 08:16:59 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:16:59 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:16:59 INFO Network latency for backend 172.16.0.153 = 7.29 us 03/02/2021 08:17:13 INFO Checking backend latencies : 03/02/2021 08:17:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:17:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:17:13 INFO Network latency for backend 172.16.0.153 = 7.35 us 03/02/2021 08:17:17 INFO GlusterFS mount attempt 03/02/2021 08:17:21 INFO Checking backend latencies : 03/02/2021 08:17:21 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:17:21 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:17:21 INFO Network latency for backend 172.16.0.153 = 7.3 us 03/02/2021 08:17:26 INFO Checking backend latencies : 03/02/2021 08:17:26 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:17:26 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:17:26 INFO Network latency for backend 172.16.0.153 = 7.15 us 03/02/2021 08:17:29 INFO Checking backend latencies : 03/02/2021 08:17:29 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:17:29 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:17:29 INFO Network latency for backend 172.16.0.153 = 7.15 us 03/02/2021 08:17:34 INFO Checking backend latencies : 03/02/2021 08:17:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:17:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:17:34 INFO Network latency for backend 172.16.0.153 = 7.11 us 03/02/2021 08:17:42 INFO Checking backend latencies : 03/02/2021 08:17:42 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:17:42 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:17:42 INFO Network latency for backend 172.16.0.153 = 6.96 us 03/02/2021 08:17:50 INFO Checking backend latencies : 03/02/2021 08:17:50 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:17:50 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:17:50 INFO Network latency for backend 172.16.0.153 = 6.99 us 03/02/2021 08:17:51 INFO GlusterFS mount attempt 03/02/2021 08:17:58 INFO Checking backend latencies : 03/02/2021 08:17:58 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:17:58 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:17:58 INFO Network latency for backend 172.16.0.153 = 7.07 us 03/02/2021 08:18:01 INFO Checking backend latencies : 03/02/2021 08:18:01 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:18:01 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:18:01 INFO Network latency for backend 172.16.0.153 = 7.11 us 03/02/2021 08:18:01 ERROR LeaderElectionBase exception dropping old sessions 03/02/2021 08:18:05 INFO Checking backend latencies : 03/02/2021 08:18:05 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:18:05 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:18:05 INFO Network latency for backend 172.16.0.153 = 7.29 us 03/02/2021 08:18:13 INFO Checking backend latencies : 03/02/2021 08:18:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:18:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:18:13 INFO Network latency for backend 172.16.0.153 = 7.22 us 03/02/2021 08:18:25 INFO Checking backend latencies : 03/02/2021 08:18:25 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:18:25 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:18:25 INFO Network latency for backend 172.16.0.153 = 7.23 us 03/02/2021 08:18:27 INFO GlusterFS mount attempt 03/02/2021 08:18:31 INFO Checking backend latencies : 03/02/2021 08:18:31 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:18:31 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:18:31 INFO Network latency for backend 172.16.0.153 = 7.29 us 03/02/2021 08:18:33 INFO Checking backend latencies : 03/02/2021 08:18:33 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:18:33 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:18:33 INFO Network latency for backend 172.16.0.153 = 7.28 us 03/02/2021 08:18:40 INFO Checking backend latencies : 03/02/2021 08:18:40 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:18:40 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:18:40 INFO Network latency for backend 172.16.0.153 = 7.12 us 03/02/2021 08:18:52 INFO Checking backend latencies : 03/02/2021 08:18:52 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:18:52 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:18:52 INFO Network latency for backend 172.16.0.153 = 7.19 us 03/02/2021 08:19:00 INFO Checking backend latencies : 03/02/2021 08:19:00 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:19:00 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:19:00 INFO Network latency for backend 172.16.0.153 = 7.22 us 03/02/2021 08:19:00 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:19:01 INFO GlusterFS mount attempt 03/02/2021 08:19:08 INFO Checking backend latencies : 03/02/2021 08:19:08 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:19:08 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:19:08 INFO Network latency for backend 172.16.0.153 = 7.16 us 03/02/2021 08:19:08 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader 03/02/2021 08:19:08 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 516, in get_key index, data = cons.get(key) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:19:15 INFO Checking backend latencies : 03/02/2021 08:19:15 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:19:15 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:19:15 INFO Network latency for backend 172.16.0.153 = 7.3 us 03/02/2021 08:19:17 INFO Checking backend latencies : 03/02/2021 08:19:17 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:19:17 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:19:17 INFO Network latency for backend 172.16.0.153 = 7.13 us 03/02/2021 08:19:17 ERROR iSCSI Service exception dropping old sessions 03/02/2021 08:19:21 INFO Checking backend latencies : 03/02/2021 08:19:21 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:19:21 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:19:21 INFO Network latency for backend 172.16.0.153 = 7.3 us 03/02/2021 08:19:35 INFO GlusterFS mount attempt 03/02/2021 08:19:42 INFO Checking backend latencies : 03/02/2021 08:19:42 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:19:42 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:19:42 INFO Network latency for backend 172.16.0.153 = 7.38 us 03/02/2021 08:19:44 INFO Checking backend latencies : 03/02/2021 08:19:44 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:19:44 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:19:44 INFO Network latency for backend 172.16.0.153 = 7.16 us 03/02/2021 08:19:51 INFO Checking backend latencies : 03/02/2021 08:19:51 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:19:51 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:19:51 INFO Network latency for backend 172.16.0.153 = 7.22 us 03/02/2021 08:20:05 INFO Checking backend latencies : 03/02/2021 08:20:05 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:20:05 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:20:05 INFO Network latency for backend 172.16.0.153 = 7.19 us 03/02/2021 08:20:10 INFO GlusterFS mount attempt 03/02/2021 08:20:13 INFO Checking backend latencies : 03/02/2021 08:20:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:20:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:20:13 INFO Network latency for backend 172.16.0.153 = 7.16 us 03/02/2021 08:20:19 INFO Checking backend latencies : 03/02/2021 08:20:19 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:20:19 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:20:19 INFO Network latency for backend 172.16.0.153 = 7.44 us 03/02/2021 08:20:19 ERROR LeaderElectionBase exception dropping old sessions 03/02/2021 08:20:27 INFO Checking backend latencies : 03/02/2021 08:20:27 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:20:27 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:20:27 INFO Network latency for backend 172.16.0.153 = 7.14 us 03/02/2021 08:20:32 INFO Checking backend latencies : 03/02/2021 08:20:32 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:20:32 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:20:32 INFO Network latency for backend 172.16.0.153 = 7.19 us 03/02/2021 08:20:36 INFO Checking backend latencies : 03/02/2021 08:20:36 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:20:36 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:20:36 INFO Network latency for backend 172.16.0.153 = 7.12 us 03/02/2021 08:20:44 INFO GlusterFS mount attempt 03/02/2021 08:20:48 INFO Checking backend latencies : 03/02/2021 08:20:48 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:20:48 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:20:48 INFO Network latency for backend 172.16.0.153 = 7.32 us 03/02/2021 08:20:50 INFO Checking backend latencies : 03/02/2021 08:20:50 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:20:50 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:20:50 INFO Network latency for backend 172.16.0.153 = 7.02 us 03/02/2021 08:21:03 INFO Checking backend latencies : 03/02/2021 08:21:03 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:21:03 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:21:03 INFO Network latency for backend 172.16.0.153 = 7.1 us 03/02/2021 08:21:08 INFO Checking backend latencies : 03/02/2021 08:21:08 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:21:08 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:21:08 INFO Network latency for backend 172.16.0.153 = 7.31 us 03/02/2021 08:21:08 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:21:11 INFO Checking backend latencies : 03/02/2021 08:21:11 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:21:11 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:21:11 INFO Network latency for backend 172.16.0.153 = 7.21 us 03/02/2021 08:21:13 INFO Checking backend latencies : 03/02/2021 08:21:13 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:21:13 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:21:13 INFO Network latency for backend 172.16.0.153 = 7.33 us 03/02/2021 08:21:17 INFO GlusterFS mount attempt 03/02/2021 08:21:29 INFO Checking backend latencies : 03/02/2021 08:21:29 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:21:29 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:21:29 INFO Network latency for backend 172.16.0.153 = 7.35 us 03/02/2021 08:21:34 INFO Checking backend latencies : 03/02/2021 08:21:34 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:21:34 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:21:34 INFO Network latency for backend 172.16.0.153 = 7.34 us 03/02/2021 08:21:38 INFO Checking backend latencies : 03/02/2021 08:21:38 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:21:38 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:21:38 INFO Network latency for backend 172.16.0.153 = 6.05 us 03/02/2021 08:21:38 ERROR iSCSI Service exception dropping old sessions 03/02/2021 08:21:40 INFO Checking backend latencies : 03/02/2021 08:21:40 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:21:40 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:21:40 INFO Network latency for backend 172.16.0.153 = 7.23 us 03/02/2021 08:21:50 INFO Checking backend latencies : 03/02/2021 08:21:50 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:21:50 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:21:50 INFO Network latency for backend 172.16.0.153 = 6.94 us 03/02/2021 08:21:53 INFO GlusterFS mount attempt 03/02/2021 08:22:01 INFO Checking backend latencies : 03/02/2021 08:22:01 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:22:01 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:22:01 INFO Network latency for backend 172.16.0.153 = 7.22 us 03/02/2021 08:22:12 INFO Checking backend latencies : 03/02/2021 08:22:12 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:22:12 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:22:12 INFO Network latency for backend 172.16.0.153 = 7.15 us 03/02/2021 08:22:14 INFO Checking backend latencies : 03/02/2021 08:22:14 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:22:14 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:22:14 INFO Network latency for backend 172.16.0.153 = 7.24 us 03/02/2021 08:22:16 INFO Checking backend latencies : 03/02/2021 08:22:16 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:22:16 INFO Network latency for backend 172.16.0.152 = 03/02/2021 08:22:16 INFO Network latency for backend 172.16.0.153 = 6.95 us 03/02/2021 08:22:16 ERROR get_key, could not get key: PetaSAN/Services/ClusterLeader 03/02/2021 08:22:16 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 516, in get_key index, data = cons.get(key) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 120, in get token=token, consistency=consistency, keys=keys, separator=separator, dc=dc) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 08:22:27 INFO GlusterFS mount attempt 03/02/2021 08:22:33 INFO Checking backend latencies : 03/02/2021 08:22:33 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:22:33 INFO Network latency for backend 172.16.0.152 = 53.8 us 03/02/2021 08:22:33 INFO Network latency for backend 172.16.0.153 = 7.75 us 03/02/2021 08:22:35 INFO Checking backend latencies : 03/02/2021 08:22:35 INFO Network latency for backend 172.16.0.151 = 03/02/2021 08:22:35 INFO Network latency for backend 172.16.0.152 = 52.2 us 03/02/2021 08:22:35 INFO Network latency for backend 172.16.0.153 = 7.95 us 03/02/2021 08:22:35 ERROR LeaderElectionBase exception dropping old sessions 03/02/2021 08:22:41 INFO iSCSI Service successfully dropped old sessions 03/02/2021 08:22:41 INFO Cleaning unused configurations. 03/02/2021 08:22:41 INFO Cleaning all mapped disks 03/02/2021 08:22:41 INFO Cleaning unused rbd images. 03/02/2021 08:22:41 INFO Cleaning unused ips. 03/02/2021 08:22:45 INFO LeaderElectionBase successfully dropped old sessions 03/02/2021 08:23:00 INFO Found pool:rbd for disk:00001 via consul 03/02/2021 08:23:00 INFO Image image-00001 mapped successfully. 03/02/2021 08:23:02 INFO GlusterFS mount attempt 03/02/2021 08:23:03 INFO LIO add_target() disk wwn is c74c6e0600001 03/02/2021 08:23:03 INFO Path 00001/2 acquired successfully 03/02/2021 08:37:42 INFO Start settings IPs 03/02/2021 08:37:48 INFO GlusterFS mount attempt 03/02/2021 08:37:56 INFO str_start_command: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consul agent -raft-protocol 2 -config-dir /opt/petasan/config/etc/consul.d/server -bind 172.16.0.153 -retry-join 172.16.0.151 -retry-join 172.16.0.152 03/02/2021 08:37:59 INFO Starting cluster file sync service 03/02/2021 08:38:00 INFO LeaderElectionBase dropping old sessions 03/02/2021 08:38:00 INFO Starting iSCSI Service 03/02/2021 08:38:00 INFO Starting Cluster Management application 03/02/2021 08:38:00 INFO Starting Node Stats Service 03/02/2021 08:38:00 INFO Starting activating PetaSAN lvs 03/02/2021 08:38:00 INFO Running command: /sbin/vgs --noheadings --readonly --units=b --nosuffix --separator=";" -o vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_count,vg_extent_size 03/02/2021 08:38:00 INFO stdout ceph-09209f5b-5f36-4ad4-82ff-17c0bfb7f60d";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 08:38:00 INFO stdout ceph-4ed4c220-9498-4351-8ce9-4e00f8b3a93b";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 08:38:00 INFO stdout ceph-d4270832-4236-4724-a362-2dc9fb139f6f";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 08:38:00 INFO stdout ceph-f3285994-1fed-4a78-a067-6141ffe2d08e";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 08:38:00 INFO Starting activating PetaSAN lvs 03/02/2021 08:38:00 INFO Running command: /sbin/vgs --noheadings --readonly --units=b --nosuffix --separator=";" -o vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_count,vg_extent_size 03/02/2021 08:38:00 INFO stdout ceph-09209f5b-5f36-4ad4-82ff-17c0bfb7f60d";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 08:38:00 INFO stdout ceph-4ed4c220-9498-4351-8ce9-4e00f8b3a93b";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 08:38:00 INFO stdout ceph-d4270832-4236-4724-a362-2dc9fb139f6f";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 08:38:00 INFO stdout ceph-f3285994-1fed-4a78-a067-6141ffe2d08e";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 08:38:00 INFO checking uptime 03/02/2021 08:38:00 INFO checking uptime of node: peta1 ip:10.10.0.151 03/02/2021 08:38:01 INFO checking uptime of node: peta2 ip:10.10.0.152 03/02/2021 08:38:01 INFO checking uptime of node: peta3 ip:10.10.0.153 03/02/2021 08:38:01 INFO checking uptime of node: peta3 is starting 03/02/2021 08:38:01 INFO Service is starting. 03/02/2021 08:38:01 INFO iSCSI Service dropping old sessions 03/02/2021 08:38:02 INFO Starting OSDs 03/02/2021 08:38:02 INFO Starting petasan tuning service 03/02/2021 08:38:02 INFO Starting Config Upload service 03/02/2021 08:38:02 INFO Starting petasan-qperf server 03/02/2021 08:38:10 INFO LeaderElectionBase successfully dropped old sessions 03/02/2021 08:38:16 INFO iSCSI Service successfully dropped old sessions 03/02/2021 08:38:16 INFO Cleaning unused configurations. 03/02/2021 08:38:16 INFO Cleaning all mapped disks 03/02/2021 08:38:16 INFO Cleaning unused rbd images. 03/02/2021 08:38:16 INFO Cleaning unused ips. 03/02/2021 11:34:29 ERROR Node Stats exception. 03/02/2021 11:34:29 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 0.83 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.03 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1024.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 30.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.2 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 167557.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 255457.28 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.19 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 156129.28 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 0.83 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.03 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1024.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 30.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.2 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 167557.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 255457.28 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.19 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 156129.28 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:35:32 ERROR Node Stats exception. 03/02/2021 11:35:32 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 3.19 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.07 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1177.6 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.21 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 98304.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 262471.68 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.2 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 90511.36 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 3.19 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.07 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1177.6 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.21 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 98304.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 262471.68 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.2 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 90511.36 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:36:36 ERROR Node Stats exception. 03/02/2021 11:36:36 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.disks.smart.sda.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sda.power_on_hours 84276 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sda.temperature_celsius 32 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.power_on_hours 84277 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.temperature_celsius 29 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.power_on_hours 20440 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.temperature_celsius 29 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdd.reallocated_sector_ct 0 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.disks.smart.sda.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sda.power_on_hours 84276 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sda.temperature_celsius 32 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.power_on_hours 84277 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.temperature_celsius 29 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.power_on_hours 20440 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.temperature_celsius 29 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdd.reallocated_sector_ct 0 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:37:40 ERROR Node Stats exception. 03/02/2021 11:37:40 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 0.83 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.08 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1986.56 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.17 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 164352.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 215183.36 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 153364.48 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 0.83 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.08 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1986.56 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.17 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 164352.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 215183.36 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 153364.48 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:38:43 ERROR Node Stats exception. 03/02/2021 11:38:43 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.17 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.07 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1044.48 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.2 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 129996.8 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 253429.76 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.19 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 121610.24 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.17 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.07 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1044.48 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.2 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 129996.8 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 253429.76 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.19 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 121610.24 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:39:46 ERROR Node Stats exception. 03/02/2021 11:39:46 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.33 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.07 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1423.36 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 124323.84 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 151459.84 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 115681.28 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.33 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.07 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1423.36 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 124323.84 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 151459.84 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 115681.28 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:40:49 ERROR Node Stats exception. 03/02/2021 11:40:49 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 2.01 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.08 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 952.32 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.21 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 174069.76 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 256440.32 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.19 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 162314.24 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 2.01 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.08 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 952.32 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.21 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 174069.76 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 256440.32 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.19 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 162314.24 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:41:52 ERROR Node Stats exception. 03/02/2021 11:41:52 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 2.16 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.11 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1536.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 40.96 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.14 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 121128.96 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 179220.48 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.14 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 113182.72 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 2.16 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.11 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1536.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 40.96 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.14 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 121128.96 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 179220.48 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.14 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 113182.72 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:42:55 ERROR Node Stats exception. 03/02/2021 11:42:55 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.83 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.19 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 798.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 30.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.14 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 149350.4 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 170997.76 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.13 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 140001.28 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.83 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.19 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 798.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 30.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.14 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 149350.4 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 170997.76 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.13 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 140001.28 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:43:59 ERROR Node Stats exception. 03/02/2021 11:43:59 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 0.83 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.18 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1740.8 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.17 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 130344.96 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 209674.24 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 121313.28 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 0.83 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.18 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1740.8 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.17 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 130344.96 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 209674.24 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 121313.28 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:45:02 ERROR Node Stats exception. 03/02/2021 11:45:02 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.17 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.19 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1024.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 92.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.21 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 146472.96 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 265758.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.2 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 136488.96 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.17 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.19 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1024.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 92.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.21 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 146472.96 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 265758.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.2 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 136488.96 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:46:05 ERROR Node Stats exception. 03/02/2021 11:46:05 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.33 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.19 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1198.08 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.22 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 190484.48 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 269854.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.21 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 177674.24 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.33 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.19 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1198.08 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.22 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 190484.48 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 269854.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.21 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 177674.24 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:47:09 ERROR Node Stats exception. 03/02/2021 11:47:09 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.disks.smart.sda.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sda.power_on_hours 84276 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sda.temperature_celsius 32 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.power_on_hours 84277 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.temperature_celsius 29 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.power_on_hours 20440 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.temperature_celsius 29 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdd.reallocated_sector_ct 0 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.disks.smart.sda.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sda.power_on_hours 84276 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sda.temperature_celsius 32 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.power_on_hours 84277 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.temperature_celsius 29 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.power_on_hours 20440 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.temperature_celsius 29 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdd.reallocated_sector_ct 0 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:48:13 ERROR Node Stats exception. 03/02/2021 11:48:13 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.0 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.22 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1280.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.22 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 159764.48 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 277022.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.21 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 149032.96 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.0 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.22 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1280.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.22 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 159764.48 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 277022.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.21 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 149032.96 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:49:16 ERROR Node Stats exception. 03/02/2021 11:49:16 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 0.34 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.22 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1085.44 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 30.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 155381.76 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 141445.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 145121.28 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 0.34 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.22 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1085.44 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 30.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 155381.76 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 141445.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 145121.28 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:50:19 ERROR Node Stats exception. 03/02/2021 11:50:19 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 2.66 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.23 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1392.64 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.27 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 147353.6 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 332308.48 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.25 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 136765.44 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 2.66 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.23 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1392.64 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.27 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 147353.6 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 332308.48 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.25 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 136765.44 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:51:22 ERROR Node Stats exception. 03/02/2021 11:51:22 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 2.34 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.24 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1361.92 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.24 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 141086.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 300728.32 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.23 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 131153.92 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 2.34 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.24 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1361.92 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.24 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 141086.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 300728.32 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.23 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 131153.92 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:52:25 ERROR Node Stats exception. 03/02/2021 11:52:25 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 2.49 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.25 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 983.04 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 155381.76 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 201871.36 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.15 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 145541.12 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 2.49 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.25 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 983.04 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 155381.76 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 201871.36 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.15 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 145541.12 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:53:29 ERROR Node Stats exception. 03/02/2021 11:53:29 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.67 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.3 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1566.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 114483.2 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 197826.56 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.15 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 106895.36 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.67 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.3 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1566.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 114483.2 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 197826.56 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.15 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 106895.36 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:54:32 ERROR Node Stats exception. 03/02/2021 11:54:32 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 2.0 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.3 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 952.32 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.18 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 190515.2 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 220528.64 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.17 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 178995.2 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 2.0 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.3 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 952.32 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.18 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 190515.2 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 220528.64 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.17 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 178995.2 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:55:35 ERROR Node Stats exception. 03/02/2021 11:55:35 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.67 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.3 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 901.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 103690.24 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 155197.44 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 96808.96 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.67 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.3 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 901.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 103690.24 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 155197.44 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 96808.96 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:56:38 ERROR Node Stats exception. 03/02/2021 11:56:38 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.0 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.29 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 747.52 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.2 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 174479.36 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 255959.04 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.19 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 162406.4 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.0 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.29 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 747.52 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.2 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 174479.36 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 255959.04 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.19 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 162406.4 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:57:43 ERROR Node Stats exception. 03/02/2021 11:57:43 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.disks.smart.sda.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sda.power_on_hours 84276 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sda.temperature_celsius 32 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.power_on_hours 84277 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.temperature_celsius 29 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.power_on_hours 20441 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.temperature_celsius 29 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdd.reallocated_sector_ct 0 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.disks.smart.sda.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sda.power_on_hours 84276 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sda.temperature_celsius 32 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.power_on_hours 84277 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.temperature_celsius 29 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.power_on_hours 20441 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.temperature_celsius 29 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdd.reallocated_sector_ct 0 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:58:46 ERROR Node Stats exception. 03/02/2021 11:58:46 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 0.83 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.29 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 983.04 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.09 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 115517.44 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 102440.96 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.09 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 108226.56 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 0.83 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.29 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 983.04 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.09 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 115517.44 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 102440.96 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.09 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 108226.56 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 11:59:49 ERROR Node Stats exception. 03/02/2021 11:59:49 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.0 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.32 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1003.52 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.22 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 190320.64 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 272906.24 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.21 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 177571.84 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.0 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.32 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1003.52 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.22 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 190320.64 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 272906.24 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.21 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 177571.84 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:00:52 ERROR Node Stats exception. 03/02/2021 12:00:52 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 0.51 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.33 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 2129.92 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.18 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 160020.48 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 219484.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.17 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 150159.36 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 0.51 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.33 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 2129.92 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.18 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 160020.48 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 219484.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.17 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 150159.36 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:01:55 ERROR Node Stats exception. 03/02/2021 12:01:55 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.18 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.32 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1413.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.2 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 178155.52 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 250152.96 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.19 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 166891.52 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.18 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.32 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1413.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.2 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 178155.52 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 250152.96 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.19 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 166891.52 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:02:58 ERROR Node Stats exception. 03/02/2021 12:02:58 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.51 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.36 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1105.92 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.15 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 98805.76 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 187228.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.14 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 91924.48 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.51 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.36 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1105.92 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.15 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 98805.76 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 187228.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.14 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 91924.48 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:04:02 ERROR Node Stats exception. 03/02/2021 12:04:02 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.83 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.35 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1884.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 128378.88 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 149360.64 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.11 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 119623.68 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.83 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.35 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1884.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 128378.88 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 149360.64 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.11 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 119623.68 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:05:05 ERROR Node Stats exception. 03/02/2021 12:05:05 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.84 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.42 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1003.52 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 30.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 137605.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 195348.48 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.15 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 128798.72 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.84 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.42 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1003.52 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 30.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 137605.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 195348.48 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.15 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 128798.72 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:06:08 ERROR Node Stats exception. 03/02/2021 12:06:08 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.49 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.44 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1105.92 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 116613.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 193914.88 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.15 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 108922.88 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.49 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.44 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1105.92 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 116613.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 193914.88 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.15 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 108922.88 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:07:11 ERROR Node Stats exception. 03/02/2021 12:07:11 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 0.17 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.44 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 890.88 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 30.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.1 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 130600.96 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 117084.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.1 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 122173.44 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 0.17 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.44 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 890.88 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 30.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.1 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 130600.96 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 117084.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.1 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 122173.44 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:08:16 ERROR Node Stats exception. 03/02/2021 12:08:16 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.disks.smart.sda.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sda.power_on_hours 84277 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sda.temperature_celsius 32 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.power_on_hours 84277 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.temperature_celsius 29 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.power_on_hours 20441 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.temperature_celsius 29 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdd.reallocated_sector_ct 0 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.disks.smart.sda.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sda.power_on_hours 84277 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sda.temperature_celsius 32 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.power_on_hours 84277 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.temperature_celsius 29 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.power_on_hours 20441 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.temperature_celsius 29 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdd.reallocated_sector_ct 0 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:09:19 ERROR Node Stats exception. 03/02/2021 12:09:19 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 0.83 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.44 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1228.8 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.23 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 168253.44 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 285286.4 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.22 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 157143.04 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 0.83 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.44 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1228.8 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.23 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 168253.44 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 285286.4 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.22 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 157143.04 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:10:22 ERROR Node Stats exception. 03/02/2021 12:10:22 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 0.67 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.45 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 727.04 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 157501.44 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 205004.8 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 147476.48 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 0.67 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.45 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 727.04 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 157501.44 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 205004.8 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 147476.48 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:11:25 ERROR Node Stats exception. 03/02/2021 12:11:25 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.33 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.47 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 839.68 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.15 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 123351.04 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 182292.48 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.14 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 114749.44 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.33 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.47 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 839.68 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.15 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 123351.04 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 182292.48 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.14 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 114749.44 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:12:28 ERROR Node Stats exception. 03/02/2021 12:12:28 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 2.81 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.52 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1198.08 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.19 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 155904.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 238673.92 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.18 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 145602.56 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 2.81 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.52 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1198.08 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.19 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 155904.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 238673.92 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.18 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 145602.56 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:13:32 ERROR Node Stats exception. 03/02/2021 12:13:32 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 2.0 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.51 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1177.6 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.15 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 164116.48 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 186951.68 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.14 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 153610.24 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 2.0 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.51 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1177.6 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.15 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 164116.48 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 186951.68 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.14 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 153610.24 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:14:35 ERROR Node Stats exception. 03/02/2021 12:14:35 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.0 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.5 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1669.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.18 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 168826.88 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 225638.4 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.17 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 157286.4 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.0 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.5 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1669.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.18 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 168826.88 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 225638.4 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.17 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 157286.4 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:15:38 ERROR Node Stats exception. 03/02/2021 12:15:38 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 0.5 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.51 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1218.56 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.18 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 157921.28 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 230369.28 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.17 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 147691.52 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 0.5 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.51 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1218.56 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.18 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 157921.28 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 230369.28 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.17 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 147691.52 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:16:41 ERROR Node Stats exception. 03/02/2021 12:16:41 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 0.67 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.51 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1218.56 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 146206.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 144332.8 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.11 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 136990.72 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 0.67 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.51 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1218.56 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 146206.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 144332.8 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.11 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 136990.72 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:17:44 ERROR Node Stats exception. 03/02/2021 12:17:44 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.0 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.52 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1495.04 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.19 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 128747.52 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 241408.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.18 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 119787.52 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.0 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.52 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1495.04 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.19 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 128747.52 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 241408.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.18 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 119787.52 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:18:49 ERROR Node Stats exception. 03/02/2021 12:18:49 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.disks.smart.sda.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sda.power_on_hours 84277 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sda.temperature_celsius 32 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.power_on_hours 84278 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.temperature_celsius 29 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.power_on_hours 20441 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.temperature_celsius 29 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdd.reallocated_sector_ct 0 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.disks.smart.sda.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sda.power_on_hours 84277 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sda.temperature_celsius 32 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.power_on_hours 84278 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.temperature_celsius 29 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.power_on_hours 20441 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.temperature_celsius 29 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdd.reallocated_sector_ct 0 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:19:52 ERROR Node Stats exception. 03/02/2021 12:19:52 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.17 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.56 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 808.96 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.18 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 182302.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 229877.76 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.17 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 170680.32 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.17 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.56 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 808.96 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.18 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 182302.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 229877.76 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.17 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 170680.32 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:20:55 ERROR Node Stats exception. 03/02/2021 12:20:55 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 2.17 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.57 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 942.08 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.17 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 158248.96 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 215746.56 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 147240.96 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 2.17 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.57 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 942.08 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.17 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 158248.96 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 215746.56 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 147240.96 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:21:58 ERROR Node Stats exception. 03/02/2021 12:21:58 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.84 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.57 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1003.52 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 128942.08 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 205731.84 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 120033.28 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.84 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.57 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1003.52 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 128942.08 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 205731.84 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 120033.28 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:23:02 ERROR Node Stats exception. 03/02/2021 12:23:02 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.17 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.57 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1054.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.15 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 173137.92 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 186449.92 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.14 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 162201.6 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.17 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.57 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1054.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.15 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 173137.92 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 186449.92 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.14 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 162201.6 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:24:05 ERROR Node Stats exception. 03/02/2021 12:24:05 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 0.83 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.61 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 727.04 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.17 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 155084.8 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 207380.48 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 144721.92 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 0.83 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.61 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 727.04 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.17 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 155084.8 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 207380.48 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 144721.92 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:25:08 ERROR Node Stats exception. 03/02/2021 12:25:08 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 0.83 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.61 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1464.32 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 20.48 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.19 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 159488.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 237926.4 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.18 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 148664.32 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 0.83 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.61 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1464.32 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 20.48 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.19 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 159488.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 237926.4 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.18 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 148664.32 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:26:11 ERROR Node Stats exception. 03/02/2021 12:26:11 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 0.67 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.6 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1617.92 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.15 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 144680.96 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 191631.36 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.15 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 134778.88 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 0.67 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.6 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1617.92 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.15 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 144680.96 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 191631.36 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.15 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 134778.88 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:27:14 ERROR Node Stats exception. 03/02/2021 12:27:14 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 0.67 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.62 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1044.48 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.23 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 141844.48 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 287191.04 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.22 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 132874.24 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 0.67 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.62 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1044.48 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.23 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 141844.48 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 287191.04 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.22 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 132874.24 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:28:17 ERROR Node Stats exception. 03/02/2021 12:28:17 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 11.85 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.99 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 2437.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 1259.52 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.18 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 148398.08 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 230041.6 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.18 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 138321.92 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 11.85 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.99 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 2437.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 1259.52 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.18 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 148398.08 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 230041.6 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.18 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 138321.92 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:29:22 ERROR Node Stats exception. 03/02/2021 12:29:22 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.disks.smart.sda.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sda.power_on_hours 84277 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sda.temperature_celsius 32 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.power_on_hours 84278 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.temperature_celsius 29 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.power_on_hours 20441 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.temperature_celsius 29 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdd.reallocated_sector_ct 0 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.disks.smart.sda.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sda.power_on_hours 84277 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sda.temperature_celsius 32 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.power_on_hours 84278 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.temperature_celsius 29 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.power_on_hours 20441 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.temperature_celsius 29 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdd.reallocated_sector_ct 0 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:30:25 ERROR Node Stats exception. 03/02/2021 12:30:25 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.51 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.71 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1136.64 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.15 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 135055.36 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 186849.28 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.14 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 125808.64 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.51 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.71 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1136.64 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.15 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 135055.36 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 186849.28 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.14 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 125808.64 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:31:28 ERROR Node Stats exception. 03/02/2021 12:31:28 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.5 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.71 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1085.44 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.19 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 164188.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 236298.24 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.18 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 153395.2 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.5 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.71 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1085.44 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.19 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 164188.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 236298.24 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.18 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 153395.2 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:32:32 ERROR Node Stats exception. 03/02/2021 12:32:32 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 2.33 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.71 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 901.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 137349.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 155269.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 128286.72 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 2.33 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.71 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 901.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 137349.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 155269.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 128286.72 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:33:35 ERROR Node Stats exception. 03/02/2021 12:33:35 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.17 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.71 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1044.48 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 30.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.15 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 117493.76 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 187883.52 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.14 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 109107.2 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.17 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.71 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1044.48 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 30.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.15 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 117493.76 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 187883.52 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.14 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 109107.2 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:34:38 ERROR Node Stats exception. 03/02/2021 12:34:38 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.0 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.71 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 972.8 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.19 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 160450.56 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 238981.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.18 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 150476.8 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.0 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.71 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 972.8 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.19 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 160450.56 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 238981.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.18 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 150476.8 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:35:41 ERROR Node Stats exception. 03/02/2021 12:35:41 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 0.17 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 860.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.13 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 156508.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 149493.76 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 146749.44 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 0.17 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 860.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.13 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 156508.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 149493.76 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 146749.44 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:36:44 ERROR Node Stats exception. 03/02/2021 12:36:44 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.66 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.77 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1341.44 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.21 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 149637.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 262389.76 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.2 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 139663.36 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.66 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.77 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1341.44 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.21 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 149637.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 262389.76 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.2 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 139663.36 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:37:47 ERROR Node Stats exception. 03/02/2021 12:37:47 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.0 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.76 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1208.32 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.19 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 152678.4 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 235059.2 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.18 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 142243.84 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.0 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.76 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1208.32 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.19 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 152678.4 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 235059.2 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.18 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 142243.84 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:38:51 ERROR Node Stats exception. 03/02/2021 12:38:51 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 0.83 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.76 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1116.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 61.44 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.1 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 128512.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 115558.4 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.1 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 120524.8 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 0.83 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.76 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1116.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 61.44 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.1 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 128512.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 115558.4 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.1 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 120524.8 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:39:55 ERROR Node Stats exception. 03/02/2021 12:39:55 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.disks.smart.sda.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sda.power_on_hours 84277 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sda.temperature_celsius 32 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.power_on_hours 84278 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.temperature_celsius 29 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.power_on_hours 20441 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.temperature_celsius 29 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdd.reallocated_sector_ct 0 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.disks.smart.sda.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sda.power_on_hours 84277 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sda.temperature_celsius 32 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.power_on_hours 84278 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.temperature_celsius 29 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.power_on_hours 20441 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.temperature_celsius 29 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdd.reallocated_sector_ct 0 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:40:58 ERROR Node Stats exception. 03/02/2021 12:40:58 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 2.34 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.86 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1873.92 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.28 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 171304.96 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 347115.52 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.26 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 159150.08 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 2.34 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 14.86 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1873.92 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.28 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 171304.96 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 347115.52 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.26 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 159150.08 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:42:02 ERROR Node Stats exception. 03/02/2021 12:42:02 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 2.0 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 15.04 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1198.08 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.23 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 169052.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 289095.68 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.22 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 156866.56 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 2.0 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 15.04 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1198.08 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.23 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 169052.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 289095.68 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.22 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 156866.56 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:43:05 ERROR Node Stats exception. 03/02/2021 12:43:05 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 4.0 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 15.26 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1218.56 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 30.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.3 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 213852.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 380190.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.29 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 199536.64 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 4.0 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 15.26 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1218.56 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 30.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.3 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 213852.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 380190.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.29 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 199536.64 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 12:44:08 ERROR Node Stats exception. 03/02/2021 12:44:08 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.33 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 15.55 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 962.56 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 30.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.19 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 168130.56 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 231557.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.18 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 156753.92 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.33 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 15.55 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 962.56 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 30.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.19 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 168130.56 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 231557.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.18 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 156753.92 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 13:00:33 INFO Stopping disk 00001 03/02/2021 13:00:33 ERROR Could not find ips for image-00001 03/02/2021 13:00:33 ERROR LIO error deleting Target None, maybe the iqn is not exists. 03/02/2021 13:00:33 ERROR Cannot unmap image image-00001. error, no mapped images found. 03/02/2021 13:00:35 INFO Found pool:rbd for disk:00001 via consul 03/02/2021 13:00:35 INFO PetaSAN removed key of stopped disk 00001 from consul. 03/02/2021 14:01:50 INFO Start settings IPs 03/02/2021 14:01:58 INFO GlusterFS mount attempt 03/02/2021 14:02:06 INFO str_start_command: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consul agent -raft-protocol 2 -config-dir /opt/petasan/config/etc/consul.d/server -bind 172.16.0.153 -retry-join 172.16.0.151 -retry-join 172.16.0.152 03/02/2021 14:02:09 INFO Starting cluster file sync service 03/02/2021 14:02:10 INFO LeaderElectionBase dropping old sessions 03/02/2021 14:02:10 INFO Starting iSCSI Service 03/02/2021 14:02:10 INFO Starting Cluster Management application 03/02/2021 14:02:10 INFO Starting Node Stats Service 03/02/2021 14:02:10 INFO Starting activating PetaSAN lvs 03/02/2021 14:02:10 INFO Running command: /sbin/vgs --noheadings --readonly --units=b --nosuffix --separator=";" -o vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_count,vg_extent_size 03/02/2021 14:02:10 INFO stdout ceph-09209f5b-5f36-4ad4-82ff-17c0bfb7f60d";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 14:02:10 INFO stdout ceph-4ed4c220-9498-4351-8ce9-4e00f8b3a93b";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 14:02:10 INFO stdout ceph-d4270832-4236-4724-a362-2dc9fb139f6f";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 14:02:10 INFO stdout ceph-f3285994-1fed-4a78-a067-6141ffe2d08e";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 14:02:10 INFO Starting activating PetaSAN lvs 03/02/2021 14:02:10 INFO Running command: /sbin/vgs --noheadings --readonly --units=b --nosuffix --separator=";" -o vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_count,vg_extent_size 03/02/2021 14:02:11 INFO stdout ceph-09209f5b-5f36-4ad4-82ff-17c0bfb7f60d";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 14:02:11 INFO stdout ceph-4ed4c220-9498-4351-8ce9-4e00f8b3a93b";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 14:02:11 INFO stdout ceph-d4270832-4236-4724-a362-2dc9fb139f6f";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 14:02:11 INFO stdout ceph-f3285994-1fed-4a78-a067-6141ffe2d08e";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 14:02:11 INFO checking uptime 03/02/2021 14:02:11 INFO checking uptime of node: peta1 ip:10.10.0.151 03/02/2021 14:02:11 INFO checking uptime of node: peta2 ip:10.10.0.152 03/02/2021 14:02:12 INFO checking uptime of node: peta2 is starting 03/02/2021 14:02:12 INFO checking uptime of node: peta3 ip:10.10.0.153 03/02/2021 14:02:13 INFO Starting OSDs 03/02/2021 14:02:13 INFO checking uptime of node: peta3 is starting 03/02/2021 14:02:13 INFO Cluster is just starting, system will delete all active disk resources 03/02/2021 14:02:13 INFO Starting petasan tuning service 03/02/2021 14:02:13 INFO Starting Config Upload service 03/02/2021 14:02:13 INFO Service is starting. 03/02/2021 14:02:13 INFO iSCSI Service dropping old sessions 03/02/2021 14:02:13 INFO Starting petasan-qperf server 03/02/2021 14:02:20 INFO LeaderElectionBase successfully dropped old sessions 03/02/2021 14:02:28 INFO iSCSI Service successfully dropped old sessions 03/02/2021 14:02:28 INFO Cleaning unused configurations. 03/02/2021 14:02:28 INFO Cleaning all mapped disks 03/02/2021 14:02:28 INFO Cleaning unused rbd images. 03/02/2021 14:02:28 INFO Cleaning unused ips. 03/02/2021 14:02:31 INFO GlusterFS mount attempt 03/02/2021 14:03:49 INFO Start settings IPs 03/02/2021 14:03:55 INFO GlusterFS mount attempt 03/02/2021 14:04:03 INFO str_start_command: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consul agent -raft-protocol 2 -config-dir /opt/petasan/config/etc/consul.d/server -bind 172.16.0.153 -retry-join 172.16.0.151 -retry-join 172.16.0.152 03/02/2021 14:04:07 INFO Starting cluster file sync service 03/02/2021 14:04:07 INFO Starting iSCSI Service 03/02/2021 14:04:07 INFO Starting Cluster Management application 03/02/2021 14:04:07 INFO LeaderElectionBase dropping old sessions 03/02/2021 14:04:07 INFO Starting Node Stats Service 03/02/2021 14:04:07 INFO Starting activating PetaSAN lvs 03/02/2021 14:04:07 INFO Running command: /sbin/vgs --noheadings --readonly --units=b --nosuffix --separator=";" -o vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_count,vg_extent_size 03/02/2021 14:04:07 INFO stdout ceph-09209f5b-5f36-4ad4-82ff-17c0bfb7f60d";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 14:04:07 INFO stdout ceph-4ed4c220-9498-4351-8ce9-4e00f8b3a93b";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 14:04:07 INFO stdout ceph-d4270832-4236-4724-a362-2dc9fb139f6f";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 14:04:07 INFO stdout ceph-f3285994-1fed-4a78-a067-6141ffe2d08e";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 14:04:07 INFO Starting activating PetaSAN lvs 03/02/2021 14:04:07 INFO Running command: /sbin/vgs --noheadings --readonly --units=b --nosuffix --separator=";" -o vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_count,vg_extent_size 03/02/2021 14:04:07 INFO stdout ceph-09209f5b-5f36-4ad4-82ff-17c0bfb7f60d";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 14:04:07 INFO stdout ceph-4ed4c220-9498-4351-8ce9-4e00f8b3a93b";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 14:04:07 INFO stdout ceph-d4270832-4236-4724-a362-2dc9fb139f6f";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 14:04:07 INFO stdout ceph-f3285994-1fed-4a78-a067-6141ffe2d08e";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 14:04:07 INFO checking uptime 03/02/2021 14:04:07 INFO checking uptime of node: peta1 ip:10.10.0.151 03/02/2021 14:04:08 INFO checking uptime of node: peta2 ip:10.10.0.152 03/02/2021 14:04:08 INFO checking uptime of node: peta3 ip:10.10.0.153 03/02/2021 14:04:08 INFO checking uptime of node: peta3 is starting 03/02/2021 14:04:08 INFO Service is starting. 03/02/2021 14:04:08 INFO iSCSI Service dropping old sessions 03/02/2021 14:04:10 INFO Starting OSDs 03/02/2021 14:04:10 INFO Starting petasan tuning service 03/02/2021 14:04:10 INFO Starting Config Upload service 03/02/2021 14:04:10 INFO Starting petasan-qperf server 03/02/2021 14:04:17 INFO LeaderElectionBase successfully dropped old sessions 03/02/2021 14:04:23 INFO iSCSI Service successfully dropped old sessions 03/02/2021 14:04:23 INFO Cleaning unused configurations. 03/02/2021 14:04:23 INFO Cleaning all mapped disks 03/02/2021 14:04:23 INFO Cleaning unused rbd images. 03/02/2021 14:04:23 INFO Cleaning unused ips. 03/02/2021 20:02:43 INFO Start settings IPs 03/02/2021 20:02:48 INFO GlusterFS mount attempt 03/02/2021 20:02:57 INFO str_start_command: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consul agent -raft-protocol 2 -config-dir /opt/petasan/config/etc/consul.d/server -bind 172.16.0.153 -retry-join 172.16.0.151 -retry-join 172.16.0.152 03/02/2021 20:03:00 INFO Starting cluster file sync service 03/02/2021 20:03:01 INFO Starting iSCSI Service 03/02/2021 20:03:01 INFO LeaderElectionBase dropping old sessions 03/02/2021 20:03:01 INFO Starting Cluster Management application 03/02/2021 20:03:01 INFO Starting Node Stats Service 03/02/2021 20:03:01 INFO Starting activating PetaSAN lvs 03/02/2021 20:03:01 INFO Running command: /sbin/vgs --noheadings --readonly --units=b --nosuffix --separator=";" -o vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_count,vg_extent_size 03/02/2021 20:03:01 INFO stdout ceph-09209f5b-5f36-4ad4-82ff-17c0bfb7f60d";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 20:03:01 INFO stdout ceph-4ed4c220-9498-4351-8ce9-4e00f8b3a93b";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 20:03:01 INFO stdout ceph-d4270832-4236-4724-a362-2dc9fb139f6f";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 20:03:01 INFO stdout ceph-f3285994-1fed-4a78-a067-6141ffe2d08e";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 20:03:01 INFO Starting activating PetaSAN lvs 03/02/2021 20:03:01 INFO Running command: /sbin/vgs --noheadings --readonly --units=b --nosuffix --separator=";" -o vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_count,vg_extent_size 03/02/2021 20:03:01 INFO stdout ceph-09209f5b-5f36-4ad4-82ff-17c0bfb7f60d";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 20:03:01 INFO stdout ceph-4ed4c220-9498-4351-8ce9-4e00f8b3a93b";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 20:03:01 INFO stdout ceph-d4270832-4236-4724-a362-2dc9fb139f6f";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 20:03:01 INFO stdout ceph-f3285994-1fed-4a78-a067-6141ffe2d08e";"1";"1";"wz--n-";"476931";"0";"4194304 03/02/2021 20:03:01 INFO checking uptime 03/02/2021 20:03:01 INFO checking uptime of node: peta1 ip:10.10.0.151 03/02/2021 20:03:01 INFO checking uptime of node: peta2 ip:10.10.0.152 03/02/2021 20:03:02 INFO checking uptime of node: peta3 ip:10.10.0.153 03/02/2021 20:03:02 INFO checking uptime of node: peta3 is starting 03/02/2021 20:03:02 INFO Service is starting. 03/02/2021 20:03:02 INFO iSCSI Service dropping old sessions 03/02/2021 20:03:03 INFO Starting OSDs 03/02/2021 20:03:03 INFO Starting petasan tuning service 03/02/2021 20:03:03 INFO Starting Config Upload service 03/02/2021 20:03:03 INFO Starting petasan-qperf server 03/02/2021 20:03:11 INFO LeaderElectionBase successfully dropped old sessions 03/02/2021 20:03:17 INFO iSCSI Service successfully dropped old sessions 03/02/2021 20:03:17 INFO Cleaning unused configurations. 03/02/2021 20:03:17 INFO Cleaning all mapped disks 03/02/2021 20:03:17 INFO Cleaning unused rbd images. 03/02/2021 20:03:17 INFO Cleaning unused ips. 03/02/2021 20:06:16 ERROR Node Stats exception. 03/02/2021 20:06:16 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 3.71 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 19.25 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1003.52 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 153.6 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.13 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 158986.24 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 154624.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 145489.92 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 3.71 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 19.25 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1003.52 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 153.6 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.13 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 158986.24 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 154624.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.12 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 145489.92 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 20:06:32 INFO Checking backend latencies : 03/02/2021 20:06:32 INFO Network latency for backend 172.16.0.151 = 03/02/2021 20:06:32 INFO Network latency for backend 172.16.0.152 = 03/02/2021 20:06:32 INFO Network latency for backend 172.16.0.153 = 7.24 us 03/02/2021 20:06:35 INFO Checking backend latencies : 03/02/2021 20:06:35 INFO Network latency for backend 172.16.0.151 = 03/02/2021 20:06:35 INFO Network latency for backend 172.16.0.152 = 48.2 us 03/02/2021 20:06:35 INFO Network latency for backend 172.16.0.153 = 03/02/2021 20:06:35 ERROR Traceback (most recent call last): File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync index, data = base.watch(self.root_path, current_index) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch index, data = cons.kv.get(key, index=current_index, recurse=True) File "/usr/lib/python3/dist-packages/consul/base.py", line 554, in get params=params) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 85, in get raise e File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get raise RetryConsulException() PetaSAN.core.consul.ps_consul.RetryConsulException 03/02/2021 20:06:45 INFO Checking backend latencies : 03/02/2021 20:06:45 INFO Network latency for backend 172.16.0.151 = 03/02/2021 20:06:45 INFO Network latency for backend 172.16.0.152 = 03/02/2021 20:06:45 INFO Network latency for backend 172.16.0.153 = 03/02/2021 20:07:19 ERROR Node Stats exception. 03/02/2021 20:07:19 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 2.18 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 19.88 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 890.88 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.21 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 262942.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 101160.96 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.2 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 249794.56 `date +%s`" | nc -q0 10.10.0.151 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 2.18 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 19.88 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 890.88 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.21 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 262942.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 101160.96 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.2 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 249794.56 `date +%s`" | nc -q0 10.10.0.151 2003 03/02/2021 20:12:37 ERROR Node Stats exception. 03/02/2021 20:12:37 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.disks.smart.sda.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sda.power_on_hours 84284 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sda.temperature_celsius 33 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.power_on_hours 84285 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.temperature_celsius 30 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.power_on_hours 20449 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.temperature_celsius 29 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdd.reallocated_sector_ct 0 `date +%s`" | nc -q0 10.10.0.152 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.disks.smart.sda.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sda.power_on_hours 84284 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sda.temperature_celsius 33 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.power_on_hours 84285 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdb.temperature_celsius 30 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.reallocated_sector_ct 0 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.power_on_hours 20449 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdc.temperature_celsius 29 `date +%s`" " PetaSAN.NodeStats.peta3.disks.smart.sdd.reallocated_sector_ct 0 `date +%s`" | nc -q0 10.10.0.152 2003 03/02/2021 20:13:40 ERROR Node Stats exception. 03/02/2021 20:13:40 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.17 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 20.59 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1617.92 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 202721.28 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 181452.8 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.15 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 190208.0 `date +%s`" | nc -q0 10.10.0.152 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.17 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 20.59 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1617.92 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.16 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 202721.28 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 181452.8 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.15 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 190208.0 `date +%s`" | nc -q0 10.10.0.152 2003 03/02/2021 20:14:43 ERROR Node Stats exception. 03/02/2021 20:14:43 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 2.34 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 20.64 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1658.88 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.25 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 315822.08 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 166645.76 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.24 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 299171.84 `date +%s`" | nc -q0 10.10.0.152 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 2.34 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 20.64 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1658.88 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.25 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 315822.08 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 166645.76 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.24 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 299171.84 `date +%s`" | nc -q0 10.10.0.152 2003 03/02/2021 20:15:46 ERROR Node Stats exception. 03/02/2021 20:15:46 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.67 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 20.65 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1136.64 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.23 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 282234.88 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 249118.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.21 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 265768.96 `date +%s`" | nc -q0 10.10.0.152 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 1.67 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 20.65 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 1136.64 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.23 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 282234.88 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 249118.72 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.21 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 265768.96 `date +%s`" | nc -q0 10.10.0.152 2003 04/02/2021 03:47:46 ERROR Node Stats exception. 04/02/2021 03:47:46 ERROR Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 2.02 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 27.65 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 2467.84 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.18 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 230051.84 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 230471.68 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.18 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 215644.16 `date +%s`" | nc -q0 10.10.0.152 2003 Traceback (most recent call last): File "/opt/petasan/scripts/node_stats.py", line 168, in get_stats() File "/opt/petasan/scripts/node_stats.py", line 66, in get_stats graphite_sender.send(leader_ip) File "/usr/lib/python3/dist-packages/PetaSAN/core/common/graphite_sender.py", line 59, in send raise Exception("Error running echo command :" + cmd) Exception: Error running echo command :echo "PetaSAN.NodeStats.peta3.cpu_all.percent_util 2.02 `date +%s`" " PetaSAN.NodeStats.peta3.memory.percent_util 27.65 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth0 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_received 2467.84 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth0_transmitted 0.0 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1 0.18 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_received 230051.84 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1_transmitted 230471.68 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.percent_util.eth1-100 0.18 `date +%s`" " PetaSAN.NodeStats.peta3.ifaces.throughput.eth1-100_received 215644.16 `date +%s`" | nc -q0 10.10.0.152 2003 04/02/2021 06:25:27 INFO GlusterFS mount attempt