Vol_id: 94cef51f-33d2-4a9a-a93c-2245f2faa120

Node02
Restart of nod02is happening here:
2017-04-25 08:47:24 827221 +0200 - ovs-node02-1604 - 1820/0x00007fa2f0484700 - volumedriverfs/DataStoreNG - 0000000000002e99 - info - updateCurrentSCO_: created new write SCO 00_00000034_00 for Volume 94cef51f-33d2-4a9a-a93c-2245f2faa120
2017-04-25 08:47:24 827555 +0200 - ovs-node02-1604 - 1820/0x00007fa2ed47e700 - volumedriverfs/BackendConnectionInterfaceLogger - 0000000000002e9a - info - Logger: Entering write 94cef51f-33d2-4a9a-a93c-2245f2faa120 00_00000033_00
2017-04-25 08:47:24 905841 +0200 - ovs-node02-1604 - 1820/0x00007fa2ed47e700 - volumedriverfs/BackendConnectionInterfaceLogger - 0000000000002e9b - info - ~Logger: Exiting write for 94cef51f-33d2-4a9a-a93c-2245f2faa120 00_00000033_00
2017-04-25 08:47:24 906210 +0200 - ovs-node02-1604 - 1820/0x00007fa2ed47e700 - volumedriverfs/BackendConnectionInterfaceLogger - 0000000000002e9c - info - ~Logger: Exiting write for 94cef51f-33d2-4a9a-a93c-2245f2faa120
Starting Open vStorage StorageDriver...
Tue Apr 25 08:52:38 CEST 2017 [1906]: No officially blessed alternative allocator found, using libc's allocator
Tue Apr 25 08:52:38 CEST 2017 [1906]: args: -f --config arakoon://config/ovs/vpools/dd08954b-aa14-4e80-ac3f-c29cf31a8134/hosts/mysyncvpoolA3xv4A5oeaXNGpkx/config?ini=%2Fopt%2FOpenvStorage%2Fconfig%2Farakoon_cacc.ini --lock-file /opt/OpenvStorage/run/storagedriver_mysyncv
pool.lock --logrotation --mountpoint /mnt/mysyncvpool --logsink console: -o big_writes -o sync_read -o allow_other -o use_ino -o default_permissions -o uid=1001 -o gid=1001 -o umask=0002
2017-04-25 08:52:39 064485 +0200 - ovs-node02-1604 - 1906/0x00007f29ab073940 - volumedriverfs/volumedriver_fs - 0000000000000000 - notice - print_version_data: version: 6.16.1
version revision: de7400cd8dcfa6a1336ad0a9c4ac1b5321b0253a

Node03
Libovsvolmedriver
Apr 25 08:49:11 ovs-node03-1604 fio.bin.latest: 2017-04-25 08:49:11 664185 +0200 - ovs-node03-1604 - 12831/0x00007fe01aab1700 - logger/libovsvolumedriver - 0000000000000008 - error - libovsvolumedriver::NetworkHAContext(0x152cbe0) - maybe_update_volume_location(407): failed to retrieve location of ci_scenario_hypervisor_ha_test_vdisk000: Input/output error
Apr 25 08:49:11 ovs-node03-1604 fio.bin.latest: 2017-04-25 08:49:11 669439 +0200 - ovs-node03-1604 - 12831/0x00007fe01aab1700 - logger/libovsvolumedriver - 0000000000000009 - info - libovsvolumedriver::NetworkXioClient(0x7fe00c001bb0) - NetworkXioClient(141): uri: tcp://10.100.199.173:26203, queue depth: 64
Apr 25 08:49:16 ovs-node03-1604 fio.bin.latest: 2017-04-25 08:49:16 682927 +0200 - ovs-node03-1604 - 12831/0x00007fe01aab1700 - logger/libovsvolumedriver - 000000000000000a - error - libovsvolumedriver::NetworkHAContext(0x152cbe0) - do_reconnect(296): failed to open ci_scenario_hypervisor_ha_test_vdisk000 at tcp://10.100.199.173:26203
Apr 25 08:49:16 ovs-node03-1604 fio.bin.latest: 2017-04-25 08:49:16 686818 +0200 - ovs-node03-1604 - 12831/0x00007fe01aab1700 - logger/libovsvolumedriver - 000000000000000b - error - libovsvolumedriver::NetworkHAContext(0x152cbe0) - reconnect(342): reconnection to URI 'tcp://10.100.199.173:26203' failed


Node03
Volumedriver

2017-04-25 08:49:16 013262 +0200 - ovs-node03-1604 - 4052/0x00007fa40ffff700 - volumedriverfs/XMLRPCTimingWrapper - 00000000000053a3 - info - execute: Arguments for volumePerformanceCountersV3 are {[reset:false,volume_id:94cef51f-33d2-4a9a-a93c-2245f2faa120,vrouter_clust
er_id:dd08954b-aa14-4e80-ac3f-c29cf31a8134]}
2017-04-25 08:49:16 013441 +0200 - ovs-node03-1604 - 4052/0x00007fa40ffff700 - volumedriverfs/XMLRPCRedirectWrapper - 00000000000053a4 - info - execute: Object 94cef51f-33d2-4a9a-a93c-2245f2faa120 is not present on this node - figuring out if it lives elsewhere.
2017-04-25 08:49:16 013878 +0200 - ovs-node03-1604 - 4052/0x00007fa40ffff700 - volumedriverfs/XMLRPCRedirectWrapper - 00000000000053a5 - info - execute: Redirecting to 10.100.199.172, port 26201
2017-04-25 08:49:16 013954 +0200 - ovs-node03-1604 - 4052/0x00007fa40ffff700 - volumedriverfs/XMLRPCTimingWrapper - 00000000000053a6 - info - execute: Call volumePerformanceCountersV3 took 0.000612 seconds
2017-04-25 08:49:16 014079 +0200 - ovs-node03-1604 - 4052/0x00007fa40ffff700 - volumedriverfs/XMLRPCConnection - 00000000000053a7 - info - operator(): Closing socket 72
2017-04-25 08:49:16 682337 +0200 - ovs-node03-1604 - 4052/0x00007fa440ff8700 - volumedriverfs/VFSRemoteNode - 00000000000053a8 - info - handle_: mysyncvpoolA3xv4A5oeaXNGpkx: remote did not respond within 5000 milliseconds milliseconds - giving up
2017-04-25 08:49:16 682622 +0200 - ovs-node03-1604 - 4052/0x00007fa440ff8700 - volumedriverfs/NetworkXioIOHandler - 00000000000053a9 - error - handle_open: failed to open volume 'ci_scenario_hypervisor_ha_test_vdisk000' request to remote node timed out
2017-04-25 08:49:19 558231 +0200 - ovs-node03-1604 - 4052/0x00007fa4177fa700 - volumedriverfs/VFSRemoteNode - 00000000000053aa - warning - check_keepalive_: mysyncvpoolA3xv4A5oeaXNGpkx: keepalive probe 2 pending
2017-04-25 08:49:23 502787 +0200 - ovs-node03-1604 - 4052/0x00007fa4437fe700 - volumedriverfs/MTServer - 00000000000053ab - info - work: You have 2 connections running
2017-04-25 08:49:23 502885 +0200 - ovs-node03-1604 - 4052/0x00007fa4437fe700 - volumedriverfs/MTServer - 00000000000053ac - info - cleanup_threads_locked: Trying to join 2 threads
2017-04-25 08:49:23 502956 +0200 - ovs-node03-1604 - 4052/0x00007fa40f7fe700 - volumedriverfs/XMLRPCTimingWrapper - 00000000000053ad - info - execute: Arguments for volumePerformanceCountersV3 are {[reset:false,volume_id:94cef51f-33d2-4a9a-a93c-2245f2faa120,vrouter_clust
er_id:dd08954b-aa14-4e80-ac3f-c29cf31a8134]}
2017-04-25 08:49:23 503110 +0200 - ovs-node03-1604 - 4052/0x00007fa40f7fe700 - volumedriverfs/XMLRPCRedirectWrapper - 00000000000053ae - info - execute: Object 94cef51f-33d2-4a9a-a93c-2245f2faa120 is not present on this node - figuring out if it lives elsewhere.
2017-04-25 08:49:23 503804 +0200 - ovs-node03-1604 - 4052/0x00007fa40f7fe700 - volumedriverfs/XMLRPCRedirectWrapper - 00000000000053af - info - execute: Redirecting to 10.100.199.172, port 26201
2017-04-25 08:49:23 503851 +0200 - ovs-node03-1604 - 4052/0x00007fa40f7fe700 - volumedriverfs/XMLRPCTimingWrapper - 00000000000053b0 - info - execute: Call volumePerformanceCountersV3 took 0.000842 seconds
2017-04-25 08:49:23 503929 +0200 - ovs-node03-1604 - 4052/0x00007fa40f7fe700 - volumedriverfs/XMLRPCConnection - 00000000000053b1 - info - operator(): Closing socket 25
2017-04-25 08:49:25 520240 +0200 - ovs-node03-1604 - 4052/0x00007fa4437fe700 - volumedriverfs/MTServer - 00000000000053b2 - info - work: You have 2 connections running
2017-04-25 08:49:25 520320 +0200 - ovs-node03-1604 - 4052/0x00007fa4437fe700 - volumedriverfs/MTServer - 00000000000053b3 - info - cleanup_threads_locked: Trying to join 2 threads
2017-04-25 08:49:25 520359 +0200 - ovs-node03-1604 - 4052/0x00007fa40ffff700 - volumedriverfs/XMLRPCTimingWrapper - 00000000000053b4 - info - execute: Arguments for volumeInfo are {[redirect_fenced:true,volume_id:94cef51f-33d2-4a9a-a93c-2245f2faa120,vrouter_cluster_id:dd
08954b-aa14-4e80-ac3f-c29cf31a8134]}