API tools faq
paste
Guest User

Untitled

a guest
Dec 14th, 2025
125
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 479.62 KB | None | 0 0
raw download clone embed print report
  1. 12/14 0:17:37.414547 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (9969 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 9114100, expcmdsn = 9114099, maxcmdsn = 9114338.
  2. 12/14 0:17:37.414590 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 9114084, itt = 9114168)
  3. 12/14 0:17:37.414672 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 9114085, itt = 9114169)
  4. 12/14 0:17:37.414687 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 9114086, itt = 9114170)
  5. 12/14 0:17:37.414694 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 9114087, itt = 9114171)
  6. 12/14 0:17:37.414703 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 9114088, itt = 9114172)
  7. 12/14 0:17:37.414762 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 9114089, itt = 9114173)
  8. 12/14 0:17:37.414770 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 9114090, itt = 9114174)
  9. 12/14 0:17:37.414778 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 9114091, itt = 9114175)
  10. 12/14 0:17:37.414789 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BF88 has failed due to unidentified error.
  11. 12/14 0:17:37.414797 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  12. 12/14 0:17:37.414944 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97EF0 synchronization task has been failed!
  13. 12/14 0:17:37.414970 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8268 has failed due to unidentified error.
  14. 12/14 0:17:37.414977 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  15. 12/14 0:17:37.414982 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F2F0EB40 synchronization task has been failed!
  16. 12/14 0:17:37.414991 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E8699A68 has failed due to unidentified error.
  17. 12/14 0:17:37.414997 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  18. 12/14 0:17:37.415002 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F2F0E2A0 synchronization task has been failed!
  19. 12/14 0:17:37.415010 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86999E8 has failed due to unidentified error.
  20. 12/14 0:17:37.415016 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  21. 12/14 0:17:37.415021 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F95990 synchronization task has been failed!
  22. 12/14 0:17:37.415030 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EC4F39D8 has failed due to unidentified error.
  23. 12/14 0:17:37.415036 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  24. 12/14 0:17:37.415041 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F2F0DCE0 synchronization task has been failed!
  25. 12/14 0:17:37.415049 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8568 has failed due to unidentified error.
  26. 12/14 0:17:37.415055 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  27. 12/14 0:17:37.415059 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F2F0D440 synchronization task has been failed!
  28. 12/14 0:17:37.415067 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7BE8 has failed due to unidentified error.
  29. 12/14 0:17:37.415074 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  30. 12/14 0:17:37.415078 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96510 synchronization task has been failed!
  31. 12/14 0:17:37.415086 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7D68 has failed due to unidentified error.
  32. 12/14 0:17:37.415092 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  33. 12/14 0:17:37.415097 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96DB0 synchronization task has been failed!
  34. 12/14 0:17:37.415353 4798 func: >>> CHAPartnerNode::SetSyncStatus
  35. 12/14 0:17:37.415390 4798 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  36. 12/14 0:17:37.415403 4798 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 3 from 2. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  37. 12/14 0:17:37.415415 4798 func: >>> HANode::reportEvent
  38. 12/14 0:17:37.415543 4798 func: >>> iScsiServer::sendNotification
  39. 12/14 0:17:37.415557 4798 func: >>> CEventDataBase::AddRecord
  40. 12/14 0:17:37.415567 4798 EventDB: CEventDataBase::AddRecord: Code 771, severity 3, additional strings: 2
  41. 12/14 0:17:37.417908 4798 func: <<< CEventDataBase::AddRecord
  42. 12/14 0:17:37.417929 4798 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 771).
  43. 12/14 0:17:37.419554 4798 Srv: iScsiServer::sendNotification: Send reaction finish.
  44. 12/14 0:17:37.419584 4798 func: <<< iScsiServer::sendNotification
  45. 12/14 0:17:37.419596 4798 func: <<< HANode::reportEvent
  46. 12/14 0:17:37.419606 4798 func: <<< CHAPartnerNode::SetSyncStatus
  47. 12/14 0:17:37.435544 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (16 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 9114101, expcmdsn = 9114099, maxcmdsn = 9114338.
  48. 12/14 0:17:37.435578 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 9114092, itt = 9114176)
  49. 12/14 0:17:37.435588 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 9114093, itt = 9114177)
  50. 12/14 0:17:37.435596 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 9114094, itt = 9114178)
  51. 12/14 0:17:37.435603 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 9114095, itt = 9114179)
  52. 12/14 0:17:37.435610 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 9114096, itt = 9114180)
  53. 12/14 0:17:37.435619 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 9114097, itt = 9114181)
  54. 12/14 0:17:37.435626 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 9114098, itt = 9114182)
  55. 12/14 0:17:37.435636 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE60D8 has failed due to unidentified error.
  56. 12/14 0:17:37.435643 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  57. 12/14 0:17:37.435649 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F2F0D160 synchronization task has been failed!
  58. 12/14 0:17:37.435667 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6258 has failed due to unidentified error.
  59. 12/14 0:17:37.435673 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  60. 12/14 0:17:37.435679 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F2F0DA00 synchronization task has been failed!
  61. 12/14 0:17:37.435689 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D85E8 has failed due to unidentified error.
  62. 12/14 0:17:37.435696 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  63. 12/14 0:17:37.435700 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F2F0DFC0 synchronization task has been failed!
  64. 12/14 0:17:37.435710 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86991E8 has failed due to unidentified error.
  65. 12/14 0:17:37.435717 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  66. 12/14 0:17:37.435722 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F2F0E860 synchronization task has been failed!
  67. 12/14 0:17:37.435732 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6158 has failed due to unidentified error.
  68. 12/14 0:17:37.435738 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  69. 12/14 0:17:37.435743 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F2F0D720 synchronization task has been failed!
  70. 12/14 0:17:37.436024 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86993E8 has failed due to unidentified error.
  71. 12/14 0:17:37.436033 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  72. 12/14 0:17:37.436038 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F2F0E580 synchronization task has been failed!
  73. 12/14 0:17:37.436050 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86994E8 has failed due to unidentified error.
  74. 12/14 0:17:37.436056 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  75. 12/14 0:17:37.436061 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F2F0EE20 synchronization task has been failed!
  76. 12/14 0:17:37.451557 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (15 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 9114101, expcmdsn = 9114099, maxcmdsn = 9114338.
  77. 12/14 0:17:37.451595 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 9114099, itt = 9114183)
  78. 12/14 0:17:37.451613 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6758 has failed due to unidentified error.
  79. 12/14 0:17:37.451622 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  80. 12/14 0:17:37.451631 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97650 synchronization task has been failed!
  81. 12/14 0:17:42.192220 38d8 func: >>> HANode::onScsiOp_CustomControl_HeartbeatPing
  82. 12/14 0:17:42.192310 38d8 Common: CStarWindStorageDevice::getLastPerfDegEventElapsedTimeMs: perfDegEventElapsedTime = -1
  83. 12/14 0:17:42.192318 38d8 func: <<< HANode::onScsiOp_CustomControl_HeartbeatPing
  84. 12/14 0:17:42.193482 38d8 func: >>> CHAPartnerNode::ShutdownNode
  85. 12/14 0:17:42.193538 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  86. 12/14 0:17:42.193562 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  87. 12/14 0:17:42.193583 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  88. 12/14 0:17:42.193611 38d8 func: <<< CHAPartnerNode::ShutdownNode
  89. 12/14 0:17:42.193701 4798 HA: CHAPartnerNode::AfterSynchronization: Event: Full synchronization has been completed (error: 1237)
  90. 12/14 0:17:42.201484 5378 func: >>> HANode::reportEvent
  91. 12/14 0:17:42.201518 5378 func: >>> iScsiServer::sendNotification
  92. 12/14 0:17:42.201528 5378 func: >>> CEventDataBase::AddRecord
  93. 12/14 0:17:42.201536 5378 EventDB: CEventDataBase::AddRecord: Code 783, severity 1, additional strings: 2
  94. 12/14 0:17:42.201624 5378 func: <<< CEventDataBase::AddRecord
  95. 12/14 0:17:42.201634 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 783).
  96. 12/14 0:17:42.202262 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  97. 12/14 0:17:42.202285 5378 func: <<< iScsiServer::sendNotification
  98. 12/14 0:17:42.202293 5378 func: <<< HANode::reportEvent
  99. 12/14 0:17:42.202300 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_COMPLETE_WITH_ERROR, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'
  100. 12/14 0:17:45.906212 38d8 func: >>> CHAPartnerNode::ShutdownNode
  101. 12/14 0:17:45.906254 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  102. 12/14 0:17:45.906266 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  103. 12/14 0:17:45.906276 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  104. 12/14 0:17:45.906285 38d8 func: <<< CHAPartnerNode::ShutdownNode
  105. 12/14 0:17:45.910338 2d00 HA: HANode::onScsiOp_CustomControl_SynchronizationRequest: The partner iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03 has requested the 1 synchronization type.
  106. 12/14 0:17:45.911454 2d00 HA: CHAPartnerNode::Synchronize: ENTERed with params: p_SyncType = 1, p_ulSnapshotID = 0(0x0), p_pStorageDevice = 0x000002C7ECEDC4B0, p_pSyncBarrier = 0x000002C7ECEDC858, p_puliStartSector = 0x000000D2528FF118 (value is 0x0), p_puliEndSector = 0x000000D2528FF110 (value is 0xAEFFFFF), p_ulIOBlockSizeInSectors = 64, p_ulSectorSizeInUCHARs = 4096, p_bContinue = no, p_bVaaiWriteSameSupport = yes
  107. 12/14 0:17:45.911492 2d00 HA: HANode::updatePartnerStatus: Statuses is already actual, we don't need to update them.
  108. 12/14 0:17:45.911518 2d00 func: >>> CHAPartnerNode::SetSyncStatus
  109. 12/14 0:17:45.911532 2d00 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  110. 12/14 0:17:45.911546 2d00 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 2 from 3. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  111. 12/14 0:17:45.911558 2d00 func: >>> HANode::reportEvent
  112. 12/14 0:17:45.911572 2d00 func: >>> iScsiServer::sendNotification
  113. 12/14 0:17:45.911584 2d00 func: >>> CEventDataBase::AddRecord
  114. 12/14 0:17:45.911595 2d00 EventDB: CEventDataBase::AddRecord: Code 770, severity 2, additional strings: 2
  115. 12/14 0:17:45.911690 2d00 func: <<< CEventDataBase::AddRecord
  116. 12/14 0:17:45.911702 2d00 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 770).
  117. 12/14 0:17:45.912422 2d00 Srv: iScsiServer::sendNotification: Send reaction finish.
  118. 12/14 0:17:45.912452 2d00 func: <<< iScsiServer::sendNotification
  119. 12/14 0:17:45.912464 2d00 func: <<< HANode::reportEvent
  120. 12/14 0:17:45.912476 2d00 func: <<< CHAPartnerNode::SetSyncStatus
  121. 12/14 0:17:45.912736 4b70 func: >>> CHAPartnerNode::SetSyncStatus
  122. 12/14 0:17:45.912758 4b70 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  123. 12/14 0:17:45.912766 4b70 func: <<< CHAPartnerNode::SetSyncStatus
  124. 12/14 0:17:45.916256 5378 func: >>> HANode::reportEvent
  125. 12/14 0:17:45.916288 5378 func: >>> iScsiServer::sendNotification
  126. 12/14 0:17:45.916297 5378 func: >>> CEventDataBase::AddRecord
  127. 12/14 0:17:45.916304 5378 EventDB: CEventDataBase::AddRecord: Code 780, severity 1, additional strings: 3
  128. 12/14 0:17:45.916364 5378 func: <<< CEventDataBase::AddRecord
  129. 12/14 0:17:45.916372 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 780).
  130. 12/14 0:17:45.916862 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  131. 12/14 0:17:45.916880 5378 func: <<< iScsiServer::sendNotification
  132. 12/14 0:17:45.916888 5378 func: <<< HANode::reportEvent
  133. 12/14 0:17:45.916895 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_STARTED, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03', Synchronization type: 'Full'
  134. 12/14 0:40:17.254522 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (9953 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 9573942, expcmdsn = 9573941, maxcmdsn = 9574180.
  135. 12/14 0:40:17.254649 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 9573926, itt = 9574011)
  136. 12/14 0:40:17.254675 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 9573927, itt = 9574012)
  137. 12/14 0:40:17.254699 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 9573928, itt = 9574013)
  138. 12/14 0:40:17.254721 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BE08 has failed due to unidentified error.
  139. 12/14 0:40:17.254737 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  140. 12/14 0:40:17.254757 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC76BE90 synchronization task has been failed!
  141. 12/14 0:40:17.254794 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86991E8 has failed due to unidentified error.
  142. 12/14 0:40:17.254809 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  143. 12/14 0:40:17.254816 4b70 func: >>> CHAPartnerNode::SetSyncStatus
  144. 12/14 0:40:17.254855 4b70 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  145. 12/14 0:40:17.254866 4b70 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 3 from 2. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  146. 12/14 0:40:17.254873 4b70 func: >>> HANode::reportEvent
  147. 12/14 0:40:17.254889 4b70 func: >>> iScsiServer::sendNotification
  148. 12/14 0:40:17.254896 4b70 func: >>> CEventDataBase::AddRecord
  149. 12/14 0:40:17.254820 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F984B0 synchronization task has been failed!
  150. 12/14 0:40:17.254903 4b70 EventDB: CEventDataBase::AddRecord: Code 771, severity 3, additional strings: 2
  151. 12/14 0:40:17.254946 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6758 has failed due to unidentified error.
  152. 12/14 0:40:17.254968 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  153. 12/14 0:40:17.254983 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC76AD50 synchronization task has been failed!
  154. 12/14 0:40:17.255244 4b70 func: <<< CEventDataBase::AddRecord
  155. 12/14 0:40:17.255256 4b70 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 771).
  156. 12/14 0:40:17.256452 4b70 Srv: iScsiServer::sendNotification: Send reaction finish.
  157. 12/14 0:40:17.256507 4b70 func: <<< iScsiServer::sendNotification
  158. 12/14 0:40:17.256526 4b70 func: <<< HANode::reportEvent
  159. 12/14 0:40:17.256533 4b70 func: <<< CHAPartnerNode::SetSyncStatus
  160. 12/14 0:40:17.273426 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (15 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 9573943, expcmdsn = 9573941, maxcmdsn = 9574180.
  161. 12/14 0:40:17.273463 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 9573929, itt = 9574014)
  162. 12/14 0:40:17.273475 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 9573930, itt = 9574015)
  163. 12/14 0:40:17.273485 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 9573931, itt = 9574016)
  164. 12/14 0:40:17.273495 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 9573932, itt = 9574017)
  165. 12/14 0:40:17.273506 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 9573933, itt = 9574018)
  166. 12/14 0:40:17.273521 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6258 has failed due to unidentified error.
  167. 12/14 0:40:17.273529 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  168. 12/14 0:40:17.273535 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC76BBB0 synchronization task has been failed!
  169. 12/14 0:40:17.273553 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EC4F39D8 has failed due to unidentified error.
  170. 12/14 0:40:17.273560 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  171. 12/14 0:40:17.273565 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC76B8D0 synchronization task has been failed!
  172. 12/14 0:40:17.273576 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7D68 has failed due to unidentified error.
  173. 12/14 0:40:17.273593 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  174. 12/14 0:40:17.273597 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97930 synchronization task has been failed!
  175. 12/14 0:40:17.273608 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E8699A68 has failed due to unidentified error.
  176. 12/14 0:40:17.273615 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  177. 12/14 0:40:17.273620 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC76C170 synchronization task has been failed!
  178. 12/14 0:40:17.273692 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8568 has failed due to unidentified error.
  179. 12/14 0:40:17.273700 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  180. 12/14 0:40:17.273705 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96510 synchronization task has been failed!
  181. 12/14 0:40:17.290513 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (16 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 9573943, expcmdsn = 9573941, maxcmdsn = 9574180.
  182. 12/14 0:40:17.290558 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 9573934, itt = 9574019)
  183. 12/14 0:40:17.290569 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 9573935, itt = 9574020)
  184. 12/14 0:40:17.290580 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 9573936, itt = 9574021)
  185. 12/14 0:40:17.290589 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 9573937, itt = 9574022)
  186. 12/14 0:40:17.290598 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 9573938, itt = 9574023)
  187. 12/14 0:40:17.290609 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 9573939, itt = 9574024)
  188. 12/14 0:40:17.290621 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BF88 has failed due to unidentified error.
  189. 12/14 0:40:17.290631 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  190. 12/14 0:40:17.290640 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F950F0 synchronization task has been failed!
  191. 12/14 0:40:17.290664 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86994E8 has failed due to unidentified error.
  192. 12/14 0:40:17.290674 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  193. 12/14 0:40:17.290681 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC76B5F0 synchronization task has been failed!
  194. 12/14 0:40:17.290694 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D85E8 has failed due to unidentified error.
  195. 12/14 0:40:17.290702 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  196. 12/14 0:40:17.290708 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F967F0 synchronization task has been failed!
  197. 12/14 0:40:17.290720 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6158 has failed due to unidentified error.
  198. 12/14 0:40:17.290729 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  199. 12/14 0:40:17.290743 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC76C450 synchronization task has been failed!
  200. 12/14 0:40:17.290762 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE60D8 has failed due to unidentified error.
  201. 12/14 0:40:17.290769 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  202. 12/14 0:40:17.290774 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC76C730 synchronization task has been failed!
  203. 12/14 0:40:17.290784 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86993E8 has failed due to unidentified error.
  204. 12/14 0:40:17.290790 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  205. 12/14 0:40:17.290795 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC76B030 synchronization task has been failed!
  206. 12/14 0:40:17.307415 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (16 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 9573943, expcmdsn = 9573941, maxcmdsn = 9574180.
  207. 12/14 0:40:17.307454 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 9573940, itt = 9574025)
  208. 12/14 0:40:17.307494 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 9573941, itt = 9574026)
  209. 12/14 0:40:17.307505 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8268 has failed due to unidentified error.
  210. 12/14 0:40:17.307512 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  211. 12/14 0:40:17.307519 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC76B310 synchronization task has been failed!
  212. 12/14 0:40:17.307536 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7BE8 has failed due to unidentified error.
  213. 12/14 0:40:17.307543 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  214. 12/14 0:40:17.307570 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC76CA10 synchronization task has been failed!
  215. 12/14 0:40:22.066538 38d8 func: >>> HANode::onScsiOp_CustomControl_HeartbeatPing
  216. 12/14 0:40:22.066724 38d8 Common: CStarWindStorageDevice::getLastPerfDegEventElapsedTimeMs: perfDegEventElapsedTime = -1
  217. 12/14 0:40:22.066747 38d8 func: <<< HANode::onScsiOp_CustomControl_HeartbeatPing
  218. 12/14 0:40:22.068099 4b70 HA: CHAPartnerNode::AfterSynchronization: Event: Full synchronization has been completed (error: 1237)
  219. 12/14 0:40:22.070568 5378 func: >>> HANode::reportEvent
  220. 12/14 0:40:22.070603 5378 func: >>> iScsiServer::sendNotification
  221. 12/14 0:40:22.070615 5378 func: >>> CEventDataBase::AddRecord
  222. 12/14 0:40:22.070624 5378 EventDB: CEventDataBase::AddRecord: Code 783, severity 1, additional strings: 2
  223. 12/14 0:40:22.070687 5378 func: <<< CEventDataBase::AddRecord
  224. 12/14 0:40:22.070698 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 783).
  225. 12/14 0:40:22.071528 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  226. 12/14 0:40:22.071683 5378 func: <<< iScsiServer::sendNotification
  227. 12/14 0:40:22.071736 5378 func: <<< HANode::reportEvent
  228. 12/14 0:40:22.071747 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_COMPLETE_WITH_ERROR, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'
  229. 12/14 0:40:23.076064 38d8 func: >>> CHAPartnerNode::ShutdownNode
  230. 12/14 0:40:23.076101 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  231. 12/14 0:40:23.076109 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  232. 12/14 0:40:23.076123 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  233. 12/14 0:40:23.076129 38d8 func: <<< CHAPartnerNode::ShutdownNode
  234. 12/14 0:40:23.380531 38d8 func: >>> CHAPartnerNode::ShutdownNode
  235. 12/14 0:40:23.380564 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  236. 12/14 0:40:23.380573 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  237. 12/14 0:40:23.380580 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  238. 12/14 0:40:23.380586 38d8 func: <<< CHAPartnerNode::ShutdownNode
  239. 12/14 0:40:23.383566 2d00 HA: HANode::onScsiOp_CustomControl_SynchronizationRequest: The partner iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03 has requested the 1 synchronization type.
  240. 12/14 0:40:23.384621 2d00 HA: CHAPartnerNode::Synchronize: ENTERed with params: p_SyncType = 1, p_ulSnapshotID = 0(0x0), p_pStorageDevice = 0x000002C7ECEDC4B0, p_pSyncBarrier = 0x000002C7ECEDC858, p_puliStartSector = 0x000000D2528FF118 (value is 0x0), p_puliEndSector = 0x000000D2528FF110 (value is 0xAEFFFFF), p_ulIOBlockSizeInSectors = 64, p_ulSectorSizeInUCHARs = 4096, p_bContinue = no, p_bVaaiWriteSameSupport = yes
  241. 12/14 0:40:23.384652 2d00 HA: HANode::updatePartnerStatus: Statuses is already actual, we don't need to update them.
  242. 12/14 0:40:23.384674 2d00 func: >>> CHAPartnerNode::SetSyncStatus
  243. 12/14 0:40:23.384684 2d00 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  244. 12/14 0:40:23.384693 2d00 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 2 from 3. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  245. 12/14 0:40:23.384703 2d00 func: >>> HANode::reportEvent
  246. 12/14 0:40:23.384713 2d00 func: >>> iScsiServer::sendNotification
  247. 12/14 0:40:23.384722 2d00 func: >>> CEventDataBase::AddRecord
  248. 12/14 0:40:23.384730 2d00 EventDB: CEventDataBase::AddRecord: Code 770, severity 2, additional strings: 2
  249. 12/14 0:40:23.384795 2d00 func: <<< CEventDataBase::AddRecord
  250. 12/14 0:40:23.384804 2d00 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 770).
  251. 12/14 0:40:23.385228 2d00 Srv: iScsiServer::sendNotification: Send reaction finish.
  252. 12/14 0:40:23.385249 2d00 func: <<< iScsiServer::sendNotification
  253. 12/14 0:40:23.385258 2d00 func: <<< HANode::reportEvent
  254. 12/14 0:40:23.385267 2d00 func: <<< CHAPartnerNode::SetSyncStatus
  255. 12/14 0:40:23.385478 6934 func: >>> CHAPartnerNode::SetSyncStatus
  256. 12/14 0:40:23.385501 6934 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  257. 12/14 0:40:23.385509 6934 func: <<< CHAPartnerNode::SetSyncStatus
  258. 12/14 0:40:23.387649 5378 func: >>> HANode::reportEvent
  259. 12/14 0:40:23.387678 5378 func: >>> iScsiServer::sendNotification
  260. 12/14 0:40:23.387686 5378 func: >>> CEventDataBase::AddRecord
  261. 12/14 0:40:23.387692 5378 EventDB: CEventDataBase::AddRecord: Code 780, severity 1, additional strings: 3
  262. 12/14 0:40:23.387741 5378 func: <<< CEventDataBase::AddRecord
  263. 12/14 0:40:23.387748 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 780).
  264. 12/14 0:40:23.388148 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  265. 12/14 0:40:23.388168 5378 func: <<< iScsiServer::sendNotification
  266. 12/14 0:40:23.388177 5378 func: <<< HANode::reportEvent
  267. 12/14 0:40:23.388185 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_STARTED, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03', Synchronization type: 'Full'
  268. 12/14 1:02:59.399339 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (9953 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 10035215, expcmdsn = 10035214, maxcmdsn = 10035453.
  269. 12/14 1:02:59.399392 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10035199, itt = 10035284)
  270. 12/14 1:02:59.399411 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10035200, itt = 10035285)
  271. 12/14 1:02:59.399423 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE60D8 has failed due to unidentified error.
  272. 12/14 1:02:59.399431 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  273. 12/14 1:02:59.399438 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F95990 synchronization task has been failed!
  274. 12/14 1:02:59.399457 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86994E8 has failed due to unidentified error.
  275. 12/14 1:02:59.399464 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  276. 12/14 1:02:59.399469 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC76C450 synchronization task has been failed!
  277. 12/14 1:02:59.399493 6934 func: >>> CHAPartnerNode::SetSyncStatus
  278. 12/14 1:02:59.399537 6934 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  279. 12/14 1:02:59.399550 6934 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 3 from 2. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  280. 12/14 1:02:59.399563 6934 func: >>> HANode::reportEvent
  281. 12/14 1:02:59.399586 6934 func: >>> iScsiServer::sendNotification
  282. 12/14 1:02:59.399598 6934 func: >>> CEventDataBase::AddRecord
  283. 12/14 1:02:59.399609 6934 EventDB: CEventDataBase::AddRecord: Code 771, severity 3, additional strings: 2
  284. 12/14 1:02:59.399709 6934 func: <<< CEventDataBase::AddRecord
  285. 12/14 1:02:59.399721 6934 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 771).
  286. 12/14 1:02:59.400524 6934 Srv: iScsiServer::sendNotification: Send reaction finish.
  287. 12/14 1:02:59.400555 6934 func: <<< iScsiServer::sendNotification
  288. 12/14 1:02:59.400568 6934 func: <<< HANode::reportEvent
  289. 12/14 1:02:59.400579 6934 func: <<< CHAPartnerNode::SetSyncStatus
  290. 12/14 1:02:59.402818 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10035201, itt = 10035286)
  291. 12/14 1:02:59.402847 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10035202, itt = 10035287)
  292. 12/14 1:02:59.402856 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10035203, itt = 10035288)
  293. 12/14 1:02:59.402865 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6258 has failed due to unidentified error.
  294. 12/14 1:02:59.402873 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  295. 12/14 1:02:59.402879 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F953D0 synchronization task has been failed!
  296. 12/14 1:02:59.402893 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EC4F39D8 has failed due to unidentified error.
  297. 12/14 1:02:59.402899 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  298. 12/14 1:02:59.402904 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97650 synchronization task has been failed!
  299. 12/14 1:02:59.402916 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BF88 has failed due to unidentified error.
  300. 12/14 1:02:59.402923 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  301. 12/14 1:02:59.402927 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC76B5F0 synchronization task has been failed!
  302. 12/14 1:02:59.419331 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (16 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 10035216, expcmdsn = 10035214, maxcmdsn = 10035453.
  303. 12/14 1:02:59.419382 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10035204, itt = 10035289)
  304. 12/14 1:02:59.419393 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10035205, itt = 10035290)
  305. 12/14 1:02:59.419401 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10035206, itt = 10035291)
  306. 12/14 1:02:59.419408 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10035207, itt = 10035292)
  307. 12/14 1:02:59.419421 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6758 has failed due to unidentified error.
  308. 12/14 1:02:59.419429 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  309. 12/14 1:02:59.419435 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96510 synchronization task has been failed!
  310. 12/14 1:02:59.419452 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E8699A68 has failed due to unidentified error.
  311. 12/14 1:02:59.419459 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  312. 12/14 1:02:59.419464 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC76AD50 synchronization task has been failed!
  313. 12/14 1:02:59.419476 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7D68 has failed due to unidentified error.
  314. 12/14 1:02:59.419504 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  315. 12/14 1:02:59.419510 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC76C170 synchronization task has been failed!
  316. 12/14 1:02:59.419522 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6158 has failed due to unidentified error.
  317. 12/14 1:02:59.419528 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  318. 12/14 1:02:59.419533 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC76BE90 synchronization task has been failed!
  319. 12/14 1:02:59.435329 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (15 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 10035216, expcmdsn = 10035214, maxcmdsn = 10035453.
  320. 12/14 1:02:59.435370 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10035208, itt = 10035293)
  321. 12/14 1:02:59.435380 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10035209, itt = 10035294)
  322. 12/14 1:02:59.435389 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10035210, itt = 10035295)
  323. 12/14 1:02:59.435398 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10035211, itt = 10035296)
  324. 12/14 1:02:59.435409 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10035212, itt = 10035297)
  325. 12/14 1:02:59.435419 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10035213, itt = 10035298)
  326. 12/14 1:02:59.435425 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10035214, itt = 10035299)
  327. 12/14 1:02:59.435435 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86991E8 has failed due to unidentified error.
  328. 12/14 1:02:59.435443 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  329. 12/14 1:02:59.435455 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC76B030 synchronization task has been failed!
  330. 12/14 1:02:59.435475 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7BE8 has failed due to unidentified error.
  331. 12/14 1:02:59.435482 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  332. 12/14 1:02:59.435487 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F950F0 synchronization task has been failed!
  333. 12/14 1:02:59.435497 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86999E8 has failed due to unidentified error.
  334. 12/14 1:02:59.435504 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  335. 12/14 1:02:59.435509 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC76B310 synchronization task has been failed!
  336. 12/14 1:02:59.435519 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8568 has failed due to unidentified error.
  337. 12/14 1:02:59.435526 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  338. 12/14 1:02:59.435531 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC76BBB0 synchronization task has been failed!
  339. 12/14 1:02:59.435540 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BE08 has failed due to unidentified error.
  340. 12/14 1:02:59.435548 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  341. 12/14 1:02:59.435555 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC76C730 synchronization task has been failed!
  342. 12/14 1:02:59.435565 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D85E8 has failed due to unidentified error.
  343. 12/14 1:02:59.435572 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  344. 12/14 1:02:59.435577 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC76CA10 synchronization task has been failed!
  345. 12/14 1:02:59.435586 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8268 has failed due to unidentified error.
  346. 12/14 1:02:59.435593 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  347. 12/14 1:02:59.435597 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC76B8D0 synchronization task has been failed!
  348. 12/14 1:03:04.033079 38d8 func: >>> HANode::onScsiOp_CustomControl_HeartbeatPing
  349. 12/14 1:03:04.033211 38d8 Common: CStarWindStorageDevice::getLastPerfDegEventElapsedTimeMs: perfDegEventElapsedTime = -1
  350. 12/14 1:03:04.033230 38d8 func: <<< HANode::onScsiOp_CustomControl_HeartbeatPing
  351. 12/14 1:03:04.034156 38d8 func: >>> CHAPartnerNode::ShutdownNode
  352. 12/14 1:03:04.034179 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  353. 12/14 1:03:04.034189 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  354. 12/14 1:03:04.034201 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  355. 12/14 1:03:04.034211 38d8 func: <<< CHAPartnerNode::ShutdownNode
  356. 12/14 1:03:04.034782 6934 HA: CHAPartnerNode::AfterSynchronization: Event: Full synchronization has been completed (error: 1237)
  357. 12/14 1:03:04.046354 5378 func: >>> HANode::reportEvent
  358. 12/14 1:03:04.046389 5378 func: >>> iScsiServer::sendNotification
  359. 12/14 1:03:04.046398 5378 func: >>> CEventDataBase::AddRecord
  360. 12/14 1:03:04.046413 5378 EventDB: CEventDataBase::AddRecord: Code 783, severity 1, additional strings: 2
  361. 12/14 1:03:04.046532 5378 func: <<< CEventDataBase::AddRecord
  362. 12/14 1:03:04.046542 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 783).
  363. 12/14 1:03:04.047110 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  364. 12/14 1:03:04.047136 5378 func: <<< iScsiServer::sendNotification
  365. 12/14 1:03:04.047182 5378 func: <<< HANode::reportEvent
  366. 12/14 1:03:04.047191 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_COMPLETE_WITH_ERROR, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'
  367. 12/14 1:03:05.549635 38d8 func: >>> CHAPartnerNode::ShutdownNode
  368. 12/14 1:03:05.549708 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  369. 12/14 1:03:05.549735 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  370. 12/14 1:03:05.549756 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  371. 12/14 1:03:05.549775 38d8 func: <<< CHAPartnerNode::ShutdownNode
  372. 12/14 1:03:05.555505 2d00 HA: HANode::onScsiOp_CustomControl_SynchronizationRequest: The partner iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03 has requested the 1 synchronization type.
  373. 12/14 1:03:05.556615 2d00 HA: CHAPartnerNode::Synchronize: ENTERed with params: p_SyncType = 1, p_ulSnapshotID = 0(0x0), p_pStorageDevice = 0x000002C7ECEDC4B0, p_pSyncBarrier = 0x000002C7ECEDC858, p_puliStartSector = 0x000000D2528FF118 (value is 0x0), p_puliEndSector = 0x000000D2528FF110 (value is 0xAEFFFFF), p_ulIOBlockSizeInSectors = 64, p_ulSectorSizeInUCHARs = 4096, p_bContinue = no, p_bVaaiWriteSameSupport = yes
  374. 12/14 1:03:05.556663 2d00 HA: HANode::updatePartnerStatus: Statuses is already actual, we don't need to update them.
  375. 12/14 1:03:05.556696 2d00 func: >>> CHAPartnerNode::SetSyncStatus
  376. 12/14 1:03:05.556716 2d00 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  377. 12/14 1:03:05.556736 2d00 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 2 from 3. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  378. 12/14 1:03:05.556756 2d00 func: >>> HANode::reportEvent
  379. 12/14 1:03:05.556776 2d00 func: >>> iScsiServer::sendNotification
  380. 12/14 1:03:05.556796 2d00 func: >>> CEventDataBase::AddRecord
  381. 12/14 1:03:05.556818 2d00 EventDB: CEventDataBase::AddRecord: Code 770, severity 2, additional strings: 2
  382. 12/14 1:03:05.556939 2d00 func: <<< CEventDataBase::AddRecord
  383. 12/14 1:03:05.556959 2d00 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 770).
  384. 12/14 1:03:05.557739 2d00 Srv: iScsiServer::sendNotification: Send reaction finish.
  385. 12/14 1:03:05.557779 2d00 func: <<< iScsiServer::sendNotification
  386. 12/14 1:03:05.557800 2d00 func: <<< HANode::reportEvent
  387. 12/14 1:03:05.557819 2d00 func: <<< CHAPartnerNode::SetSyncStatus
  388. 12/14 1:03:05.558107 5f64 func: >>> CHAPartnerNode::SetSyncStatus
  389. 12/14 1:03:05.558141 5f64 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  390. 12/14 1:03:05.558152 5f64 func: <<< CHAPartnerNode::SetSyncStatus
  391. 12/14 1:03:05.560699 5378 func: >>> HANode::reportEvent
  392. 12/14 1:03:05.560733 5378 func: >>> iScsiServer::sendNotification
  393. 12/14 1:03:05.560742 5378 func: >>> CEventDataBase::AddRecord
  394. 12/14 1:03:05.560748 5378 EventDB: CEventDataBase::AddRecord: Code 780, severity 1, additional strings: 3
  395. 12/14 1:03:05.560803 5378 func: <<< CEventDataBase::AddRecord
  396. 12/14 1:03:05.560810 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 780).
  397. 12/14 1:03:05.561289 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  398. 12/14 1:03:05.561311 5378 func: <<< iScsiServer::sendNotification
  399. 12/14 1:03:05.561319 5378 func: <<< HANode::reportEvent
  400. 12/14 1:03:05.561327 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_STARTED, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03', Synchronization type: 'Full'
  401. 12/14 1:25:39.538718 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (9954 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 10495894, expcmdsn = 10495893, maxcmdsn = 10496132.
  402. 12/14 1:25:39.538839 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10495878, itt = 10495963)
  403. 12/14 1:25:39.538854 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10495879, itt = 10495964)
  404. 12/14 1:25:39.538866 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10495880, itt = 10495965)
  405. 12/14 1:25:39.538877 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10495881, itt = 10495966)
  406. 12/14 1:25:39.538887 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10495882, itt = 10495967)
  407. 12/14 1:25:39.538900 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10495883, itt = 10495968)
  408. 12/14 1:25:39.538910 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10495884, itt = 10495969)
  409. 12/14 1:25:39.538924 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8568 has failed due to unidentified error.
  410. 12/14 1:25:39.538935 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  411. 12/14 1:25:39.538944 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F956B0 synchronization task has been failed!
  412. 12/14 1:25:39.538969 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7BE8 has failed due to unidentified error.
  413. 12/14 1:25:39.538980 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  414. 12/14 1:25:39.538987 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC5371D0 synchronization task has been failed!
  415. 12/14 1:25:39.538995 5f64 func: >>> CHAPartnerNode::SetSyncStatus
  416. 12/14 1:25:39.539032 5f64 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  417. 12/14 1:25:39.538999 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6158 has failed due to unidentified error.
  418. 12/14 1:25:39.539073 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  419. 12/14 1:25:39.539039 5f64 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 3 from 2. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  420. 12/14 1:25:39.539085 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F95C70 synchronization task has been failed!
  421. 12/14 1:25:39.539138 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86999E8 has failed due to unidentified error.
  422. 12/14 1:25:39.539111 5f64 func: >>> HANode::reportEvent
  423. 12/14 1:25:39.539148 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  424. 12/14 1:25:39.539177 5f64 func: >>> iScsiServer::sendNotification
  425. 12/14 1:25:39.539193 5f64 func: >>> CEventDataBase::AddRecord
  426. 12/14 1:25:39.539204 5f64 EventDB: CEventDataBase::AddRecord: Code 771, severity 3, additional strings: 2
  427. 12/14 1:25:39.539179 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC537790 synchronization task has been failed!
  428. 12/14 1:25:39.539236 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7D68 has failed due to unidentified error.
  429. 12/14 1:25:39.539256 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  430. 12/14 1:25:39.539263 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC538BB0 synchronization task has been failed!
  431. 12/14 1:25:39.539274 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6258 has failed due to unidentified error.
  432. 12/14 1:25:39.539286 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  433. 12/14 1:25:39.539292 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F950F0 synchronization task has been failed!
  434. 12/14 1:25:39.539303 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE60D8 has failed due to unidentified error.
  435. 12/14 1:25:39.539312 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  436. 12/14 1:25:39.539318 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC538310 synchronization task has been failed!
  437. 12/14 1:25:39.539335 5f64 func: <<< CEventDataBase::AddRecord
  438. 12/14 1:25:39.539347 5f64 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 771).
  439. 12/14 1:25:39.539989 5f64 Srv: iScsiServer::sendNotification: Send reaction finish.
  440. 12/14 1:25:39.540012 5f64 func: <<< iScsiServer::sendNotification
  441. 12/14 1:25:39.540022 5f64 func: <<< HANode::reportEvent
  442. 12/14 1:25:39.540030 5f64 func: <<< CHAPartnerNode::SetSyncStatus
  443. 12/14 1:25:39.557365 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (15 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 10495895, expcmdsn = 10495893, maxcmdsn = 10496132.
  444. 12/14 1:25:39.557397 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10495885, itt = 10495970)
  445. 12/14 1:25:39.557407 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10495886, itt = 10495971)
  446. 12/14 1:25:39.557415 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10495887, itt = 10495972)
  447. 12/14 1:25:39.557422 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10495888, itt = 10495973)
  448. 12/14 1:25:39.557430 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10495889, itt = 10495974)
  449. 12/14 1:25:39.557439 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10495890, itt = 10495975)
  450. 12/14 1:25:39.557449 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BF88 has failed due to unidentified error.
  451. 12/14 1:25:39.557457 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  452. 12/14 1:25:39.557463 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F95F50 synchronization task has been failed!
  453. 12/14 1:25:39.557480 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E8699A68 has failed due to unidentified error.
  454. 12/14 1:25:39.557487 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  455. 12/14 1:25:39.557493 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC538030 synchronization task has been failed!
  456. 12/14 1:25:39.557504 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86991E8 has failed due to unidentified error.
  457. 12/14 1:25:39.557511 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  458. 12/14 1:25:39.557516 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC537D50 synchronization task has been failed!
  459. 12/14 1:25:39.557535 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6758 has failed due to unidentified error.
  460. 12/14 1:25:39.557542 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  461. 12/14 1:25:39.557547 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC5385F0 synchronization task has been failed!
  462. 12/14 1:25:39.557559 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8268 has failed due to unidentified error.
  463. 12/14 1:25:39.557566 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  464. 12/14 1:25:39.557571 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F98790 synchronization task has been failed!
  465. 12/14 1:25:39.557581 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86993E8 has failed due to unidentified error.
  466. 12/14 1:25:39.557588 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  467. 12/14 1:25:39.557595 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC5388D0 synchronization task has been failed!
  468. 12/14 1:25:39.574233 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (16 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 10495895, expcmdsn = 10495893, maxcmdsn = 10496132.
  469. 12/14 1:25:39.574273 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10495891, itt = 10495976)
  470. 12/14 1:25:39.574286 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10495892, itt = 10495977)
  471. 12/14 1:25:39.574298 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EC4F39D8 has failed due to unidentified error.
  472. 12/14 1:25:39.574310 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  473. 12/14 1:25:39.574319 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC538E90 synchronization task has been failed!
  474. 12/14 1:25:39.574340 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BE08 has failed due to unidentified error.
  475. 12/14 1:25:39.574349 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  476. 12/14 1:25:39.574357 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC5374B0 synchronization task has been failed!
  477. 12/14 1:25:39.589089 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (15 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 10495895, expcmdsn = 10495893, maxcmdsn = 10496132.
  478. 12/14 1:25:39.589127 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10495893, itt = 10495978)
  479. 12/14 1:25:39.589142 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D85E8 has failed due to unidentified error.
  480. 12/14 1:25:39.589152 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  481. 12/14 1:25:39.589159 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC537A70 synchronization task has been failed!
  482. 12/14 1:25:44.329942 5f64 HA: CHAPartnerNode::AfterSynchronization: Event: Full synchronization has been completed (error: 1237)
  483. 12/14 1:25:44.332846 5378 func: >>> HANode::reportEvent
  484. 12/14 1:25:44.332880 5378 func: >>> iScsiServer::sendNotification
  485. 12/14 1:25:44.332889 5378 func: >>> CEventDataBase::AddRecord
  486. 12/14 1:25:44.332906 5378 EventDB: CEventDataBase::AddRecord: Code 783, severity 1, additional strings: 2
  487. 12/14 1:25:44.332960 5378 func: <<< CEventDataBase::AddRecord
  488. 12/14 1:25:44.332968 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 783).
  489. 12/14 1:25:44.333189 38d8 func: >>> HANode::onScsiOp_CustomControl_HeartbeatPing
  490. 12/14 1:25:44.333307 38d8 Common: CStarWindStorageDevice::getLastPerfDegEventElapsedTimeMs: perfDegEventElapsedTime = -1
  491. 12/14 1:25:44.333319 38d8 func: <<< HANode::onScsiOp_CustomControl_HeartbeatPing
  492. 12/14 1:25:44.333519 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  493. 12/14 1:25:44.333544 5378 func: <<< iScsiServer::sendNotification
  494. 12/14 1:25:44.333552 5378 func: <<< HANode::reportEvent
  495. 12/14 1:25:44.333559 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_COMPLETE_WITH_ERROR, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'
  496. 12/14 1:25:44.334235 38d8 func: >>> CHAPartnerNode::ShutdownNode
  497. 12/14 1:25:44.334288 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  498. 12/14 1:25:44.334313 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  499. 12/14 1:25:44.334334 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  500. 12/14 1:25:44.334352 38d8 func: <<< CHAPartnerNode::ShutdownNode
  501. 12/14 1:25:47.708659 38d8 func: >>> CHAPartnerNode::ShutdownNode
  502. 12/14 1:25:47.708735 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  503. 12/14 1:25:47.708757 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  504. 12/14 1:25:47.708778 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  505. 12/14 1:25:47.708796 38d8 func: <<< CHAPartnerNode::ShutdownNode
  506. 12/14 1:25:47.713652 2d00 HA: HANode::onScsiOp_CustomControl_SynchronizationRequest: The partner iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03 has requested the 1 synchronization type.
  507. 12/14 1:25:47.715305 2d00 HA: CHAPartnerNode::Synchronize: ENTERed with params: p_SyncType = 1, p_ulSnapshotID = 0(0x0), p_pStorageDevice = 0x000002C7ECEDC4B0, p_pSyncBarrier = 0x000002C7ECEDC858, p_puliStartSector = 0x000000D2528FF118 (value is 0x0), p_puliEndSector = 0x000000D2528FF110 (value is 0xAEFFFFF), p_ulIOBlockSizeInSectors = 64, p_ulSectorSizeInUCHARs = 4096, p_bContinue = no, p_bVaaiWriteSameSupport = yes
  508. 12/14 1:25:47.715362 2d00 HA: HANode::updatePartnerStatus: Statuses is already actual, we don't need to update them.
  509. 12/14 1:25:47.715395 2d00 func: >>> CHAPartnerNode::SetSyncStatus
  510. 12/14 1:25:47.715417 2d00 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  511. 12/14 1:25:47.715438 2d00 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 2 from 3. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  512. 12/14 1:25:47.715460 2d00 func: >>> HANode::reportEvent
  513. 12/14 1:25:47.715483 2d00 func: >>> iScsiServer::sendNotification
  514. 12/14 1:25:47.715504 2d00 func: >>> CEventDataBase::AddRecord
  515. 12/14 1:25:47.715524 2d00 EventDB: CEventDataBase::AddRecord: Code 770, severity 2, additional strings: 2
  516. 12/14 1:25:47.715650 2d00 func: <<< CEventDataBase::AddRecord
  517. 12/14 1:25:47.715672 2d00 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 770).
  518. 12/14 1:25:47.716505 2d00 Srv: iScsiServer::sendNotification: Send reaction finish.
  519. 12/14 1:25:47.716546 2d00 func: <<< iScsiServer::sendNotification
  520. 12/14 1:25:47.716567 2d00 func: <<< HANode::reportEvent
  521. 12/14 1:25:47.716586 2d00 func: <<< CHAPartnerNode::SetSyncStatus
  522. 12/14 1:25:47.716881 65c8 func: >>> CHAPartnerNode::SetSyncStatus
  523. 12/14 1:25:47.716906 65c8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  524. 12/14 1:25:47.716913 65c8 func: <<< CHAPartnerNode::SetSyncStatus
  525. 12/14 1:25:47.719411 5378 func: >>> HANode::reportEvent
  526. 12/14 1:25:47.719442 5378 func: >>> iScsiServer::sendNotification
  527. 12/14 1:25:47.719451 5378 func: >>> CEventDataBase::AddRecord
  528. 12/14 1:25:47.719457 5378 EventDB: CEventDataBase::AddRecord: Code 780, severity 1, additional strings: 3
  529. 12/14 1:25:47.719505 5378 func: <<< CEventDataBase::AddRecord
  530. 12/14 1:25:47.719512 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 780).
  531. 12/14 1:25:47.720032 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  532. 12/14 1:25:47.720056 5378 func: <<< iScsiServer::sendNotification
  533. 12/14 1:25:47.720064 5378 func: <<< HANode::reportEvent
  534. 12/14 1:25:47.720070 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_STARTED, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03', Synchronization type: 'Full'
  535. 12/14 1:48:21.954032 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (9953 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 10956788, expcmdsn = 10956787, maxcmdsn = 10957026.
  536. 12/14 1:48:21.954073 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10956772, itt = 10956858)
  537. 12/14 1:48:21.954084 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10956773, itt = 10956859)
  538. 12/14 1:48:21.954093 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10956774, itt = 10956860)
  539. 12/14 1:48:21.954099 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10956775, itt = 10956861)
  540. 12/14 1:48:21.954107 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10956776, itt = 10956862)
  541. 12/14 1:48:21.954120 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BF88 has failed due to unidentified error.
  542. 12/14 1:48:21.954128 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  543. 12/14 1:48:21.954134 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F0799CE0 synchronization task has been failed!
  544. 12/14 1:48:21.954152 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86999E8 has failed due to unidentified error.
  545. 12/14 1:48:21.954159 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  546. 12/14 1:48:21.954163 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F0798300 synchronization task has been failed!
  547. 12/14 1:48:21.954172 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7D68 has failed due to unidentified error.
  548. 12/14 1:48:21.954178 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  549. 12/14 1:48:21.954183 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F3D3CEE0 synchronization task has been failed!
  550. 12/14 1:48:21.954181 65c8 func: >>> CHAPartnerNode::SetSyncStatus
  551. 12/14 1:48:21.954219 65c8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  552. 12/14 1:48:21.954227 65c8 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 3 from 2. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  553. 12/14 1:48:21.954192 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E8699A68 has failed due to unidentified error.
  554. 12/14 1:48:21.954266 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  555. 12/14 1:48:21.954234 65c8 func: >>> HANode::reportEvent
  556. 12/14 1:48:21.954280 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F07988C0 synchronization task has been failed!
  557. 12/14 1:48:21.954346 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7BE8 has failed due to unidentified error.
  558. 12/14 1:48:21.954325 65c8 func: >>> iScsiServer::sendNotification
  559. 12/14 1:48:21.954357 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  560. 12/14 1:48:21.954387 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F3D3B7E0 synchronization task has been failed!
  561. 12/14 1:48:21.954371 65c8 func: >>> CEventDataBase::AddRecord
  562. 12/14 1:48:21.954408 65c8 EventDB: CEventDataBase::AddRecord: Code 771, severity 3, additional strings: 2
  563. 12/14 1:48:21.954482 65c8 func: <<< CEventDataBase::AddRecord
  564. 12/14 1:48:21.954489 65c8 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 771).
  565. 12/14 1:48:21.955023 65c8 Srv: iScsiServer::sendNotification: Send reaction finish.
  566. 12/14 1:48:21.955042 65c8 func: <<< iScsiServer::sendNotification
  567. 12/14 1:48:21.955049 65c8 func: <<< HANode::reportEvent
  568. 12/14 1:48:21.955055 65c8 func: <<< CHAPartnerNode::SetSyncStatus
  569. 12/14 1:48:21.973432 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (16 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 10956789, expcmdsn = 10956787, maxcmdsn = 10957026.
  570. 12/14 1:48:21.973472 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10956777, itt = 10956863)
  571. 12/14 1:48:21.973482 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10956778, itt = 10956864)
  572. 12/14 1:48:21.973490 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10956779, itt = 10956865)
  573. 12/14 1:48:21.973497 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10956780, itt = 10956866)
  574. 12/14 1:48:21.973504 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10956781, itt = 10956867)
  575. 12/14 1:48:21.973511 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10956782, itt = 10956868)
  576. 12/14 1:48:21.973521 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86993E8 has failed due to unidentified error.
  577. 12/14 1:48:21.973528 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  578. 12/14 1:48:21.973535 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F0799160 synchronization task has been failed!
  579. 12/14 1:48:21.973554 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D85E8 has failed due to unidentified error.
  580. 12/14 1:48:21.973560 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  581. 12/14 1:48:21.973565 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F07985E0 synchronization task has been failed!
  582. 12/14 1:48:21.973576 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6258 has failed due to unidentified error.
  583. 12/14 1:48:21.973582 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  584. 12/14 1:48:21.973587 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F0799440 synchronization task has been failed!
  585. 12/14 1:48:21.973597 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE60D8 has failed due to unidentified error.
  586. 12/14 1:48:21.973603 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  587. 12/14 1:48:21.973608 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F3D3D1C0 synchronization task has been failed!
  588. 12/14 1:48:21.973628 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BE08 has failed due to unidentified error.
  589. 12/14 1:48:21.973635 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  590. 12/14 1:48:21.973640 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F3D3BAC0 synchronization task has been failed!
  591. 12/14 1:48:21.973649 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8568 has failed due to unidentified error.
  592. 12/14 1:48:21.973655 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  593. 12/14 1:48:21.973660 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F0798BA0 synchronization task has been failed!
  594. 12/14 1:48:21.990185 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (16 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 10956789, expcmdsn = 10956787, maxcmdsn = 10957026.
  595. 12/14 1:48:21.990232 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10956783, itt = 10956869)
  596. 12/14 1:48:21.990245 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10956784, itt = 10956870)
  597. 12/14 1:48:21.990256 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10956785, itt = 10956871)
  598. 12/14 1:48:21.990268 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86991E8 has failed due to unidentified error.
  599. 12/14 1:48:21.990278 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  600. 12/14 1:48:21.990287 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F0798020 synchronization task has been failed!
  601. 12/14 1:48:21.990307 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EC4F39D8 has failed due to unidentified error.
  602. 12/14 1:48:21.990317 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  603. 12/14 1:48:21.990323 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F0798E80 synchronization task has been failed!
  604. 12/14 1:48:21.990336 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8268 has failed due to unidentified error.
  605. 12/14 1:48:21.990345 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  606. 12/14 1:48:21.990352 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F0799720 synchronization task has been failed!
  607. 12/14 1:48:22.005924 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (15 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 10956789, expcmdsn = 10956787, maxcmdsn = 10957026.
  608. 12/14 1:48:22.005997 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10956786, itt = 10956872)
  609. 12/14 1:48:22.006018 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 10956787, itt = 10956873)
  610. 12/14 1:48:22.006041 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6158 has failed due to unidentified error.
  611. 12/14 1:48:22.006059 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  612. 12/14 1:48:22.006117 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F0799A00 synchronization task has been failed!
  613. 12/14 1:48:22.006154 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6758 has failed due to unidentified error.
  614. 12/14 1:48:22.006187 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  615. 12/14 1:48:22.006200 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F3D3C640 synchronization task has been failed!
  616. 12/14 1:48:26.756616 65c8 HA: CHAPartnerNode::AfterSynchronization: Event: Full synchronization has been completed (error: 1237)
  617. 12/14 1:48:26.759056 5378 func: >>> HANode::reportEvent
  618. 12/14 1:48:26.759088 5378 func: >>> iScsiServer::sendNotification
  619. 12/14 1:48:26.759097 5378 func: >>> CEventDataBase::AddRecord
  620. 12/14 1:48:26.759104 5378 EventDB: CEventDataBase::AddRecord: Code 783, severity 1, additional strings: 2
  621. 12/14 1:48:26.759156 5378 func: <<< CEventDataBase::AddRecord
  622. 12/14 1:48:26.759164 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 783).
  623. 12/14 1:48:26.759740 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  624. 12/14 1:48:26.759763 5378 func: <<< iScsiServer::sendNotification
  625. 12/14 1:48:26.759771 5378 func: <<< HANode::reportEvent
  626. 12/14 1:48:26.759777 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_COMPLETE_WITH_ERROR, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'
  627. 12/14 1:48:26.773297 38d8 func: >>> HANode::onScsiOp_CustomControl_HeartbeatPing
  628. 12/14 1:48:26.773394 38d8 Common: CStarWindStorageDevice::getLastPerfDegEventElapsedTimeMs: perfDegEventElapsedTime = -1
  629. 12/14 1:48:26.773401 38d8 func: <<< HANode::onScsiOp_CustomControl_HeartbeatPing
  630. 12/14 1:48:26.774168 38d8 func: >>> CHAPartnerNode::ShutdownNode
  631. 12/14 1:48:26.774183 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  632. 12/14 1:48:26.774192 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  633. 12/14 1:48:26.774200 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  634. 12/14 1:48:26.774206 38d8 func: <<< CHAPartnerNode::ShutdownNode
  635. 12/14 1:48:30.065549 38d8 func: >>> CHAPartnerNode::ShutdownNode
  636. 12/14 1:48:30.065591 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  637. 12/14 1:48:30.065599 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  638. 12/14 1:48:30.065605 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  639. 12/14 1:48:30.065611 38d8 func: <<< CHAPartnerNode::ShutdownNode
  640. 12/14 1:48:30.071144 2d00 HA: HANode::onScsiOp_CustomControl_SynchronizationRequest: The partner iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03 has requested the 1 synchronization type.
  641. 12/14 1:48:30.074460 2d00 HA: CHAPartnerNode::Synchronize: ENTERed with params: p_SyncType = 1, p_ulSnapshotID = 0(0x0), p_pStorageDevice = 0x000002C7ECEDC4B0, p_pSyncBarrier = 0x000002C7ECEDC858, p_puliStartSector = 0x000000D2528FF118 (value is 0x0), p_puliEndSector = 0x000000D2528FF110 (value is 0xAEFFFFF), p_ulIOBlockSizeInSectors = 64, p_ulSectorSizeInUCHARs = 4096, p_bContinue = no, p_bVaaiWriteSameSupport = yes
  642. 12/14 1:48:30.074519 2d00 HA: HANode::updatePartnerStatus: Statuses is already actual, we don't need to update them.
  643. 12/14 1:48:30.074546 2d00 func: >>> CHAPartnerNode::SetSyncStatus
  644. 12/14 1:48:30.074563 2d00 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  645. 12/14 1:48:30.074580 2d00 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 2 from 3. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  646. 12/14 1:48:30.074596 2d00 func: >>> HANode::reportEvent
  647. 12/14 1:48:30.074614 2d00 func: >>> iScsiServer::sendNotification
  648. 12/14 1:48:30.074630 2d00 func: >>> CEventDataBase::AddRecord
  649. 12/14 1:48:30.074646 2d00 EventDB: CEventDataBase::AddRecord: Code 770, severity 2, additional strings: 2
  650. 12/14 1:48:30.074759 2d00 func: <<< CEventDataBase::AddRecord
  651. 12/14 1:48:30.074776 2d00 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 770).
  652. 12/14 1:48:30.075509 2d00 Srv: iScsiServer::sendNotification: Send reaction finish.
  653. 12/14 1:48:30.075544 2d00 func: <<< iScsiServer::sendNotification
  654. 12/14 1:48:30.075561 2d00 func: <<< HANode::reportEvent
  655. 12/14 1:48:30.075577 2d00 func: <<< CHAPartnerNode::SetSyncStatus
  656. 12/14 1:48:30.075981 4ce0 func: >>> CHAPartnerNode::SetSyncStatus
  657. 12/14 1:48:30.076027 4ce0 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  658. 12/14 1:48:30.076045 4ce0 func: <<< CHAPartnerNode::SetSyncStatus
  659. 12/14 1:48:30.079390 5378 func: >>> HANode::reportEvent
  660. 12/14 1:48:30.079439 5378 func: >>> iScsiServer::sendNotification
  661. 12/14 1:48:30.079579 5378 func: >>> CEventDataBase::AddRecord
  662. 12/14 1:48:30.079594 5378 EventDB: CEventDataBase::AddRecord: Code 780, severity 1, additional strings: 3
  663. 12/14 1:48:30.079669 5378 func: <<< CEventDataBase::AddRecord
  664. 12/14 1:48:30.079681 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 780).
  665. 12/14 1:48:30.080522 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  666. 12/14 1:48:30.080645 5378 func: <<< iScsiServer::sendNotification
  667. 12/14 1:48:30.080670 5378 func: <<< HANode::reportEvent
  668. 12/14 1:48:30.080681 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_STARTED, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03', Synchronization type: 'Full'
  669. 12/14 2:11:06.033683 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (9985 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 11418138, expcmdsn = 11418136, maxcmdsn = 11418376.
  670. 12/14 2:11:06.034394 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 11418122, itt = 11418209)
  671. 12/14 2:11:06.034427 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 11418123, itt = 11418210)
  672. 12/14 2:11:06.034450 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 11418124, itt = 11418211)
  673. 12/14 2:11:06.034469 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 11418125, itt = 11418212)
  674. 12/14 2:11:06.034495 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 11418126, itt = 11418213)
  675. 12/14 2:11:06.034516 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 11418127, itt = 11418214)
  676. 12/14 2:11:06.034536 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 11418128, itt = 11418215)
  677. 12/14 2:11:06.034556 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 11418129, itt = 11418216)
  678. 12/14 2:11:06.034575 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 11418130, itt = 11418217)
  679. 12/14 2:11:06.034594 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 11418131, itt = 11418218)
  680. 12/14 2:11:06.034617 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 11418132, itt = 11418219)
  681. 12/14 2:11:06.034641 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EC4F39D8 has failed due to unidentified error.
  682. 12/14 2:11:06.034662 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  683. 12/14 2:11:06.034679 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC538310 synchronization task has been failed!
  684. 12/14 2:11:06.034720 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BF88 has failed due to unidentified error.
  685. 12/14 2:11:06.034740 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  686. 12/14 2:11:06.034765 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC538BB0 synchronization task has been failed!
  687. 12/14 2:11:06.034806 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6158 has failed due to unidentified error.
  688. 12/14 2:11:06.034825 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  689. 12/14 2:11:06.034839 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC5388D0 synchronization task has been failed!
  690. 12/14 2:11:06.034862 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D85E8 has failed due to unidentified error.
  691. 12/14 2:11:06.034764 4ce0 func: >>> CHAPartnerNode::SetSyncStatus
  692. 12/14 2:11:06.034954 4ce0 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  693. 12/14 2:11:06.034990 4ce0 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 3 from 2. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  694. 12/14 2:11:06.035018 4ce0 func: >>> HANode::reportEvent
  695. 12/14 2:11:06.034881 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  696. 12/14 2:11:06.035071 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC537A70 synchronization task has been failed!
  697. 12/14 2:11:06.035097 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86999E8 has failed due to unidentified error.
  698. 12/14 2:11:06.035117 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  699. 12/14 2:11:06.035130 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC5385F0 synchronization task has been failed!
  700. 12/14 2:11:06.035152 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86991E8 has failed due to unidentified error.
  701. 12/14 2:11:06.035159 4ce0 func: >>> iScsiServer::sendNotification
  702. 12/14 2:11:06.035276 4ce0 func: >>> CEventDataBase::AddRecord
  703. 12/14 2:11:06.035314 4ce0 EventDB: CEventDataBase::AddRecord: Code 771, severity 3, additional strings: 2
  704. 12/14 2:11:06.035172 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  705. 12/14 2:11:06.035364 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F3D3CEE0 synchronization task has been failed!
  706. 12/14 2:11:06.035390 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8568 has failed due to unidentified error.
  707. 12/14 2:11:06.035409 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  708. 12/14 2:11:06.035423 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC538030 synchronization task has been failed!
  709. 12/14 2:11:06.035446 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BE08 has failed due to unidentified error.
  710. 12/14 2:11:06.035467 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  711. 12/14 2:11:06.035482 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC537D50 synchronization task has been failed!
  712. 12/14 2:11:06.035509 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86994E8 has failed due to unidentified error.
  713. 12/14 2:11:06.035528 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  714. 12/14 2:11:06.035548 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC537790 synchronization task has been failed!
  715. 12/14 2:11:06.035576 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE60D8 has failed due to unidentified error.
  716. 12/14 2:11:06.035595 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  717. 12/14 2:11:06.035609 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F3D3D1C0 synchronization task has been failed!
  718. 12/14 2:11:06.035633 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E8699A68 has failed due to unidentified error.
  719. 12/14 2:11:06.035654 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  720. 12/14 2:11:06.035668 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC538E90 synchronization task has been failed!
  721. 12/14 2:11:06.035700 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 11418133, itt = 11418220)
  722. 12/14 2:11:06.035666 4ce0 func: <<< CEventDataBase::AddRecord
  723. 12/14 2:11:06.035760 4ce0 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 771).
  724. 12/14 2:11:06.035721 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 11418134, itt = 11418221)
  725. 12/14 2:11:06.035801 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 11418135, itt = 11418222)
  726. 12/14 2:11:06.035821 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 11418136, itt = 11418223)
  727. 12/14 2:11:06.035841 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 11418137, itt = 11418224)
  728. 12/14 2:11:06.035861 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7BE8 has failed due to unidentified error.
  729. 12/14 2:11:06.035881 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  730. 12/14 2:11:06.035895 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC5374B0 synchronization task has been failed!
  731. 12/14 2:11:06.035919 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86993E8 has failed due to unidentified error.
  732. 12/14 2:11:06.035938 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  733. 12/14 2:11:06.035952 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F3D3D4A0 synchronization task has been failed!
  734. 12/14 2:11:06.035974 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8268 has failed due to unidentified error.
  735. 12/14 2:11:06.035993 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  736. 12/14 2:11:06.036006 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F3D3BAC0 synchronization task has been failed!
  737. 12/14 2:11:06.036028 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6258 has failed due to unidentified error.
  738. 12/14 2:11:06.036047 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  739. 12/14 2:11:06.036060 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC5371D0 synchronization task has been failed!
  740. 12/14 2:11:06.036082 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6758 has failed due to unidentified error.
  741. 12/14 2:11:06.036101 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  742. 12/14 2:11:06.036118 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F3D3DA60 synchronization task has been failed!
  743. 12/14 2:11:06.037039 4ce0 Srv: iScsiServer::sendNotification: Send reaction finish.
  744. 12/14 2:11:06.037084 4ce0 func: <<< iScsiServer::sendNotification
  745. 12/14 2:11:06.037104 4ce0 func: <<< HANode::reportEvent
  746. 12/14 2:11:06.037122 4ce0 func: <<< CHAPartnerNode::SetSyncStatus
  747. 12/14 2:11:10.810279 38d8 func: >>> HANode::onScsiOp_CustomControl_HeartbeatPing
  748. 12/14 2:11:10.810457 38d8 Common: CStarWindStorageDevice::getLastPerfDegEventElapsedTimeMs: perfDegEventElapsedTime = -1
  749. 12/14 2:11:10.810482 38d8 func: <<< HANode::onScsiOp_CustomControl_HeartbeatPing
  750. 12/14 2:11:10.811730 38d8 func: >>> CHAPartnerNode::ShutdownNode
  751. 12/14 2:11:10.811776 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  752. 12/14 2:11:10.811803 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  753. 12/14 2:11:10.811823 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  754. 12/14 2:11:10.811841 38d8 func: <<< CHAPartnerNode::ShutdownNode
  755. 12/14 2:11:10.812833 4ce0 HA: CHAPartnerNode::AfterSynchronization: Event: Full synchronization has been completed (error: 1237)
  756. 12/14 2:11:10.817918 5378 Common: *** sw_common::Sw_Disk_Header::backup: Asynchronous header 'C:\Program Files\StarWind Software\StarWind\headers\datastore03\datastore03_HA.swdsk' operation in progress!
  757. 12/14 2:11:10.817950 5378 func: >>> HANode::reportEvent
  758. 12/14 2:11:10.817963 5378 func: >>> iScsiServer::sendNotification
  759. 12/14 2:11:10.817973 5378 func: >>> CEventDataBase::AddRecord
  760. 12/14 2:11:10.817981 5378 EventDB: CEventDataBase::AddRecord: Code 783, severity 1, additional strings: 2
  761. 12/14 2:11:10.818035 5378 func: <<< CEventDataBase::AddRecord
  762. 12/14 2:11:10.818044 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 783).
  763. 12/14 2:11:10.818517 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  764. 12/14 2:11:10.818540 5378 func: <<< iScsiServer::sendNotification
  765. 12/14 2:11:10.818548 5378 func: <<< HANode::reportEvent
  766. 12/14 2:11:10.818555 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_COMPLETE_WITH_ERROR, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'
  767. 12/14 2:11:12.386122 38d8 func: >>> CHAPartnerNode::ShutdownNode
  768. 12/14 2:11:12.386192 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  769. 12/14 2:11:12.386218 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  770. 12/14 2:11:12.386240 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  771. 12/14 2:11:12.386259 38d8 func: <<< CHAPartnerNode::ShutdownNode
  772. 12/14 2:11:12.393021 2d00 HA: HANode::onScsiOp_CustomControl_SynchronizationRequest: The partner iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03 has requested the 1 synchronization type.
  773. 12/14 2:11:12.393545 2d00 HA: CHAPartnerNode::Synchronize: ENTERed with params: p_SyncType = 1, p_ulSnapshotID = 0(0x0), p_pStorageDevice = 0x000002C7ECEDC4B0, p_pSyncBarrier = 0x000002C7ECEDC858, p_puliStartSector = 0x000000D2528FF118 (value is 0x0), p_puliEndSector = 0x000000D2528FF110 (value is 0xAEFFFFF), p_ulIOBlockSizeInSectors = 64, p_ulSectorSizeInUCHARs = 4096, p_bContinue = no, p_bVaaiWriteSameSupport = yes
  774. 12/14 2:11:12.393580 2d00 HA: HANode::updatePartnerStatus: Statuses is already actual, we don't need to update them.
  775. 12/14 2:11:12.393608 2d00 func: >>> CHAPartnerNode::SetSyncStatus
  776. 12/14 2:11:12.393626 2d00 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  777. 12/14 2:11:12.393644 2d00 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 2 from 3. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  778. 12/14 2:11:12.393662 2d00 func: >>> HANode::reportEvent
  779. 12/14 2:11:12.393692 2d00 func: >>> iScsiServer::sendNotification
  780. 12/14 2:11:12.393741 2d00 func: >>> CEventDataBase::AddRecord
  781. 12/14 2:11:12.393758 2d00 EventDB: CEventDataBase::AddRecord: Code 770, severity 2, additional strings: 2
  782. 12/14 2:11:12.393875 2d00 func: <<< CEventDataBase::AddRecord
  783. 12/14 2:11:12.393893 2d00 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 770).
  784. 12/14 2:11:12.394693 2d00 Srv: iScsiServer::sendNotification: Send reaction finish.
  785. 12/14 2:11:12.394731 2d00 func: <<< iScsiServer::sendNotification
  786. 12/14 2:11:12.394748 2d00 func: <<< HANode::reportEvent
  787. 12/14 2:11:12.394765 2d00 func: <<< CHAPartnerNode::SetSyncStatus
  788. 12/14 2:11:12.395018 6efc func: >>> CHAPartnerNode::SetSyncStatus
  789. 12/14 2:11:12.395044 6efc HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  790. 12/14 2:11:12.395051 6efc func: <<< CHAPartnerNode::SetSyncStatus
  791. 12/14 2:11:12.400539 5378 func: >>> HANode::reportEvent
  792. 12/14 2:11:12.400600 5378 func: >>> iScsiServer::sendNotification
  793. 12/14 2:11:12.400623 5378 func: >>> CEventDataBase::AddRecord
  794. 12/14 2:11:12.400643 5378 EventDB: CEventDataBase::AddRecord: Code 780, severity 1, additional strings: 3
  795. 12/14 2:11:12.400756 5378 func: <<< CEventDataBase::AddRecord
  796. 12/14 2:11:12.400778 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 780).
  797. 12/14 2:11:12.401929 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  798. 12/14 2:11:12.401990 5378 func: <<< iScsiServer::sendNotification
  799. 12/14 2:11:12.402026 5378 func: <<< HANode::reportEvent
  800. 12/14 2:11:12.402058 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_STARTED, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03', Synchronization type: 'Full'
  801. 12/14 2:33:47.064641 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (9968 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 11879126, expcmdsn = 11879124, maxcmdsn = 11879364.
  802. 12/14 2:33:47.064729 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 11879110, itt = 11879197)
  803. 12/14 2:33:47.064756 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 11879111, itt = 11879198)
  804. 12/14 2:33:47.064780 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 11879112, itt = 11879199)
  805. 12/14 2:33:47.064800 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 11879113, itt = 11879200)
  806. 12/14 2:33:47.064821 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 11879114, itt = 11879201)
  807. 12/14 2:33:47.064843 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 11879115, itt = 11879202)
  808. 12/14 2:33:47.064863 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 11879116, itt = 11879203)
  809. 12/14 2:33:47.064884 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 11879117, itt = 11879204)
  810. 12/14 2:33:47.064902 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 11879118, itt = 11879205)
  811. 12/14 2:33:47.064928 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7BE8 has failed due to unidentified error.
  812. 12/14 2:33:47.064949 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  813. 12/14 2:33:47.064966 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC9A2850 synchronization task has been failed!
  814. 12/14 2:33:47.065006 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86999E8 has failed due to unidentified error.
  815. 12/14 2:33:47.065026 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  816. 12/14 2:33:47.065040 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC9A2290 synchronization task has been failed!
  817. 12/14 2:33:47.065061 6efc func: >>> CHAPartnerNode::SetSyncStatus
  818. 12/14 2:33:47.065134 6efc HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  819. 12/14 2:33:47.065078 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8568 has failed due to unidentified error.
  820. 12/14 2:33:47.065158 6efc HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 3 from 2. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  821. 12/14 2:33:47.065243 6efc func: >>> HANode::reportEvent
  822. 12/14 2:33:47.065204 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  823. 12/14 2:33:47.065279 6efc func: >>> iScsiServer::sendNotification
  824. 12/14 2:33:47.065292 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC9A1150 synchronization task has been failed!
  825. 12/14 2:33:47.065310 6efc func: >>> CEventDataBase::AddRecord
  826. 12/14 2:33:47.065375 6efc EventDB: CEventDataBase::AddRecord: Code 771, severity 3, additional strings: 2
  827. 12/14 2:33:47.065347 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E8699A68 has failed due to unidentified error.
  828. 12/14 2:33:47.065447 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  829. 12/14 2:33:47.065467 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C78119A190 synchronization task has been failed!
  830. 12/14 2:33:47.065497 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D85E8 has failed due to unidentified error.
  831. 12/14 2:33:47.065519 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  832. 12/14 2:33:47.065535 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C781197950 synchronization task has been failed!
  833. 12/14 2:33:47.065560 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8268 has failed due to unidentified error.
  834. 12/14 2:33:47.065595 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  835. 12/14 2:33:47.065613 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC9A19F0 synchronization task has been failed!
  836. 12/14 2:33:47.065637 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE60D8 has failed due to unidentified error.
  837. 12/14 2:33:47.065656 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  838. 12/14 2:33:47.065670 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7811987B0 synchronization task has been failed!
  839. 12/14 2:33:47.065692 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7D68 has failed due to unidentified error.
  840. 12/14 2:33:47.065712 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  841. 12/14 2:33:47.065726 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC9A1710 synchronization task has been failed!
  842. 12/14 2:33:47.065757 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6158 has failed due to unidentified error.
  843. 12/14 2:33:47.065781 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  844. 12/14 2:33:47.065729 6efc func: <<< CEventDataBase::AddRecord
  845. 12/14 2:33:47.065829 6efc Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 771).
  846. 12/14 2:33:47.065795 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC9A2570 synchronization task has been failed!
  847. 12/14 2:33:47.066853 6efc Srv: iScsiServer::sendNotification: Send reaction finish.
  848. 12/14 2:33:47.066894 6efc func: <<< iScsiServer::sendNotification
  849. 12/14 2:33:47.066914 6efc func: <<< HANode::reportEvent
  850. 12/14 2:33:47.066932 6efc func: <<< CHAPartnerNode::SetSyncStatus
  851. 12/14 2:33:47.083794 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (16 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 11879127, expcmdsn = 11879124, maxcmdsn = 11879364.
  852. 12/14 2:33:47.083876 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 11879119, itt = 11879206)
  853. 12/14 2:33:47.083903 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 11879120, itt = 11879207)
  854. 12/14 2:33:47.083925 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 11879121, itt = 11879208)
  855. 12/14 2:33:47.083945 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 11879122, itt = 11879209)
  856. 12/14 2:33:47.083970 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86993E8 has failed due to unidentified error.
  857. 12/14 2:33:47.083991 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  858. 12/14 2:33:47.084008 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC9A1430 synchronization task has been failed!
  859. 12/14 2:33:47.084047 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86991E8 has failed due to unidentified error.
  860. 12/14 2:33:47.084070 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  861. 12/14 2:33:47.084083 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC9A1CD0 synchronization task has been failed!
  862. 12/14 2:33:47.084116 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BF88 has failed due to unidentified error.
  863. 12/14 2:33:47.084143 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  864. 12/14 2:33:47.084157 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C781197C30 synchronization task has been failed!
  865. 12/14 2:33:47.084184 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6258 has failed due to unidentified error.
  866. 12/14 2:33:47.084262 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  867. 12/14 2:33:47.084279 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC9A1FB0 synchronization task has been failed!
  868. 12/14 2:33:47.099610 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (16 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 11879127, expcmdsn = 11879124, maxcmdsn = 11879364.
  869. 12/14 2:33:47.099671 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 11879123, itt = 11879210)
  870. 12/14 2:33:47.099690 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 11879124, itt = 11879211)
  871. 12/14 2:33:47.099712 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 11879125, itt = 11879212)
  872. 12/14 2:33:47.099737 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EC4F39D8 has failed due to unidentified error.
  873. 12/14 2:33:47.099752 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  874. 12/14 2:33:47.099771 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC9A2B30 synchronization task has been failed!
  875. 12/14 2:33:47.099804 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86994E8 has failed due to unidentified error.
  876. 12/14 2:33:47.099821 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  877. 12/14 2:33:47.099831 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C781199BD0 synchronization task has been failed!
  878. 12/14 2:33:47.099851 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BE08 has failed due to unidentified error.
  879. 12/14 2:33:47.099865 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  880. 12/14 2:33:47.099875 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC9A0E70 synchronization task has been failed!
  881. 12/14 2:33:51.891873 6efc HA: CHAPartnerNode::AfterSynchronization: Event: Full synchronization has been completed (error: 1237)
  882. 12/14 2:33:51.896028 5378 func: >>> HANode::reportEvent
  883. 12/14 2:33:51.896090 5378 func: >>> iScsiServer::sendNotification
  884. 12/14 2:33:51.896115 5378 func: >>> CEventDataBase::AddRecord
  885. 12/14 2:33:51.896137 5378 EventDB: CEventDataBase::AddRecord: Code 783, severity 1, additional strings: 2
  886. 12/14 2:33:51.896221 5378 func: <<< CEventDataBase::AddRecord
  887. 12/14 2:33:51.896243 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 783).
  888. 12/14 2:33:51.897261 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  889. 12/14 2:33:51.897323 5378 func: <<< iScsiServer::sendNotification
  890. 12/14 2:33:51.897357 5378 func: <<< HANode::reportEvent
  891. 12/14 2:33:51.897387 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_COMPLETE_WITH_ERROR, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'
  892. 12/14 2:33:51.902903 38d8 func: >>> HANode::onScsiOp_CustomControl_HeartbeatPing
  893. 12/14 2:33:51.903025 38d8 Common: CStarWindStorageDevice::getLastPerfDegEventElapsedTimeMs: perfDegEventElapsedTime = -1
  894. 12/14 2:33:51.903039 38d8 func: <<< HANode::onScsiOp_CustomControl_HeartbeatPing
  895. 12/14 2:33:51.903953 38d8 func: >>> CHAPartnerNode::ShutdownNode
  896. 12/14 2:33:51.904005 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  897. 12/14 2:33:51.904026 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  898. 12/14 2:33:51.904047 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  899. 12/14 2:33:51.904065 38d8 func: <<< CHAPartnerNode::ShutdownNode
  900. 12/14 2:33:54.800786 38d8 func: >>> CHAPartnerNode::ShutdownNode
  901. 12/14 2:33:54.800852 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  902. 12/14 2:33:54.800874 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  903. 12/14 2:33:54.800893 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  904. 12/14 2:33:54.800908 38d8 func: <<< CHAPartnerNode::ShutdownNode
  905. 12/14 2:33:54.805873 2d00 HA: HANode::onScsiOp_CustomControl_SynchronizationRequest: The partner iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03 has requested the 1 synchronization type.
  906. 12/14 2:33:54.807668 2d00 HA: CHAPartnerNode::Synchronize: ENTERed with params: p_SyncType = 1, p_ulSnapshotID = 0(0x0), p_pStorageDevice = 0x000002C7ECEDC4B0, p_pSyncBarrier = 0x000002C7ECEDC858, p_puliStartSector = 0x000000D2528FF118 (value is 0x0), p_puliEndSector = 0x000000D2528FF110 (value is 0xAEFFFFF), p_ulIOBlockSizeInSectors = 64, p_ulSectorSizeInUCHARs = 4096, p_bContinue = no, p_bVaaiWriteSameSupport = yes
  907. 12/14 2:33:54.807691 2d00 HA: HANode::updatePartnerStatus: Statuses is already actual, we don't need to update them.
  908. 12/14 2:33:54.807705 2d00 func: >>> CHAPartnerNode::SetSyncStatus
  909. 12/14 2:33:54.807720 2d00 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  910. 12/14 2:33:54.807726 2d00 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 2 from 3. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  911. 12/14 2:33:54.807733 2d00 func: >>> HANode::reportEvent
  912. 12/14 2:33:54.807741 2d00 func: >>> iScsiServer::sendNotification
  913. 12/14 2:33:54.807748 2d00 func: >>> CEventDataBase::AddRecord
  914. 12/14 2:33:54.807754 2d00 EventDB: CEventDataBase::AddRecord: Code 770, severity 2, additional strings: 2
  915. 12/14 2:33:54.807789 2d00 func: <<< CEventDataBase::AddRecord
  916. 12/14 2:33:54.807795 2d00 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 770).
  917. 12/14 2:33:54.808223 2d00 Srv: iScsiServer::sendNotification: Send reaction finish.
  918. 12/14 2:33:54.808242 2d00 func: <<< iScsiServer::sendNotification
  919. 12/14 2:33:54.808249 2d00 func: <<< HANode::reportEvent
  920. 12/14 2:33:54.808255 2d00 func: <<< CHAPartnerNode::SetSyncStatus
  921. 12/14 2:33:54.808528 66e8 func: >>> CHAPartnerNode::SetSyncStatus
  922. 12/14 2:33:54.808570 66e8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  923. 12/14 2:33:54.808585 66e8 func: <<< CHAPartnerNode::SetSyncStatus
  924. 12/14 2:33:54.811079 5378 func: >>> HANode::reportEvent
  925. 12/14 2:33:54.811123 5378 func: >>> iScsiServer::sendNotification
  926. 12/14 2:33:54.811132 5378 func: >>> CEventDataBase::AddRecord
  927. 12/14 2:33:54.811139 5378 EventDB: CEventDataBase::AddRecord: Code 780, severity 1, additional strings: 3
  928. 12/14 2:33:54.811190 5378 func: <<< CEventDataBase::AddRecord
  929. 12/14 2:33:54.811197 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 780).
  930. 12/14 2:33:54.811756 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  931. 12/14 2:33:54.811792 5378 func: <<< iScsiServer::sendNotification
  932. 12/14 2:33:54.811806 5378 func: <<< HANode::reportEvent
  933. 12/14 2:33:54.811816 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_STARTED, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03', Synchronization type: 'Full'
  934. 12/14 2:56:29.913560 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (9969 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 12339043, expcmdsn = 12339042, maxcmdsn = 12339281.
  935. 12/14 2:56:29.913758 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 12339027, itt = 12339114)
  936. 12/14 2:56:29.913777 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 12339028, itt = 12339115)
  937. 12/14 2:56:29.913790 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 12339029, itt = 12339116)
  938. 12/14 2:56:29.913801 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 12339030, itt = 12339117)
  939. 12/14 2:56:29.913826 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8568 has failed due to unidentified error.
  940. 12/14 2:56:29.913839 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  941. 12/14 2:56:29.913848 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7E93983C0 synchronization task has been failed!
  942. 12/14 2:56:29.913871 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6758 has failed due to unidentified error.
  943. 12/14 2:56:29.913882 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  944. 12/14 2:56:29.913890 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7E93986A0 synchronization task has been failed!
  945. 12/14 2:56:29.913893 66e8 func: >>> CHAPartnerNode::SetSyncStatus
  946. 12/14 2:56:29.914022 66e8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  947. 12/14 2:56:29.914033 66e8 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 3 from 2. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  948. 12/14 2:56:29.913904 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86993E8 has failed due to unidentified error.
  949. 12/14 2:56:29.914068 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  950. 12/14 2:56:29.914082 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7E9399DA0 synchronization task has been failed!
  951. 12/14 2:56:29.914040 66e8 func: >>> HANode::reportEvent
  952. 12/14 2:56:29.914102 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86991E8 has failed due to unidentified error.
  953. 12/14 2:56:29.914116 66e8 func: >>> iScsiServer::sendNotification
  954. 12/14 2:56:29.914121 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  955. 12/14 2:56:29.914142 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7E9399AC0 synchronization task has been failed!
  956. 12/14 2:56:29.914131 66e8 func: >>> CEventDataBase::AddRecord
  957. 12/14 2:56:29.914167 66e8 EventDB: CEventDataBase::AddRecord: Code 771, severity 3, additional strings: 2
  958. 12/14 2:56:29.914259 66e8 func: <<< CEventDataBase::AddRecord
  959. 12/14 2:56:29.914268 66e8 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 771).
  960. 12/14 2:56:29.914916 66e8 Srv: iScsiServer::sendNotification: Send reaction finish.
  961. 12/14 2:56:29.914936 66e8 func: <<< iScsiServer::sendNotification
  962. 12/14 2:56:29.914945 66e8 func: <<< HANode::reportEvent
  963. 12/14 2:56:29.914954 66e8 func: <<< CHAPartnerNode::SetSyncStatus
  964. 12/14 2:56:29.917094 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 12339031, itt = 12339118)
  965. 12/14 2:56:29.917117 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 12339032, itt = 12339119)
  966. 12/14 2:56:29.917126 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 12339033, itt = 12339120)
  967. 12/14 2:56:29.917134 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 12339034, itt = 12339121)
  968. 12/14 2:56:29.917143 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 12339035, itt = 12339122)
  969. 12/14 2:56:29.917151 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 12339036, itt = 12339123)
  970. 12/14 2:56:29.917161 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86994E8 has failed due to unidentified error.
  971. 12/14 2:56:29.917169 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  972. 12/14 2:56:29.917175 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7E9398C60 synchronization task has been failed!
  973. 12/14 2:56:29.917192 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86999E8 has failed due to unidentified error.
  974. 12/14 2:56:29.917200 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  975. 12/14 2:56:29.917206 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C781E93990 synchronization task has been failed!
  976. 12/14 2:56:29.917217 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE60D8 has failed due to unidentified error.
  977. 12/14 2:56:29.917225 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  978. 12/14 2:56:29.917240 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7E93980E0 synchronization task has been failed!
  979. 12/14 2:56:29.917254 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8268 has failed due to unidentified error.
  980. 12/14 2:56:29.917262 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  981. 12/14 2:56:29.917267 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7E9398980 synchronization task has been failed!
  982. 12/14 2:56:29.917277 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E8699A68 has failed due to unidentified error.
  983. 12/14 2:56:29.917285 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  984. 12/14 2:56:29.917290 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C781E95EF0 synchronization task has been failed!
  985. 12/14 2:56:29.917301 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EC4F39D8 has failed due to unidentified error.
  986. 12/14 2:56:29.917309 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  987. 12/14 2:56:29.917314 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7E93997E0 synchronization task has been failed!
  988. 12/14 2:56:29.932533 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (15 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 12339044, expcmdsn = 12339042, maxcmdsn = 12339281.
  989. 12/14 2:56:29.932567 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 12339037, itt = 12339124)
  990. 12/14 2:56:29.932577 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 12339038, itt = 12339125)
  991. 12/14 2:56:29.932584 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 12339039, itt = 12339126)
  992. 12/14 2:56:29.932591 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 12339040, itt = 12339127)
  993. 12/14 2:56:29.932600 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 12339041, itt = 12339128)
  994. 12/14 2:56:29.932607 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 12339042, itt = 12339129)
  995. 12/14 2:56:29.932616 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D85E8 has failed due to unidentified error.
  996. 12/14 2:56:29.932623 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  997. 12/14 2:56:29.932629 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7E9399500 synchronization task has been failed!
  998. 12/14 2:56:29.932649 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6158 has failed due to unidentified error.
  999. 12/14 2:56:29.932657 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1000. 12/14 2:56:29.932662 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C781E92E10 synchronization task has been failed!
  1001. 12/14 2:56:29.932670 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7D68 has failed due to unidentified error.
  1002. 12/14 2:56:29.932677 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1003. 12/14 2:56:29.932682 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7E9399220 synchronization task has been failed!
  1004. 12/14 2:56:29.932690 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BF88 has failed due to unidentified error.
  1005. 12/14 2:56:29.932705 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1006. 12/14 2:56:29.932710 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C781E95C10 synchronization task has been failed!
  1007. 12/14 2:56:29.932718 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7BE8 has failed due to unidentified error.
  1008. 12/14 2:56:29.932725 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1009. 12/14 2:56:29.932730 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7E9398F40 synchronization task has been failed!
  1010. 12/14 2:56:29.932738 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6258 has failed due to unidentified error.
  1011. 12/14 2:56:29.932744 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1012. 12/14 2:56:29.932749 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C781E95090 synchronization task has been failed!
  1013. 12/14 2:56:34.555144 66e8 HA: CHAPartnerNode::AfterSynchronization: Event: Full synchronization has been completed (error: 1237)
  1014. 12/14 2:56:34.557646 5378 func: >>> HANode::reportEvent
  1015. 12/14 2:56:34.557680 5378 func: >>> iScsiServer::sendNotification
  1016. 12/14 2:56:34.557689 5378 func: >>> CEventDataBase::AddRecord
  1017. 12/14 2:56:34.557695 5378 EventDB: CEventDataBase::AddRecord: Code 783, severity 1, additional strings: 2
  1018. 12/14 2:56:34.557732 5378 func: <<< CEventDataBase::AddRecord
  1019. 12/14 2:56:34.557739 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 783).
  1020. 12/14 2:56:34.558232 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  1021. 12/14 2:56:34.558253 5378 func: <<< iScsiServer::sendNotification
  1022. 12/14 2:56:34.558260 5378 func: <<< HANode::reportEvent
  1023. 12/14 2:56:34.558266 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_COMPLETE_WITH_ERROR, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'
  1024. 12/14 2:56:34.558393 38d8 func: >>> HANode::onScsiOp_CustomControl_HeartbeatPing
  1025. 12/14 2:56:34.558476 38d8 Common: CStarWindStorageDevice::getLastPerfDegEventElapsedTimeMs: perfDegEventElapsedTime = -1
  1026. 12/14 2:56:34.558484 38d8 func: <<< HANode::onScsiOp_CustomControl_HeartbeatPing
  1027. 12/14 2:56:34.559314 38d8 func: >>> CHAPartnerNode::ShutdownNode
  1028. 12/14 2:56:34.559327 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  1029. 12/14 2:56:34.559333 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  1030. 12/14 2:56:34.559340 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  1031. 12/14 2:56:34.559346 38d8 func: <<< CHAPartnerNode::ShutdownNode
  1032. 12/14 2:56:37.030780 38d8 func: >>> CHAPartnerNode::ShutdownNode
  1033. 12/14 2:56:37.030902 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  1034. 12/14 2:56:37.030929 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  1035. 12/14 2:56:37.030957 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  1036. 12/14 2:56:37.030976 38d8 func: <<< CHAPartnerNode::ShutdownNode
  1037. 12/14 2:56:37.036631 2d00 HA: HANode::onScsiOp_CustomControl_SynchronizationRequest: The partner iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03 has requested the 1 synchronization type.
  1038. 12/14 2:56:37.037488 2d00 HA: CHAPartnerNode::Synchronize: ENTERed with params: p_SyncType = 1, p_ulSnapshotID = 0(0x0), p_pStorageDevice = 0x000002C7ECEDC4B0, p_pSyncBarrier = 0x000002C7ECEDC858, p_puliStartSector = 0x000000D2528FF118 (value is 0x0), p_puliEndSector = 0x000000D2528FF110 (value is 0xAEFFFFF), p_ulIOBlockSizeInSectors = 64, p_ulSectorSizeInUCHARs = 4096, p_bContinue = no, p_bVaaiWriteSameSupport = yes
  1039. 12/14 2:56:37.037553 2d00 HA: HANode::updatePartnerStatus: Statuses is already actual, we don't need to update them.
  1040. 12/14 2:56:37.037588 2d00 func: >>> CHAPartnerNode::SetSyncStatus
  1041. 12/14 2:56:37.037608 2d00 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  1042. 12/14 2:56:37.037628 2d00 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 2 from 3. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  1043. 12/14 2:56:37.037648 2d00 func: >>> HANode::reportEvent
  1044. 12/14 2:56:37.037668 2d00 func: >>> iScsiServer::sendNotification
  1045. 12/14 2:56:37.037687 2d00 func: >>> CEventDataBase::AddRecord
  1046. 12/14 2:56:37.037706 2d00 EventDB: CEventDataBase::AddRecord: Code 770, severity 2, additional strings: 2
  1047. 12/14 2:56:37.037788 2d00 func: <<< CEventDataBase::AddRecord
  1048. 12/14 2:56:37.037807 2d00 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 770).
  1049. 12/14 2:56:37.038560 2d00 Srv: iScsiServer::sendNotification: Send reaction finish.
  1050. 12/14 2:56:37.038598 2d00 func: <<< iScsiServer::sendNotification
  1051. 12/14 2:56:37.038617 2d00 func: <<< HANode::reportEvent
  1052. 12/14 2:56:37.038635 2d00 func: <<< CHAPartnerNode::SetSyncStatus
  1053. 12/14 2:56:37.038897 54f4 func: >>> CHAPartnerNode::SetSyncStatus
  1054. 12/14 2:56:37.038919 54f4 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  1055. 12/14 2:56:37.038926 54f4 func: <<< CHAPartnerNode::SetSyncStatus
  1056. 12/14 2:56:37.041799 5378 func: >>> HANode::reportEvent
  1057. 12/14 2:56:37.041942 5378 func: >>> iScsiServer::sendNotification
  1058. 12/14 2:56:37.041960 5378 func: >>> CEventDataBase::AddRecord
  1059. 12/14 2:56:37.042044 5378 EventDB: CEventDataBase::AddRecord: Code 780, severity 1, additional strings: 3
  1060. 12/14 2:56:37.042106 5378 func: <<< CEventDataBase::AddRecord
  1061. 12/14 2:56:37.042120 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 780).
  1062. 12/14 2:56:37.042806 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  1063. 12/14 2:56:37.042839 5378 func: <<< iScsiServer::sendNotification
  1064. 12/14 2:56:37.042853 5378 func: <<< HANode::reportEvent
  1065. 12/14 2:56:37.042866 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_STARTED, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03', Synchronization type: 'Full'
  1066. 12/14 3:19:11.298528 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (9953 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 12798641, expcmdsn = 12798639, maxcmdsn = 12798879.
  1067. 12/14 3:19:11.298637 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 12798625, itt = 12798712)
  1068. 12/14 3:19:11.298652 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 12798626, itt = 12798713)
  1069. 12/14 3:19:11.298662 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 12798627, itt = 12798714)
  1070. 12/14 3:19:11.298680 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86993E8 has failed due to unidentified error.
  1071. 12/14 3:19:11.298690 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1072. 12/14 3:19:11.298698 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F0798BA0 synchronization task has been failed!
  1073. 12/14 3:19:11.298719 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8268 has failed due to unidentified error.
  1074. 12/14 3:19:11.298728 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1075. 12/14 3:19:11.298734 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C78653D0E0 synchronization task has been failed!
  1076. 12/14 3:19:11.298756 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7D68 has failed due to unidentified error.
  1077. 12/14 3:19:11.298766 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1078. 12/14 3:19:11.298773 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C78653D6A0 synchronization task has been failed!
  1079. 12/14 3:19:11.298932 54f4 func: >>> CHAPartnerNode::SetSyncStatus
  1080. 12/14 3:19:11.298960 54f4 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  1081. 12/14 3:19:11.298970 54f4 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 3 from 2. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  1082. 12/14 3:19:11.298979 54f4 func: >>> HANode::reportEvent
  1083. 12/14 3:19:11.298995 54f4 func: >>> iScsiServer::sendNotification
  1084. 12/14 3:19:11.299005 54f4 func: >>> CEventDataBase::AddRecord
  1085. 12/14 3:19:11.299012 54f4 EventDB: CEventDataBase::AddRecord: Code 771, severity 3, additional strings: 2
  1086. 12/14 3:19:11.299106 54f4 func: <<< CEventDataBase::AddRecord
  1087. 12/14 3:19:11.299113 54f4 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 771).
  1088. 12/14 3:19:11.299767 54f4 Srv: iScsiServer::sendNotification: Send reaction finish.
  1089. 12/14 3:19:11.299787 54f4 func: <<< iScsiServer::sendNotification
  1090. 12/14 3:19:11.299795 54f4 func: <<< HANode::reportEvent
  1091. 12/14 3:19:11.299802 54f4 func: <<< CHAPartnerNode::SetSyncStatus
  1092. 12/14 3:19:11.318597 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (16 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 12798642, expcmdsn = 12798639, maxcmdsn = 12798879.
  1093. 12/14 3:19:11.318646 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 12798628, itt = 12798715)
  1094. 12/14 3:19:11.318659 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 12798629, itt = 12798716)
  1095. 12/14 3:19:11.318670 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 12798630, itt = 12798717)
  1096. 12/14 3:19:11.318680 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 12798631, itt = 12798718)
  1097. 12/14 3:19:11.318690 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 12798632, itt = 12798719)
  1098. 12/14 3:19:11.318700 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 12798633, itt = 12798720)
  1099. 12/14 3:19:11.318709 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 12798634, itt = 12798721)
  1100. 12/14 3:19:11.318719 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 12798635, itt = 12798722)
  1101. 12/14 3:19:11.318730 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EC4F39D8 has failed due to unidentified error.
  1102. 12/14 3:19:11.318740 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1103. 12/14 3:19:11.318747 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F0799A00 synchronization task has been failed!
  1104. 12/14 3:19:11.318768 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8568 has failed due to unidentified error.
  1105. 12/14 3:19:11.318777 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1106. 12/14 3:19:11.318784 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F0799160 synchronization task has been failed!
  1107. 12/14 3:19:11.318796 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7BE8 has failed due to unidentified error.
  1108. 12/14 3:19:11.318804 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1109. 12/14 3:19:11.318818 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F0798300 synchronization task has been failed!
  1110. 12/14 3:19:11.318835 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86994E8 has failed due to unidentified error.
  1111. 12/14 3:19:11.318844 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1112. 12/14 3:19:11.318851 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F0799440 synchronization task has been failed!
  1113. 12/14 3:19:11.318864 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BF88 has failed due to unidentified error.
  1114. 12/14 3:19:11.318873 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1115. 12/14 3:19:11.318881 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F0799720 synchronization task has been failed!
  1116. 12/14 3:19:11.318895 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6758 has failed due to unidentified error.
  1117. 12/14 3:19:11.318903 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1118. 12/14 3:19:11.318909 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F0798E80 synchronization task has been failed!
  1119. 12/14 3:19:11.318923 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D85E8 has failed due to unidentified error.
  1120. 12/14 3:19:11.318931 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1121. 12/14 3:19:11.318938 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F0799CE0 synchronization task has been failed!
  1122. 12/14 3:19:11.318950 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86991E8 has failed due to unidentified error.
  1123. 12/14 3:19:11.318959 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1124. 12/14 3:19:11.318966 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C78653A2E0 synchronization task has been failed!
  1125. 12/14 3:19:11.335687 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (16 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 12798642, expcmdsn = 12798639, maxcmdsn = 12798879.
  1126. 12/14 3:19:11.335729 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 12798636, itt = 12798723)
  1127. 12/14 3:19:11.335741 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 12798637, itt = 12798724)
  1128. 12/14 3:19:11.335788 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 12798638, itt = 12798725)
  1129. 12/14 3:19:11.335800 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6258 has failed due to unidentified error.
  1130. 12/14 3:19:11.335809 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1131. 12/14 3:19:11.335817 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F07988C0 synchronization task has been failed!
  1132. 12/14 3:19:11.335837 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6158 has failed due to unidentified error.
  1133. 12/14 3:19:11.335845 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1134. 12/14 3:19:11.335856 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C78653A8A0 synchronization task has been failed!
  1135. 12/14 3:19:11.335872 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86999E8 has failed due to unidentified error.
  1136. 12/14 3:19:11.335880 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1137. 12/14 3:19:11.335886 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F07985E0 synchronization task has been failed!
  1138. 12/14 3:19:11.351318 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (15 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 12798642, expcmdsn = 12798639, maxcmdsn = 12798879.
  1139. 12/14 3:19:11.351358 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 12798639, itt = 12798726)
  1140. 12/14 3:19:11.351370 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 12798640, itt = 12798727)
  1141. 12/14 3:19:11.351382 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE60D8 has failed due to unidentified error.
  1142. 12/14 3:19:11.351391 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1143. 12/14 3:19:11.351399 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C78653D3C0 synchronization task has been failed!
  1144. 12/14 3:19:11.351416 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BE08 has failed due to unidentified error.
  1145. 12/14 3:19:11.351425 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1146. 12/14 3:19:11.351431 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F0798020 synchronization task has been failed!
  1147. 12/14 3:19:15.931203 54f4 HA: CHAPartnerNode::AfterSynchronization: Event: Full synchronization has been completed (error: 1237)
  1148. 12/14 3:19:15.934930 5378 func: >>> HANode::reportEvent
  1149. 12/14 3:19:15.934975 5378 func: >>> iScsiServer::sendNotification
  1150. 12/14 3:19:15.934991 5378 func: >>> CEventDataBase::AddRecord
  1151. 12/14 3:19:15.935002 5378 EventDB: CEventDataBase::AddRecord: Code 783, severity 1, additional strings: 2
  1152. 12/14 3:19:15.935089 5378 func: <<< CEventDataBase::AddRecord
  1153. 12/14 3:19:15.935098 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 783).
  1154. 12/14 3:19:15.935720 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  1155. 12/14 3:19:15.935742 5378 func: <<< iScsiServer::sendNotification
  1156. 12/14 3:19:15.935750 5378 func: <<< HANode::reportEvent
  1157. 12/14 3:19:15.935756 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_COMPLETE_WITH_ERROR, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'
  1158. 12/14 3:19:15.949410 38d8 func: >>> HANode::onScsiOp_CustomControl_HeartbeatPing
  1159. 12/14 3:19:15.949500 38d8 Common: CStarWindStorageDevice::getLastPerfDegEventElapsedTimeMs: perfDegEventElapsedTime = -1
  1160. 12/14 3:19:15.949508 38d8 func: <<< HANode::onScsiOp_CustomControl_HeartbeatPing
  1161. 12/14 3:19:15.950218 38d8 func: >>> CHAPartnerNode::ShutdownNode
  1162. 12/14 3:19:15.950241 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  1163. 12/14 3:19:15.950250 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  1164. 12/14 3:19:15.950259 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  1165. 12/14 3:19:15.950266 38d8 func: <<< CHAPartnerNode::ShutdownNode
  1166. 12/14 3:19:19.194780 38d8 func: >>> CHAPartnerNode::ShutdownNode
  1167. 12/14 3:19:19.194848 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  1168. 12/14 3:19:19.194873 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  1169. 12/14 3:19:19.194908 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  1170. 12/14 3:19:19.194927 38d8 func: <<< CHAPartnerNode::ShutdownNode
  1171. 12/14 3:19:19.198952 2d00 HA: HANode::onScsiOp_CustomControl_SynchronizationRequest: The partner iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03 has requested the 1 synchronization type.
  1172. 12/14 3:19:19.202505 2d00 HA: CHAPartnerNode::Synchronize: ENTERed with params: p_SyncType = 1, p_ulSnapshotID = 0(0x0), p_pStorageDevice = 0x000002C7ECEDC4B0, p_pSyncBarrier = 0x000002C7ECEDC858, p_puliStartSector = 0x000000D2528FF118 (value is 0x0), p_puliEndSector = 0x000000D2528FF110 (value is 0xAEFFFFF), p_ulIOBlockSizeInSectors = 64, p_ulSectorSizeInUCHARs = 4096, p_bContinue = no, p_bVaaiWriteSameSupport = yes
  1173. 12/14 3:19:19.202546 2d00 HA: HANode::updatePartnerStatus: Statuses is already actual, we don't need to update them.
  1174. 12/14 3:19:19.202568 2d00 func: >>> CHAPartnerNode::SetSyncStatus
  1175. 12/14 3:19:19.202583 2d00 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  1176. 12/14 3:19:19.202598 2d00 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 2 from 3. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  1177. 12/14 3:19:19.202612 2d00 func: >>> HANode::reportEvent
  1178. 12/14 3:19:19.202628 2d00 func: >>> iScsiServer::sendNotification
  1179. 12/14 3:19:19.202642 2d00 func: >>> CEventDataBase::AddRecord
  1180. 12/14 3:19:19.202656 2d00 EventDB: CEventDataBase::AddRecord: Code 770, severity 2, additional strings: 2
  1181. 12/14 3:19:19.202722 2d00 func: <<< CEventDataBase::AddRecord
  1182. 12/14 3:19:19.202736 2d00 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 770).
  1183. 12/14 3:19:19.203386 2d00 Srv: iScsiServer::sendNotification: Send reaction finish.
  1184. 12/14 3:19:19.203417 2d00 func: <<< iScsiServer::sendNotification
  1185. 12/14 3:19:19.203431 2d00 func: <<< HANode::reportEvent
  1186. 12/14 3:19:19.203445 2d00 func: <<< CHAPartnerNode::SetSyncStatus
  1187. 12/14 3:19:19.203851 b3c func: >>> CHAPartnerNode::SetSyncStatus
  1188. 12/14 3:19:19.203891 b3c HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  1189. 12/14 3:19:19.203909 b3c func: <<< CHAPartnerNode::SetSyncStatus
  1190. 12/14 3:19:19.206900 5378 func: >>> HANode::reportEvent
  1191. 12/14 3:19:19.206931 5378 func: >>> iScsiServer::sendNotification
  1192. 12/14 3:19:19.206940 5378 func: >>> CEventDataBase::AddRecord
  1193. 12/14 3:19:19.206947 5378 EventDB: CEventDataBase::AddRecord: Code 780, severity 1, additional strings: 3
  1194. 12/14 3:19:19.206984 5378 func: <<< CEventDataBase::AddRecord
  1195. 12/14 3:19:19.206992 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 780).
  1196. 12/14 3:19:19.207589 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  1197. 12/14 3:19:19.207613 5378 func: <<< iScsiServer::sendNotification
  1198. 12/14 3:19:19.207621 5378 func: <<< HANode::reportEvent
  1199. 12/14 3:19:19.207629 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_STARTED, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03', Synchronization type: 'Full'
  1200. 12/14 3:41:55.038579 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (9937 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 13258804, expcmdsn = 13258803, maxcmdsn = 13259042.
  1201. 12/14 3:41:55.038648 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 13258788, itt = 13258876)
  1202. 12/14 3:41:55.038678 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BF88 has failed due to unidentified error.
  1203. 12/14 3:41:55.038697 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1204. 12/14 3:41:55.038713 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F0799CE0 synchronization task has been failed!
  1205. 12/14 3:41:55.038792 b3c func: >>> CHAPartnerNode::SetSyncStatus
  1206. 12/14 3:41:55.038829 b3c HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  1207. 12/14 3:41:55.038839 b3c HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 3 from 2. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  1208. 12/14 3:41:55.038848 b3c func: >>> HANode::reportEvent
  1209. 12/14 3:41:55.038866 b3c func: >>> iScsiServer::sendNotification
  1210. 12/14 3:41:55.038933 b3c func: >>> CEventDataBase::AddRecord
  1211. 12/14 3:41:55.038945 b3c EventDB: CEventDataBase::AddRecord: Code 771, severity 3, additional strings: 2
  1212. 12/14 3:41:55.038996 b3c func: <<< CEventDataBase::AddRecord
  1213. 12/14 3:41:55.039003 b3c Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 771).
  1214. 12/14 3:41:55.039796 b3c Srv: iScsiServer::sendNotification: Send reaction finish.
  1215. 12/14 3:41:55.039819 b3c func: <<< iScsiServer::sendNotification
  1216. 12/14 3:41:55.039828 b3c func: <<< HANode::reportEvent
  1217. 12/14 3:41:55.039835 b3c func: <<< CHAPartnerNode::SetSyncStatus
  1218. 12/14 3:41:55.057866 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (16 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 13258805, expcmdsn = 13258803, maxcmdsn = 13259042.
  1219. 12/14 3:41:55.057929 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 13258789, itt = 13258877)
  1220. 12/14 3:41:55.057953 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 13258790, itt = 13258878)
  1221. 12/14 3:41:55.057973 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 13258791, itt = 13258879)
  1222. 12/14 3:41:55.057992 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 13258792, itt = 13258880)
  1223. 12/14 3:41:55.058012 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 13258793, itt = 13258881)
  1224. 12/14 3:41:55.058036 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 13258794, itt = 13258882)
  1225. 12/14 3:41:55.058058 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8268 has failed due to unidentified error.
  1226. 12/14 3:41:55.058077 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1227. 12/14 3:41:55.058092 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F0799440 synchronization task has been failed!
  1228. 12/14 3:41:55.058134 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BE08 has failed due to unidentified error.
  1229. 12/14 3:41:55.058156 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1230. 12/14 3:41:55.058170 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F0799720 synchronization task has been failed!
  1231. 12/14 3:41:55.058207 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86991E8 has failed due to unidentified error.
  1232. 12/14 3:41:55.058230 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1233. 12/14 3:41:55.058243 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F0798300 synchronization task has been failed!
  1234. 12/14 3:41:55.058269 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7BE8 has failed due to unidentified error.
  1235. 12/14 3:41:55.058287 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1236. 12/14 3:41:55.058300 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F07988C0 synchronization task has been failed!
  1237. 12/14 3:41:55.058339 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE60D8 has failed due to unidentified error.
  1238. 12/14 3:41:55.058357 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1239. 12/14 3:41:55.058369 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F0798020 synchronization task has been failed!
  1240. 12/14 3:41:55.058395 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8568 has failed due to unidentified error.
  1241. 12/14 3:41:55.058413 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1242. 12/14 3:41:55.058425 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C78653A8A0 synchronization task has been failed!
  1243. 12/14 3:41:55.074617 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (16 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 13258805, expcmdsn = 13258803, maxcmdsn = 13259042.
  1244. 12/14 3:41:55.074673 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 13258795, itt = 13258883)
  1245. 12/14 3:41:55.074695 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 13258796, itt = 13258884)
  1246. 12/14 3:41:55.074714 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 13258797, itt = 13258885)
  1247. 12/14 3:41:55.074730 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 13258798, itt = 13258886)
  1248. 12/14 3:41:55.074752 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7D68 has failed due to unidentified error.
  1249. 12/14 3:41:55.074770 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1250. 12/14 3:41:55.074784 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F0799160 synchronization task has been failed!
  1251. 12/14 3:41:55.074815 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86993E8 has failed due to unidentified error.
  1252. 12/14 3:41:55.074832 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1253. 12/14 3:41:55.074844 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C78653C840 synchronization task has been failed!
  1254. 12/14 3:41:55.074867 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86994E8 has failed due to unidentified error.
  1255. 12/14 3:41:55.074884 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1256. 12/14 3:41:55.074896 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F0799A00 synchronization task has been failed!
  1257. 12/14 3:41:55.074919 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6158 has failed due to unidentified error.
  1258. 12/14 3:41:55.074935 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1259. 12/14 3:41:55.074948 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F0798BA0 synchronization task has been failed!
  1260. 12/14 3:41:55.090301 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (15 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 13258805, expcmdsn = 13258803, maxcmdsn = 13259042.
  1261. 12/14 3:41:55.090394 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 13258799, itt = 13258887)
  1262. 12/14 3:41:55.090431 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 13258800, itt = 13258888)
  1263. 12/14 3:41:55.090469 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 13258801, itt = 13258889)
  1264. 12/14 3:41:55.090490 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 13258802, itt = 13258890)
  1265. 12/14 3:41:55.090515 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D85E8 has failed due to unidentified error.
  1266. 12/14 3:41:55.090536 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1267. 12/14 3:41:55.090553 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F0798E80 synchronization task has been failed!
  1268. 12/14 3:41:55.090591 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6758 has failed due to unidentified error.
  1269. 12/14 3:41:55.090615 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1270. 12/14 3:41:55.090630 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C78653A2E0 synchronization task has been failed!
  1271. 12/14 3:41:55.090657 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E8699A68 has failed due to unidentified error.
  1272. 12/14 3:41:55.090677 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1273. 12/14 3:41:55.090691 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C78653C280 synchronization task has been failed!
  1274. 12/14 3:41:55.090714 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EC4F39D8 has failed due to unidentified error.
  1275. 12/14 3:41:55.090739 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1276. 12/14 3:41:55.090760 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C78653A5C0 synchronization task has been failed!
  1277. 12/14 3:41:55.106300 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (16 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 13258805, expcmdsn = 13258803, maxcmdsn = 13259042.
  1278. 12/14 3:41:55.106375 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 13258803, itt = 13258891)
  1279. 12/14 3:41:55.106403 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6258 has failed due to unidentified error.
  1280. 12/14 3:41:55.106424 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1281. 12/14 3:41:55.106440 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7F07985E0 synchronization task has been failed!
  1282. 12/14 3:41:59.737141 b3c HA: CHAPartnerNode::AfterSynchronization: Event: Full synchronization has been completed (error: 1237)
  1283. 12/14 3:41:59.739724 5378 func: >>> HANode::reportEvent
  1284. 12/14 3:41:59.739764 5378 func: >>> iScsiServer::sendNotification
  1285. 12/14 3:41:59.739777 5378 func: >>> CEventDataBase::AddRecord
  1286. 12/14 3:41:59.739789 5378 EventDB: CEventDataBase::AddRecord: Code 783, severity 1, additional strings: 2
  1287. 12/14 3:41:59.739846 5378 func: <<< CEventDataBase::AddRecord
  1288. 12/14 3:41:59.739863 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 783).
  1289. 12/14 3:41:59.740451 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  1290. 12/14 3:41:59.740480 5378 func: <<< iScsiServer::sendNotification
  1291. 12/14 3:41:59.740494 5378 func: <<< HANode::reportEvent
  1292. 12/14 3:41:59.740505 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_COMPLETE_WITH_ERROR, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'
  1293. 12/14 3:41:59.746564 38d8 func: >>> HANode::onScsiOp_CustomControl_HeartbeatPing
  1294. 12/14 3:41:59.746738 38d8 Common: CStarWindStorageDevice::getLastPerfDegEventElapsedTimeMs: perfDegEventElapsedTime = -1
  1295. 12/14 3:41:59.746769 38d8 func: <<< HANode::onScsiOp_CustomControl_HeartbeatPing
  1296. 12/14 3:41:59.747605 38d8 func: >>> CHAPartnerNode::ShutdownNode
  1297. 12/14 3:41:59.747633 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  1298. 12/14 3:41:59.747652 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  1299. 12/14 3:41:59.747672 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  1300. 12/14 3:41:59.747701 38d8 func: <<< CHAPartnerNode::ShutdownNode
  1301. 12/14 3:42:01.539281 38d8 func: >>> CHAPartnerNode::ShutdownNode
  1302. 12/14 3:42:01.539324 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  1303. 12/14 3:42:01.539338 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  1304. 12/14 3:42:01.539347 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  1305. 12/14 3:42:01.539353 38d8 func: <<< CHAPartnerNode::ShutdownNode
  1306. 12/14 3:42:01.544167 2d00 HA: HANode::onScsiOp_CustomControl_SynchronizationRequest: The partner iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03 has requested the 1 synchronization type.
  1307. 12/14 3:42:01.546369 2d00 HA: CHAPartnerNode::Synchronize: ENTERed with params: p_SyncType = 1, p_ulSnapshotID = 0(0x0), p_pStorageDevice = 0x000002C7ECEDC4B0, p_pSyncBarrier = 0x000002C7ECEDC858, p_puliStartSector = 0x000000D2528FF118 (value is 0x0), p_puliEndSector = 0x000000D2528FF110 (value is 0xAEFFFFF), p_ulIOBlockSizeInSectors = 64, p_ulSectorSizeInUCHARs = 4096, p_bContinue = no, p_bVaaiWriteSameSupport = yes
  1308. 12/14 3:42:01.546404 2d00 HA: HANode::updatePartnerStatus: Statuses is already actual, we don't need to update them.
  1309. 12/14 3:42:01.546422 2d00 func: >>> CHAPartnerNode::SetSyncStatus
  1310. 12/14 3:42:01.546433 2d00 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  1311. 12/14 3:42:01.546444 2d00 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 2 from 3. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  1312. 12/14 3:42:01.546454 2d00 func: >>> HANode::reportEvent
  1313. 12/14 3:42:01.546466 2d00 func: >>> iScsiServer::sendNotification
  1314. 12/14 3:42:01.546478 2d00 func: >>> CEventDataBase::AddRecord
  1315. 12/14 3:42:01.546492 2d00 EventDB: CEventDataBase::AddRecord: Code 770, severity 2, additional strings: 2
  1316. 12/14 3:42:01.546560 2d00 func: <<< CEventDataBase::AddRecord
  1317. 12/14 3:42:01.546576 2d00 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 770).
  1318. 12/14 3:42:01.547061 2d00 Srv: iScsiServer::sendNotification: Send reaction finish.
  1319. 12/14 3:42:01.547086 2d00 func: <<< iScsiServer::sendNotification
  1320. 12/14 3:42:01.547098 2d00 func: <<< HANode::reportEvent
  1321. 12/14 3:42:01.547108 2d00 func: <<< CHAPartnerNode::SetSyncStatus
  1322. 12/14 3:42:01.547428 ee4 func: >>> CHAPartnerNode::SetSyncStatus
  1323. 12/14 3:42:01.547475 ee4 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  1324. 12/14 3:42:01.547493 ee4 func: <<< CHAPartnerNode::SetSyncStatus
  1325. 12/14 3:42:01.551636 5378 func: >>> HANode::reportEvent
  1326. 12/14 3:42:01.551669 5378 func: >>> iScsiServer::sendNotification
  1327. 12/14 3:42:01.551718 5378 func: >>> CEventDataBase::AddRecord
  1328. 12/14 3:42:01.551726 5378 EventDB: CEventDataBase::AddRecord: Code 780, severity 1, additional strings: 3
  1329. 12/14 3:42:01.551763 5378 func: <<< CEventDataBase::AddRecord
  1330. 12/14 3:42:01.551769 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 780).
  1331. 12/14 3:42:01.552394 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  1332. 12/14 3:42:01.552417 5378 func: <<< iScsiServer::sendNotification
  1333. 12/14 3:42:01.552424 5378 func: <<< HANode::reportEvent
  1334. 12/14 3:42:01.552431 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_STARTED, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03', Synchronization type: 'Full'
  1335. 12/14 4:04:39.393013 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (9953 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 13719369, expcmdsn = 13719368, maxcmdsn = 13719607.
  1336. 12/14 4:04:39.393124 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 13719353, itt = 13719441)
  1337. 12/14 4:04:39.393139 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 13719354, itt = 13719442)
  1338. 12/14 4:04:39.393150 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 13719355, itt = 13719443)
  1339. 12/14 4:04:39.393160 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 13719356, itt = 13719444)
  1340. 12/14 4:04:39.393169 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 13719357, itt = 13719445)
  1341. 12/14 4:04:39.393250 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86999E8 has failed due to unidentified error.
  1342. 12/14 4:04:39.393308 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1343. 12/14 4:04:39.393321 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EB5717A0 synchronization task has been failed!
  1344. 12/14 4:04:39.393347 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7D68 has failed due to unidentified error.
  1345. 12/14 4:04:39.393358 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1346. 12/14 4:04:39.393366 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EB570C20 synchronization task has been failed!
  1347. 12/14 4:04:39.393380 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6158 has failed due to unidentified error.
  1348. 12/14 4:04:39.393395 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1349. 12/14 4:04:39.393378 ee4 func: >>> CHAPartnerNode::SetSyncStatus
  1350. 12/14 4:04:39.393429 ee4 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  1351. 12/14 4:04:39.393436 ee4 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 3 from 2. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  1352. 12/14 4:04:39.393403 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EB570660 synchronization task has been failed!
  1353. 12/14 4:04:39.393441 ee4 func: >>> HANode::reportEvent
  1354. 12/14 4:04:39.393466 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8568 has failed due to unidentified error.
  1355. 12/14 4:04:39.393485 ee4 func: >>> iScsiServer::sendNotification
  1356. 12/14 4:04:39.393486 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1357. 12/14 4:04:39.393504 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C78653B420 synchronization task has been failed!
  1358. 12/14 4:04:39.393494 ee4 func: >>> CEventDataBase::AddRecord
  1359. 12/14 4:04:39.393517 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE60D8 has failed due to unidentified error.
  1360. 12/14 4:04:39.393535 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1361. 12/14 4:04:39.393524 ee4 EventDB: CEventDataBase::AddRecord: Code 771, severity 3, additional strings: 2
  1362. 12/14 4:04:39.393543 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EB5700A0 synchronization task has been failed!
  1363. 12/14 4:04:39.393639 ee4 func: <<< CEventDataBase::AddRecord
  1364. 12/14 4:04:39.393647 ee4 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 771).
  1365. 12/14 4:04:39.394196 ee4 Srv: iScsiServer::sendNotification: Send reaction finish.
  1366. 12/14 4:04:39.394212 ee4 func: <<< iScsiServer::sendNotification
  1367. 12/14 4:04:39.394217 ee4 func: <<< HANode::reportEvent
  1368. 12/14 4:04:39.394222 ee4 func: <<< CHAPartnerNode::SetSyncStatus
  1369. 12/14 4:04:39.411813 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (15 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 13719370, expcmdsn = 13719368, maxcmdsn = 13719607.
  1370. 12/14 4:04:39.411841 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 13719358, itt = 13719446)
  1371. 12/14 4:04:39.411850 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 13719359, itt = 13719447)
  1372. 12/14 4:04:39.411860 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 13719360, itt = 13719448)
  1373. 12/14 4:04:39.411870 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E8699A68 has failed due to unidentified error.
  1374. 12/14 4:04:39.411877 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1375. 12/14 4:04:39.411883 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EB5714C0 synchronization task has been failed!
  1376. 12/14 4:04:39.411901 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BF88 has failed due to unidentified error.
  1377. 12/14 4:04:39.411908 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1378. 12/14 4:04:39.411913 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C78653D0E0 synchronization task has been failed!
  1379. 12/14 4:04:39.411921 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86991E8 has failed due to unidentified error.
  1380. 12/14 4:04:39.411927 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1381. 12/14 4:04:39.411932 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C78653CE00 synchronization task has been failed!
  1382. 12/14 4:04:39.427861 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (16 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 13719370, expcmdsn = 13719368, maxcmdsn = 13719607.
  1383. 12/14 4:04:39.427916 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 13719361, itt = 13719449)
  1384. 12/14 4:04:39.427935 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 13719362, itt = 13719450)
  1385. 12/14 4:04:39.427951 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 13719363, itt = 13719451)
  1386. 12/14 4:04:39.427965 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 13719364, itt = 13719452)
  1387. 12/14 4:04:39.427981 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 13719365, itt = 13719453)
  1388. 12/14 4:04:39.428002 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 13719366, itt = 13719454)
  1389. 12/14 4:04:39.428019 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D85E8 has failed due to unidentified error.
  1390. 12/14 4:04:39.428034 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1391. 12/14 4:04:39.428046 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EB571A80 synchronization task has been failed!
  1392. 12/14 4:04:39.428075 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8268 has failed due to unidentified error.
  1393. 12/14 4:04:39.428102 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1394. 12/14 4:04:39.428112 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EB5711E0 synchronization task has been failed!
  1395. 12/14 4:04:39.428133 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BE08 has failed due to unidentified error.
  1396. 12/14 4:04:39.428147 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1397. 12/14 4:04:39.428157 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EB56FDC0 synchronization task has been failed!
  1398. 12/14 4:04:39.428177 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6758 has failed due to unidentified error.
  1399. 12/14 4:04:39.428190 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1400. 12/14 4:04:39.428200 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EB570F00 synchronization task has been failed!
  1401. 12/14 4:04:39.428220 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EC4F39D8 has failed due to unidentified error.
  1402. 12/14 4:04:39.428233 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1403. 12/14 4:04:39.428244 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C78653A5C0 synchronization task has been failed!
  1404. 12/14 4:04:39.428262 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6258 has failed due to unidentified error.
  1405. 12/14 4:04:39.428276 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1406. 12/14 4:04:39.428287 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C78653CB20 synchronization task has been failed!
  1407. 12/14 4:04:39.444008 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (16 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 13719370, expcmdsn = 13719368, maxcmdsn = 13719607.
  1408. 12/14 4:04:39.444044 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 13719367, itt = 13719455)
  1409. 12/14 4:04:39.444053 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 13719368, itt = 13719456)
  1410. 12/14 4:04:39.444063 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86993E8 has failed due to unidentified error.
  1411. 12/14 4:04:39.444071 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1412. 12/14 4:04:39.444077 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EB570940 synchronization task has been failed!
  1413. 12/14 4:04:39.444096 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7BE8 has failed due to unidentified error.
  1414. 12/14 4:04:39.444103 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1415. 12/14 4:04:39.444108 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EB570380 synchronization task has been failed!
  1416. 12/14 4:04:44.083126 ee4 HA: CHAPartnerNode::AfterSynchronization: Event: Full synchronization has been completed (error: 1237)
  1417. 12/14 4:04:44.086383 5378 func: >>> HANode::reportEvent
  1418. 12/14 4:04:44.086418 5378 func: >>> iScsiServer::sendNotification
  1419. 12/14 4:04:44.086428 5378 func: >>> CEventDataBase::AddRecord
  1420. 12/14 4:04:44.086444 5378 EventDB: CEventDataBase::AddRecord: Code 783, severity 1, additional strings: 2
  1421. 12/14 4:04:44.086487 5378 func: <<< CEventDataBase::AddRecord
  1422. 12/14 4:04:44.086495 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 783).
  1423. 12/14 4:04:44.087039 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  1424. 12/14 4:04:44.087065 5378 func: <<< iScsiServer::sendNotification
  1425. 12/14 4:04:44.087073 5378 func: <<< HANode::reportEvent
  1426. 12/14 4:04:44.087081 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_COMPLETE_WITH_ERROR, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'
  1427. 12/14 4:04:44.089836 38d8 func: >>> HANode::onScsiOp_CustomControl_HeartbeatPing
  1428. 12/14 4:04:44.089956 38d8 Common: CStarWindStorageDevice::getLastPerfDegEventElapsedTimeMs: perfDegEventElapsedTime = -1
  1429. 12/14 4:04:44.089972 38d8 func: <<< HANode::onScsiOp_CustomControl_HeartbeatPing
  1430. 12/14 4:04:44.090848 38d8 func: >>> CHAPartnerNode::ShutdownNode
  1431. 12/14 4:04:44.090874 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  1432. 12/14 4:04:44.090888 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  1433. 12/14 4:04:44.090901 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  1434. 12/14 4:04:44.090913 38d8 func: <<< CHAPartnerNode::ShutdownNode
  1435. 12/14 4:04:49.001258 38d8 func: >>> CHAPartnerNode::ShutdownNode
  1436. 12/14 4:04:49.001307 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  1437. 12/14 4:04:49.001321 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  1438. 12/14 4:04:49.001333 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  1439. 12/14 4:04:49.001343 38d8 func: <<< CHAPartnerNode::ShutdownNode
  1440. 12/14 4:04:49.005554 2d00 HA: HANode::onScsiOp_CustomControl_SynchronizationRequest: The partner iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03 has requested the 1 synchronization type.
  1441. 12/14 4:04:49.007088 2d00 HA: CHAPartnerNode::Synchronize: ENTERed with params: p_SyncType = 1, p_ulSnapshotID = 0(0x0), p_pStorageDevice = 0x000002C7ECEDC4B0, p_pSyncBarrier = 0x000002C7ECEDC858, p_puliStartSector = 0x000000D2528FF118 (value is 0x0), p_puliEndSector = 0x000000D2528FF110 (value is 0xAEFFFFF), p_ulIOBlockSizeInSectors = 64, p_ulSectorSizeInUCHARs = 4096, p_bContinue = no, p_bVaaiWriteSameSupport = yes
  1442. 12/14 4:04:49.007131 2d00 HA: HANode::updatePartnerStatus: Statuses is already actual, we don't need to update them.
  1443. 12/14 4:04:49.007155 2d00 func: >>> CHAPartnerNode::SetSyncStatus
  1444. 12/14 4:04:49.007170 2d00 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  1445. 12/14 4:04:49.007185 2d00 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 2 from 3. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  1446. 12/14 4:04:49.007200 2d00 func: >>> HANode::reportEvent
  1447. 12/14 4:04:49.007216 2d00 func: >>> iScsiServer::sendNotification
  1448. 12/14 4:04:49.007332 2d00 func: >>> CEventDataBase::AddRecord
  1449. 12/14 4:04:49.007349 2d00 EventDB: CEventDataBase::AddRecord: Code 770, severity 2, additional strings: 2
  1450. 12/14 4:04:49.007410 2d00 func: <<< CEventDataBase::AddRecord
  1451. 12/14 4:04:49.007424 2d00 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 770).
  1452. 12/14 4:04:49.008100 2d00 Srv: iScsiServer::sendNotification: Send reaction finish.
  1453. 12/14 4:04:49.008131 2d00 func: <<< iScsiServer::sendNotification
  1454. 12/14 4:04:49.008146 2d00 func: <<< HANode::reportEvent
  1455. 12/14 4:04:49.008159 2d00 func: <<< CHAPartnerNode::SetSyncStatus
  1456. 12/14 4:04:49.008419 28f0 func: >>> CHAPartnerNode::SetSyncStatus
  1457. 12/14 4:04:49.008448 28f0 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  1458. 12/14 4:04:49.008459 28f0 func: <<< CHAPartnerNode::SetSyncStatus
  1459. 12/14 4:04:49.011174 5378 func: >>> HANode::reportEvent
  1460. 12/14 4:04:49.011206 5378 func: >>> iScsiServer::sendNotification
  1461. 12/14 4:04:49.011215 5378 func: >>> CEventDataBase::AddRecord
  1462. 12/14 4:04:49.011222 5378 EventDB: CEventDataBase::AddRecord: Code 780, severity 1, additional strings: 3
  1463. 12/14 4:04:49.011263 5378 func: <<< CEventDataBase::AddRecord
  1464. 12/14 4:04:49.011270 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 780).
  1465. 12/14 4:04:49.011744 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  1466. 12/14 4:04:49.011763 5378 func: <<< iScsiServer::sendNotification
  1467. 12/14 4:04:49.011771 5378 func: <<< HANode::reportEvent
  1468. 12/14 4:04:49.011777 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_STARTED, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03', Synchronization type: 'Full'
  1469. 12/14 4:27:20.395873 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (9969 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 14178109, expcmdsn = 14178108, maxcmdsn = 14178347.
  1470. 12/14 4:27:20.395978 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 14178093, itt = 14178182)
  1471. 12/14 4:27:20.396006 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 14178094, itt = 14178183)
  1472. 12/14 4:27:20.396029 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 14178095, itt = 14178184)
  1473. 12/14 4:27:20.396056 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86991E8 has failed due to unidentified error.
  1474. 12/14 4:27:20.396077 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1475. 12/14 4:27:20.396095 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EB5717A0 synchronization task has been failed!
  1476. 12/14 4:27:20.396139 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E8699A68 has failed due to unidentified error.
  1477. 12/14 4:27:20.396158 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1478. 12/14 4:27:20.396173 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EB570F00 synchronization task has been failed!
  1479. 12/14 4:27:20.396197 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7BE8 has failed due to unidentified error.
  1480. 12/14 4:27:20.396197 28f0 func: >>> CHAPartnerNode::SetSyncStatus
  1481. 12/14 4:27:20.396291 28f0 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  1482. 12/14 4:27:20.396324 28f0 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 3 from 2. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  1483. 12/14 4:27:20.396353 28f0 func: >>> HANode::reportEvent
  1484. 12/14 4:27:20.396217 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1485. 12/14 4:27:20.396396 28f0 func: >>> iScsiServer::sendNotification
  1486. 12/14 4:27:20.396434 28f0 func: >>> CEventDataBase::AddRecord
  1487. 12/14 4:27:20.396409 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EB570940 synchronization task has been failed!
  1488. 12/14 4:27:20.396453 28f0 EventDB: CEventDataBase::AddRecord: Code 771, severity 3, additional strings: 2
  1489. 12/14 4:27:20.396619 28f0 func: <<< CEventDataBase::AddRecord
  1490. 12/14 4:27:20.396644 28f0 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 771).
  1491. 12/14 4:27:20.398133 28f0 Srv: iScsiServer::sendNotification: Send reaction finish.
  1492. 12/14 4:27:20.398181 28f0 func: <<< iScsiServer::sendNotification
  1493. 12/14 4:27:20.398202 28f0 func: <<< HANode::reportEvent
  1494. 12/14 4:27:20.398230 28f0 func: <<< CHAPartnerNode::SetSyncStatus
  1495. 12/14 4:27:20.399789 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 14178096, itt = 14178185)
  1496. 12/14 4:27:20.399879 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 14178097, itt = 14178186)
  1497. 12/14 4:27:20.399907 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 14178098, itt = 14178187)
  1498. 12/14 4:27:20.399956 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 14178099, itt = 14178188)
  1499. 12/14 4:27:20.399978 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 14178100, itt = 14178189)
  1500. 12/14 4:27:20.399997 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 14178101, itt = 14178190)
  1501. 12/14 4:27:20.400016 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 14178102, itt = 14178191)
  1502. 12/14 4:27:20.400041 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BF88 has failed due to unidentified error.
  1503. 12/14 4:27:20.400062 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1504. 12/14 4:27:20.400077 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C781E95650 synchronization task has been failed!
  1505. 12/14 4:27:20.400115 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE60D8 has failed due to unidentified error.
  1506. 12/14 4:27:20.400135 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1507. 12/14 4:27:20.400149 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EB5700A0 synchronization task has been failed!
  1508. 12/14 4:27:20.400177 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8568 has failed due to unidentified error.
  1509. 12/14 4:27:20.400196 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1510. 12/14 4:27:20.400210 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EB571A80 synchronization task has been failed!
  1511. 12/14 4:27:20.400237 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6158 has failed due to unidentified error.
  1512. 12/14 4:27:20.400256 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1513. 12/14 4:27:20.400270 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C781E94230 synchronization task has been failed!
  1514. 12/14 4:27:20.400297 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7D68 has failed due to unidentified error.
  1515. 12/14 4:27:20.400316 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1516. 12/14 4:27:20.400330 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EB5714C0 synchronization task has been failed!
  1517. 12/14 4:27:20.400352 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6258 has failed due to unidentified error.
  1518. 12/14 4:27:20.400371 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1519. 12/14 4:27:20.400385 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EB56FDC0 synchronization task has been failed!
  1520. 12/14 4:27:20.400467 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BE08 has failed due to unidentified error.
  1521. 12/14 4:27:20.400491 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1522. 12/14 4:27:20.400513 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EB570C20 synchronization task has been failed!
  1523. 12/14 4:27:20.416635 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (16 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 14178110, expcmdsn = 14178108, maxcmdsn = 14178347.
  1524. 12/14 4:27:20.416778 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 14178103, itt = 14178192)
  1525. 12/14 4:27:20.416807 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 14178104, itt = 14178193)
  1526. 12/14 4:27:20.416829 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 14178105, itt = 14178194)
  1527. 12/14 4:27:20.416848 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 14178106, itt = 14178195)
  1528. 12/14 4:27:20.416869 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 14178107, itt = 14178196)
  1529. 12/14 4:27:20.416889 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 14178108, itt = 14178197)
  1530. 12/14 4:27:20.416911 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86999E8 has failed due to unidentified error.
  1531. 12/14 4:27:20.416932 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1532. 12/14 4:27:20.416948 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EB570380 synchronization task has been failed!
  1533. 12/14 4:27:20.416987 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D85E8 has failed due to unidentified error.
  1534. 12/14 4:27:20.417008 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1535. 12/14 4:27:20.417022 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C781E961D0 synchronization task has been failed!
  1536. 12/14 4:27:20.417050 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6758 has failed due to unidentified error.
  1537. 12/14 4:27:20.417070 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1538. 12/14 4:27:20.417084 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EB570660 synchronization task has been failed!
  1539. 12/14 4:27:20.417112 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86994E8 has failed due to unidentified error.
  1540. 12/14 4:27:20.417131 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1541. 12/14 4:27:20.417145 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C781E95EF0 synchronization task has been failed!
  1542. 12/14 4:27:20.417252 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86993E8 has failed due to unidentified error.
  1543. 12/14 4:27:20.417345 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1544. 12/14 4:27:20.417364 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C781E93F50 synchronization task has been failed!
  1545. 12/14 4:27:20.417462 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8268 has failed due to unidentified error.
  1546. 12/14 4:27:20.417490 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1547. 12/14 4:27:20.417591 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EB5711E0 synchronization task has been failed!
  1548. 12/14 4:27:25.273427 28f0 HA: CHAPartnerNode::AfterSynchronization: Event: Full synchronization has been completed (error: 1237)
  1549. 12/14 4:27:25.277340 5378 func: >>> HANode::reportEvent
  1550. 12/14 4:27:25.277377 5378 func: >>> iScsiServer::sendNotification
  1551. 12/14 4:27:25.277388 5378 func: >>> CEventDataBase::AddRecord
  1552. 12/14 4:27:25.277397 5378 EventDB: CEventDataBase::AddRecord: Code 783, severity 1, additional strings: 2
  1553. 12/14 4:27:25.277443 5378 func: <<< CEventDataBase::AddRecord
  1554. 12/14 4:27:25.277453 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 783).
  1555. 12/14 4:27:25.278137 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  1556. 12/14 4:27:25.278164 5378 func: <<< iScsiServer::sendNotification
  1557. 12/14 4:27:25.278174 5378 func: <<< HANode::reportEvent
  1558. 12/14 4:27:25.278183 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_COMPLETE_WITH_ERROR, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'
  1559. 12/14 4:27:25.282515 38d8 func: >>> HANode::onScsiOp_CustomControl_HeartbeatPing
  1560. 12/14 4:27:25.282691 38d8 Common: CStarWindStorageDevice::getLastPerfDegEventElapsedTimeMs: perfDegEventElapsedTime = -1
  1561. 12/14 4:27:25.282711 38d8 func: <<< HANode::onScsiOp_CustomControl_HeartbeatPing
  1562. 12/14 4:27:25.284015 38d8 func: >>> CHAPartnerNode::ShutdownNode
  1563. 12/14 4:27:25.284054 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  1564. 12/14 4:27:25.284073 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  1565. 12/14 4:27:25.284093 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  1566. 12/14 4:27:25.284111 38d8 func: <<< CHAPartnerNode::ShutdownNode
  1567. 12/14 4:27:26.373327 38d8 func: >>> CHAPartnerNode::ShutdownNode
  1568. 12/14 4:27:26.373411 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  1569. 12/14 4:27:26.373435 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  1570. 12/14 4:27:26.373455 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  1571. 12/14 4:27:26.373473 38d8 func: <<< CHAPartnerNode::ShutdownNode
  1572. 12/14 4:27:26.378399 2d00 HA: HANode::onScsiOp_CustomControl_SynchronizationRequest: The partner iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03 has requested the 1 synchronization type.
  1573. 12/14 4:27:26.380279 2d00 HA: CHAPartnerNode::Synchronize: ENTERed with params: p_SyncType = 1, p_ulSnapshotID = 0(0x0), p_pStorageDevice = 0x000002C7ECEDC4B0, p_pSyncBarrier = 0x000002C7ECEDC858, p_puliStartSector = 0x000000D2528FF118 (value is 0x0), p_puliEndSector = 0x000000D2528FF110 (value is 0xAEFFFFF), p_ulIOBlockSizeInSectors = 64, p_ulSectorSizeInUCHARs = 4096, p_bContinue = no, p_bVaaiWriteSameSupport = yes
  1574. 12/14 4:27:26.380331 2d00 HA: HANode::updatePartnerStatus: Statuses is already actual, we don't need to update them.
  1575. 12/14 4:27:26.380362 2d00 func: >>> CHAPartnerNode::SetSyncStatus
  1576. 12/14 4:27:26.380381 2d00 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  1577. 12/14 4:27:26.380400 2d00 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 2 from 3. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  1578. 12/14 4:27:26.380420 2d00 func: >>> HANode::reportEvent
  1579. 12/14 4:27:26.380441 2d00 func: >>> iScsiServer::sendNotification
  1580. 12/14 4:27:26.380459 2d00 func: >>> CEventDataBase::AddRecord
  1581. 12/14 4:27:26.380477 2d00 EventDB: CEventDataBase::AddRecord: Code 770, severity 2, additional strings: 2
  1582. 12/14 4:27:26.380551 2d00 func: <<< CEventDataBase::AddRecord
  1583. 12/14 4:27:26.380569 2d00 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 770).
  1584. 12/14 4:27:26.381365 2d00 Srv: iScsiServer::sendNotification: Send reaction finish.
  1585. 12/14 4:27:26.381405 2d00 func: <<< iScsiServer::sendNotification
  1586. 12/14 4:27:26.381424 2d00 func: <<< HANode::reportEvent
  1587. 12/14 4:27:26.381443 2d00 func: <<< CHAPartnerNode::SetSyncStatus
  1588. 12/14 4:27:26.381828 6984 func: >>> CHAPartnerNode::SetSyncStatus
  1589. 12/14 4:27:26.381888 6984 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  1590. 12/14 4:27:26.381910 6984 func: <<< CHAPartnerNode::SetSyncStatus
  1591. 12/14 4:27:26.384305 5378 func: >>> HANode::reportEvent
  1592. 12/14 4:27:26.384344 5378 func: >>> iScsiServer::sendNotification
  1593. 12/14 4:27:26.384358 5378 func: >>> CEventDataBase::AddRecord
  1594. 12/14 4:27:26.384366 5378 EventDB: CEventDataBase::AddRecord: Code 780, severity 1, additional strings: 3
  1595. 12/14 4:27:26.384402 5378 func: <<< CEventDataBase::AddRecord
  1596. 12/14 4:27:26.384409 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 780).
  1597. 12/14 4:27:26.384843 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  1598. 12/14 4:27:26.384862 5378 func: <<< iScsiServer::sendNotification
  1599. 12/14 4:27:26.384870 5378 func: <<< HANode::reportEvent
  1600. 12/14 4:27:26.384876 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_STARTED, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03', Synchronization type: 'Full'
  1601. 12/14 4:50:01.923557 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (9953 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 14638181, expcmdsn = 14638180, maxcmdsn = 14638419.
  1602. 12/14 4:50:01.923639 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 14638165, itt = 14638254)
  1603. 12/14 4:50:01.923650 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 14638166, itt = 14638255)
  1604. 12/14 4:50:01.923659 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 14638167, itt = 14638256)
  1605. 12/14 4:50:01.923665 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 14638168, itt = 14638257)
  1606. 12/14 4:50:01.923678 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86999E8 has failed due to unidentified error.
  1607. 12/14 4:50:01.923687 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1608. 12/14 4:50:01.923694 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F95F50 synchronization task has been failed!
  1609. 12/14 4:50:01.923714 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8568 has failed due to unidentified error.
  1610. 12/14 4:50:01.923721 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1611. 12/14 4:50:01.923726 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97930 synchronization task has been failed!
  1612. 12/14 4:50:01.923734 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6158 has failed due to unidentified error.
  1613. 12/14 4:50:01.923742 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1614. 12/14 4:50:01.923746 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EB118320 synchronization task has been failed!
  1615. 12/14 4:50:01.923755 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6758 has failed due to unidentified error.
  1616. 12/14 4:50:01.923762 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1617. 12/14 4:50:01.923766 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EB119180 synchronization task has been failed!
  1618. 12/14 4:50:01.923808 6984 func: >>> CHAPartnerNode::SetSyncStatus
  1619. 12/14 4:50:01.923841 6984 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  1620. 12/14 4:50:01.923852 6984 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 3 from 2. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  1621. 12/14 4:50:01.923870 6984 func: >>> HANode::reportEvent
  1622. 12/14 4:50:01.923889 6984 func: >>> iScsiServer::sendNotification
  1623. 12/14 4:50:01.923899 6984 func: >>> CEventDataBase::AddRecord
  1624. 12/14 4:50:01.923909 6984 EventDB: CEventDataBase::AddRecord: Code 771, severity 3, additional strings: 2
  1625. 12/14 4:50:01.924022 6984 func: <<< CEventDataBase::AddRecord
  1626. 12/14 4:50:01.924032 6984 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 771).
  1627. 12/14 4:50:01.924761 6984 Srv: iScsiServer::sendNotification: Send reaction finish.
  1628. 12/14 4:50:01.924785 6984 func: <<< iScsiServer::sendNotification
  1629. 12/14 4:50:01.924795 6984 func: <<< HANode::reportEvent
  1630. 12/14 4:50:01.924804 6984 func: <<< CHAPartnerNode::SetSyncStatus
  1631. 12/14 4:50:01.941841 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (15 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 14638182, expcmdsn = 14638180, maxcmdsn = 14638419.
  1632. 12/14 4:50:01.941885 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 14638169, itt = 14638258)
  1633. 12/14 4:50:01.941898 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 14638170, itt = 14638259)
  1634. 12/14 4:50:01.941908 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 14638171, itt = 14638260)
  1635. 12/14 4:50:01.941922 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE60D8 has failed due to unidentified error.
  1636. 12/14 4:50:01.941932 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1637. 12/14 4:50:01.941940 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EB118600 synchronization task has been failed!
  1638. 12/14 4:50:01.941961 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BF88 has failed due to unidentified error.
  1639. 12/14 4:50:01.941970 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1640. 12/14 4:50:01.941977 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EB119A20 synchronization task has been failed!
  1641. 12/14 4:50:01.941990 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8268 has failed due to unidentified error.
  1642. 12/14 4:50:01.941999 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1643. 12/14 4:50:01.942006 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EB118040 synchronization task has been failed!
  1644. 12/14 4:50:01.957855 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (16 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 14638182, expcmdsn = 14638180, maxcmdsn = 14638419.
  1645. 12/14 4:50:01.957913 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 14638172, itt = 14638261)
  1646. 12/14 4:50:01.957933 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 14638173, itt = 14638262)
  1647. 12/14 4:50:01.957950 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 14638174, itt = 14638263)
  1648. 12/14 4:50:01.957966 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 14638175, itt = 14638264)
  1649. 12/14 4:50:01.957983 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 14638176, itt = 14638265)
  1650. 12/14 4:50:01.958001 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 14638177, itt = 14638266)
  1651. 12/14 4:50:01.958017 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 14638178, itt = 14638267)
  1652. 12/14 4:50:01.958044 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 14638179, itt = 14638268)
  1653. 12/14 4:50:01.958063 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7D68 has failed due to unidentified error.
  1654. 12/14 4:50:01.958079 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1655. 12/14 4:50:01.958092 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97C10 synchronization task has been failed!
  1656. 12/14 4:50:01.958124 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86994E8 has failed due to unidentified error.
  1657. 12/14 4:50:01.958140 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1658. 12/14 4:50:01.958156 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F98A70 synchronization task has been failed!
  1659. 12/14 4:50:01.958178 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EC4F39D8 has failed due to unidentified error.
  1660. 12/14 4:50:01.958194 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1661. 12/14 4:50:01.958205 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EB117D60 synchronization task has been failed!
  1662. 12/14 4:50:01.958224 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E8699A68 has failed due to unidentified error.
  1663. 12/14 4:50:01.958239 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1664. 12/14 4:50:01.958250 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EB119740 synchronization task has been failed!
  1665. 12/14 4:50:01.958269 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7BE8 has failed due to unidentified error.
  1666. 12/14 4:50:01.958284 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1667. 12/14 4:50:01.958295 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EB1188E0 synchronization task has been failed!
  1668. 12/14 4:50:01.958318 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BE08 has failed due to unidentified error.
  1669. 12/14 4:50:01.958333 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1670. 12/14 4:50:01.958345 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EB118EA0 synchronization task has been failed!
  1671. 12/14 4:50:01.958367 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6258 has failed due to unidentified error.
  1672. 12/14 4:50:01.958382 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1673. 12/14 4:50:01.958393 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EB118BC0 synchronization task has been failed!
  1674. 12/14 4:50:01.958415 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86993E8 has failed due to unidentified error.
  1675. 12/14 4:50:01.958431 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1676. 12/14 4:50:01.958442 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96AD0 synchronization task has been failed!
  1677. 12/14 4:50:01.973893 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (16 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 14638182, expcmdsn = 14638180, maxcmdsn = 14638419.
  1678. 12/14 4:50:01.973980 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 14638180, itt = 14638269)
  1679. 12/14 4:50:01.974008 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D85E8 has failed due to unidentified error.
  1680. 12/14 4:50:01.974031 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1681. 12/14 4:50:01.974048 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EB119460 synchronization task has been failed!
  1682. 12/14 4:50:06.677650 38d8 func: >>> HANode::onScsiOp_CustomControl_HeartbeatPing
  1683. 12/14 4:50:06.677743 38d8 Common: CStarWindStorageDevice::getLastPerfDegEventElapsedTimeMs: perfDegEventElapsedTime = -1
  1684. 12/14 4:50:06.677751 38d8 func: <<< HANode::onScsiOp_CustomControl_HeartbeatPing
  1685. 12/14 4:50:06.678918 38d8 func: >>> CHAPartnerNode::ShutdownNode
  1686. 12/14 4:50:06.678943 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  1687. 12/14 4:50:06.678950 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  1688. 12/14 4:50:06.678957 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  1689. 12/14 4:50:06.678963 38d8 func: <<< CHAPartnerNode::ShutdownNode
  1690. 12/14 4:50:06.680435 6984 HA: CHAPartnerNode::AfterSynchronization: Event: Full synchronization has been completed (error: 1237)
  1691. 12/14 4:50:06.684968 5378 Common: *** sw_common::Sw_Disk_Header::backup: Asynchronous header 'C:\Program Files\StarWind Software\StarWind\headers\datastore03\datastore03_HA.swdsk' operation in progress!
  1692. 12/14 4:50:06.685009 5378 func: >>> HANode::reportEvent
  1693. 12/14 4:50:06.685029 5378 func: >>> iScsiServer::sendNotification
  1694. 12/14 4:50:06.685045 5378 func: >>> CEventDataBase::AddRecord
  1695. 12/14 4:50:06.685060 5378 EventDB: CEventDataBase::AddRecord: Code 783, severity 1, additional strings: 2
  1696. 12/14 4:50:06.685122 5378 func: <<< CEventDataBase::AddRecord
  1697. 12/14 4:50:06.685133 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 783).
  1698. 12/14 4:50:06.685741 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  1699. 12/14 4:50:06.685776 5378 func: <<< iScsiServer::sendNotification
  1700. 12/14 4:50:06.685791 5378 func: <<< HANode::reportEvent
  1701. 12/14 4:50:06.685801 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_COMPLETE_WITH_ERROR, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'
  1702. 12/14 4:50:08.692315 38d8 func: >>> CHAPartnerNode::ShutdownNode
  1703. 12/14 4:50:08.692390 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  1704. 12/14 4:50:08.692417 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  1705. 12/14 4:50:08.692440 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  1706. 12/14 4:50:08.692461 38d8 func: <<< CHAPartnerNode::ShutdownNode
  1707. 12/14 4:50:08.697672 2d00 HA: HANode::onScsiOp_CustomControl_SynchronizationRequest: The partner iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03 has requested the 1 synchronization type.
  1708. 12/14 4:50:08.700009 2d00 HA: CHAPartnerNode::Synchronize: ENTERed with params: p_SyncType = 1, p_ulSnapshotID = 0(0x0), p_pStorageDevice = 0x000002C7ECEDC4B0, p_pSyncBarrier = 0x000002C7ECEDC858, p_puliStartSector = 0x000000D2528FF118 (value is 0x0), p_puliEndSector = 0x000000D2528FF110 (value is 0xAEFFFFF), p_ulIOBlockSizeInSectors = 64, p_ulSectorSizeInUCHARs = 4096, p_bContinue = no, p_bVaaiWriteSameSupport = yes
  1709. 12/14 4:50:08.700066 2d00 HA: HANode::updatePartnerStatus: Statuses is already actual, we don't need to update them.
  1710. 12/14 4:50:08.700094 2d00 func: >>> CHAPartnerNode::SetSyncStatus
  1711. 12/14 4:50:08.700114 2d00 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  1712. 12/14 4:50:08.700133 2d00 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 2 from 3. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  1713. 12/14 4:50:08.700211 2d00 func: >>> HANode::reportEvent
  1714. 12/14 4:50:08.700234 2d00 func: >>> iScsiServer::sendNotification
  1715. 12/14 4:50:08.700253 2d00 func: >>> CEventDataBase::AddRecord
  1716. 12/14 4:50:08.700271 2d00 EventDB: CEventDataBase::AddRecord: Code 770, severity 2, additional strings: 2
  1717. 12/14 4:50:08.700347 2d00 func: <<< CEventDataBase::AddRecord
  1718. 12/14 4:50:08.700365 2d00 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 770).
  1719. 12/14 4:50:08.701131 2d00 Srv: iScsiServer::sendNotification: Send reaction finish.
  1720. 12/14 4:50:08.701213 2d00 func: <<< iScsiServer::sendNotification
  1721. 12/14 4:50:08.701233 2d00 func: <<< HANode::reportEvent
  1722. 12/14 4:50:08.701251 2d00 func: <<< CHAPartnerNode::SetSyncStatus
  1723. 12/14 4:50:08.701854 6130 func: >>> CHAPartnerNode::SetSyncStatus
  1724. 12/14 4:50:08.701911 6130 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  1725. 12/14 4:50:08.701933 6130 func: <<< CHAPartnerNode::SetSyncStatus
  1726. 12/14 4:50:08.704477 5378 func: >>> HANode::reportEvent
  1727. 12/14 4:50:08.704517 5378 func: >>> iScsiServer::sendNotification
  1728. 12/14 4:50:08.704531 5378 func: >>> CEventDataBase::AddRecord
  1729. 12/14 4:50:08.704544 5378 EventDB: CEventDataBase::AddRecord: Code 780, severity 1, additional strings: 3
  1730. 12/14 4:50:08.704597 5378 func: <<< CEventDataBase::AddRecord
  1731. 12/14 4:50:08.704609 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 780).
  1732. 12/14 4:50:08.705122 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  1733. 12/14 4:50:08.705146 5378 func: <<< iScsiServer::sendNotification
  1734. 12/14 4:50:08.705158 5378 func: <<< HANode::reportEvent
  1735. 12/14 4:50:08.705171 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_STARTED, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03', Synchronization type: 'Full'
  1736. 12/14 5:12:43.641903 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (9953 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 15098089, expcmdsn = 15098088, maxcmdsn = 15098327.
  1737. 12/14 5:12:43.641988 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 15098073, itt = 15098162)
  1738. 12/14 5:12:43.642017 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 15098074, itt = 15098163)
  1739. 12/14 5:12:43.642049 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EC4F39D8 has failed due to unidentified error.
  1740. 12/14 5:12:43.642073 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1741. 12/14 5:12:43.642092 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97C10 synchronization task has been failed!
  1742. 12/14 5:12:43.642133 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8268 has failed due to unidentified error.
  1743. 12/14 5:12:43.642155 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1744. 12/14 5:12:43.642172 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96230 synchronization task has been failed!
  1745. 12/14 5:12:43.642186 6130 func: >>> CHAPartnerNode::SetSyncStatus
  1746. 12/14 5:12:43.642252 6130 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  1747. 12/14 5:12:43.642277 6130 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 3 from 2. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  1748. 12/14 5:12:43.642299 6130 func: >>> HANode::reportEvent
  1749. 12/14 5:12:43.642334 6130 func: >>> iScsiServer::sendNotification
  1750. 12/14 5:12:43.642370 6130 func: >>> CEventDataBase::AddRecord
  1751. 12/14 5:12:43.642388 6130 EventDB: CEventDataBase::AddRecord: Code 771, severity 3, additional strings: 2
  1752. 12/14 5:12:43.642466 6130 func: <<< CEventDataBase::AddRecord
  1753. 12/14 5:12:43.642485 6130 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 771).
  1754. 12/14 5:12:43.643670 6130 Srv: iScsiServer::sendNotification: Send reaction finish.
  1755. 12/14 5:12:43.643717 6130 func: <<< iScsiServer::sendNotification
  1756. 12/14 5:12:43.643737 6130 func: <<< HANode::reportEvent
  1757. 12/14 5:12:43.643755 6130 func: <<< CHAPartnerNode::SetSyncStatus
  1758. 12/14 5:12:43.661471 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (16 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 15098090, expcmdsn = 15098088, maxcmdsn = 15098327.
  1759. 12/14 5:12:43.661535 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 15098075, itt = 15098164)
  1760. 12/14 5:12:43.661565 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 15098076, itt = 15098165)
  1761. 12/14 5:12:43.661596 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 15098077, itt = 15098166)
  1762. 12/14 5:12:43.661622 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7D68 has failed due to unidentified error.
  1763. 12/14 5:12:43.661642 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1764. 12/14 5:12:43.661659 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97370 synchronization task has been failed!
  1765. 12/14 5:12:43.661696 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86999E8 has failed due to unidentified error.
  1766. 12/14 5:12:43.661717 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1767. 12/14 5:12:43.661730 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F98790 synchronization task has been failed!
  1768. 12/14 5:12:43.661757 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6158 has failed due to unidentified error.
  1769. 12/14 5:12:43.661777 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1770. 12/14 5:12:43.661790 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97EF0 synchronization task has been failed!
  1771. 12/14 5:12:43.678278 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (16 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 15098090, expcmdsn = 15098088, maxcmdsn = 15098327.
  1772. 12/14 5:12:43.678345 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 15098078, itt = 15098167)
  1773. 12/14 5:12:43.678370 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 15098079, itt = 15098168)
  1774. 12/14 5:12:43.678392 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 15098080, itt = 15098169)
  1775. 12/14 5:12:43.678411 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 15098081, itt = 15098170)
  1776. 12/14 5:12:43.678432 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 15098082, itt = 15098171)
  1777. 12/14 5:12:43.678451 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 15098083, itt = 15098172)
  1778. 12/14 5:12:43.678471 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 15098084, itt = 15098173)
  1779. 12/14 5:12:43.678491 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 15098085, itt = 15098174)
  1780. 12/14 5:12:43.678510 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 15098086, itt = 15098175)
  1781. 12/14 5:12:43.678544 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 15098087, itt = 15098176)
  1782. 12/14 5:12:43.678564 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 15098088, itt = 15098177)
  1783. 12/14 5:12:43.678588 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86993E8 has failed due to unidentified error.
  1784. 12/14 5:12:43.678609 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1785. 12/14 5:12:43.678625 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96510 synchronization task has been failed!
  1786. 12/14 5:12:43.678665 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86991E8 has failed due to unidentified error.
  1787. 12/14 5:12:43.678685 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1788. 12/14 5:12:43.678699 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F98A70 synchronization task has been failed!
  1789. 12/14 5:12:43.678723 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7BE8 has failed due to unidentified error.
  1790. 12/14 5:12:43.678742 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1791. 12/14 5:12:43.678755 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F984B0 synchronization task has been failed!
  1792. 12/14 5:12:43.678782 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BE08 has failed due to unidentified error.
  1793. 12/14 5:12:43.678802 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1794. 12/14 5:12:43.678815 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F953D0 synchronization task has been failed!
  1795. 12/14 5:12:43.678843 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E8699A68 has failed due to unidentified error.
  1796. 12/14 5:12:43.678862 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1797. 12/14 5:12:43.678875 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F95F50 synchronization task has been failed!
  1798. 12/14 5:12:43.678903 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8568 has failed due to unidentified error.
  1799. 12/14 5:12:43.678922 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1800. 12/14 5:12:43.678936 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97650 synchronization task has been failed!
  1801. 12/14 5:12:43.678962 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6758 has failed due to unidentified error.
  1802. 12/14 5:12:43.678981 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1803. 12/14 5:12:43.678995 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97090 synchronization task has been failed!
  1804. 12/14 5:12:43.679021 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BF88 has failed due to unidentified error.
  1805. 12/14 5:12:43.679040 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1806. 12/14 5:12:43.679054 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F95C70 synchronization task has been failed!
  1807. 12/14 5:12:43.679080 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86994E8 has failed due to unidentified error.
  1808. 12/14 5:12:43.679104 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1809. 12/14 5:12:43.679118 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F950F0 synchronization task has been failed!
  1810. 12/14 5:12:43.679144 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D85E8 has failed due to unidentified error.
  1811. 12/14 5:12:43.679163 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1812. 12/14 5:12:43.679177 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F98D50 synchronization task has been failed!
  1813. 12/14 5:12:43.679209 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE60D8 has failed due to unidentified error.
  1814. 12/14 5:12:43.679229 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1815. 12/14 5:12:43.679243 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F956B0 synchronization task has been failed!
  1816. 12/14 5:12:48.464739 6130 HA: CHAPartnerNode::AfterSynchronization: Event: Full synchronization has been completed (error: 1237)
  1817. 12/14 5:12:48.468413 5378 func: >>> HANode::reportEvent
  1818. 12/14 5:12:48.468462 5378 func: >>> iScsiServer::sendNotification
  1819. 12/14 5:12:48.468477 5378 func: >>> CEventDataBase::AddRecord
  1820. 12/14 5:12:48.468490 5378 EventDB: CEventDataBase::AddRecord: Code 783, severity 1, additional strings: 2
  1821. 12/14 5:12:48.468550 5378 func: <<< CEventDataBase::AddRecord
  1822. 12/14 5:12:48.468563 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 783).
  1823. 12/14 5:12:48.468785 38d8 func: >>> HANode::onScsiOp_CustomControl_HeartbeatPing
  1824. 12/14 5:12:48.468944 38d8 Common: CStarWindStorageDevice::getLastPerfDegEventElapsedTimeMs: perfDegEventElapsedTime = -1
  1825. 12/14 5:12:48.468961 38d8 func: <<< HANode::onScsiOp_CustomControl_HeartbeatPing
  1826. 12/14 5:12:48.469319 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  1827. 12/14 5:12:48.469355 5378 func: <<< iScsiServer::sendNotification
  1828. 12/14 5:12:48.469370 5378 func: <<< HANode::reportEvent
  1829. 12/14 5:12:48.469383 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_COMPLETE_WITH_ERROR, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'
  1830. 12/14 5:12:48.469767 38d8 func: >>> CHAPartnerNode::ShutdownNode
  1831. 12/14 5:12:48.469808 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  1832. 12/14 5:12:48.469825 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  1833. 12/14 5:12:48.469842 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  1834. 12/14 5:12:48.469857 38d8 func: <<< CHAPartnerNode::ShutdownNode
  1835. 12/14 5:12:50.944984 38d8 func: >>> CHAPartnerNode::ShutdownNode
  1836. 12/14 5:12:50.945057 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  1837. 12/14 5:12:50.945082 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  1838. 12/14 5:12:50.945104 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  1839. 12/14 5:12:50.945123 38d8 func: <<< CHAPartnerNode::ShutdownNode
  1840. 12/14 5:12:50.950166 2d00 HA: HANode::onScsiOp_CustomControl_SynchronizationRequest: The partner iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03 has requested the 1 synchronization type.
  1841. 12/14 5:12:50.952483 2d00 HA: CHAPartnerNode::Synchronize: ENTERed with params: p_SyncType = 1, p_ulSnapshotID = 0(0x0), p_pStorageDevice = 0x000002C7ECEDC4B0, p_pSyncBarrier = 0x000002C7ECEDC858, p_puliStartSector = 0x000000D2528FF118 (value is 0x0), p_puliEndSector = 0x000000D2528FF110 (value is 0xAEFFFFF), p_ulIOBlockSizeInSectors = 64, p_ulSectorSizeInUCHARs = 4096, p_bContinue = no, p_bVaaiWriteSameSupport = yes
  1842. 12/14 5:12:50.952547 2d00 HA: HANode::updatePartnerStatus: Statuses is already actual, we don't need to update them.
  1843. 12/14 5:12:50.952585 2d00 func: >>> CHAPartnerNode::SetSyncStatus
  1844. 12/14 5:12:50.952607 2d00 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  1845. 12/14 5:12:50.952629 2d00 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 2 from 3. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  1846. 12/14 5:12:50.952650 2d00 func: >>> HANode::reportEvent
  1847. 12/14 5:12:50.952672 2d00 func: >>> iScsiServer::sendNotification
  1848. 12/14 5:12:50.952692 2d00 func: >>> CEventDataBase::AddRecord
  1849. 12/14 5:12:50.952712 2d00 EventDB: CEventDataBase::AddRecord: Code 770, severity 2, additional strings: 2
  1850. 12/14 5:12:50.952796 2d00 func: <<< CEventDataBase::AddRecord
  1851. 12/14 5:12:50.952816 2d00 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 770).
  1852. 12/14 5:12:50.953608 2d00 Srv: iScsiServer::sendNotification: Send reaction finish.
  1853. 12/14 5:12:50.953649 2d00 func: <<< iScsiServer::sendNotification
  1854. 12/14 5:12:50.953669 2d00 func: <<< HANode::reportEvent
  1855. 12/14 5:12:50.953687 2d00 func: <<< CHAPartnerNode::SetSyncStatus
  1856. 12/14 5:12:50.954076 5d98 func: >>> CHAPartnerNode::SetSyncStatus
  1857. 12/14 5:12:50.954128 5d98 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  1858. 12/14 5:12:50.954149 5d98 func: <<< CHAPartnerNode::SetSyncStatus
  1859. 12/14 5:12:50.957541 5378 func: >>> HANode::reportEvent
  1860. 12/14 5:12:50.957578 5378 func: >>> iScsiServer::sendNotification
  1861. 12/14 5:12:50.957591 5378 func: >>> CEventDataBase::AddRecord
  1862. 12/14 5:12:50.957602 5378 EventDB: CEventDataBase::AddRecord: Code 780, severity 1, additional strings: 3
  1863. 12/14 5:12:50.957651 5378 func: <<< CEventDataBase::AddRecord
  1864. 12/14 5:12:50.957662 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 780).
  1865. 12/14 5:12:50.958336 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  1866. 12/14 5:12:50.958365 5378 func: <<< iScsiServer::sendNotification
  1867. 12/14 5:12:50.958376 5378 func: <<< HANode::reportEvent
  1868. 12/14 5:12:50.958387 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_STARTED, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03', Synchronization type: 'Full'
  1869. 12/14 5:35:25.730370 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (9953 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 15557772, expcmdsn = 15557771, maxcmdsn = 15558010.
  1870. 12/14 5:35:25.730462 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 15557756, itt = 15557845)
  1871. 12/14 5:35:25.730474 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 15557757, itt = 15557846)
  1872. 12/14 5:35:25.730483 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 15557758, itt = 15557847)
  1873. 12/14 5:35:25.730496 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8568 has failed due to unidentified error.
  1874. 12/14 5:35:25.730504 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1875. 12/14 5:35:25.730512 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F950F0 synchronization task has been failed!
  1876. 12/14 5:35:25.730532 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7BE8 has failed due to unidentified error.
  1877. 12/14 5:35:25.730540 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1878. 12/14 5:35:25.730545 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97EF0 synchronization task has been failed!
  1879. 12/14 5:35:25.730565 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BF88 has failed due to unidentified error.
  1880. 12/14 5:35:25.730563 5d98 func: >>> CHAPartnerNode::SetSyncStatus
  1881. 12/14 5:35:25.730607 5d98 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  1882. 12/14 5:35:25.730572 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1883. 12/14 5:35:25.730633 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97930 synchronization task has been failed!
  1884. 12/14 5:35:25.730618 5d98 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 3 from 2. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  1885. 12/14 5:35:25.730654 5d98 func: >>> HANode::reportEvent
  1886. 12/14 5:35:25.730674 5d98 func: >>> iScsiServer::sendNotification
  1887. 12/14 5:35:25.730684 5d98 func: >>> CEventDataBase::AddRecord
  1888. 12/14 5:35:25.730692 5d98 EventDB: CEventDataBase::AddRecord: Code 771, severity 3, additional strings: 2
  1889. 12/14 5:35:25.730809 5d98 func: <<< CEventDataBase::AddRecord
  1890. 12/14 5:35:25.730819 5d98 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 771).
  1891. 12/14 5:35:25.731440 5d98 Srv: iScsiServer::sendNotification: Send reaction finish.
  1892. 12/14 5:35:25.731464 5d98 func: <<< iScsiServer::sendNotification
  1893. 12/14 5:35:25.731475 5d98 func: <<< HANode::reportEvent
  1894. 12/14 5:35:25.731483 5d98 func: <<< CHAPartnerNode::SetSyncStatus
  1895. 12/14 5:35:25.733823 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 15557759, itt = 15557848)
  1896. 12/14 5:35:25.733844 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 15557760, itt = 15557849)
  1897. 12/14 5:35:25.733852 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 15557761, itt = 15557850)
  1898. 12/14 5:35:25.733859 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 15557762, itt = 15557851)
  1899. 12/14 5:35:25.733867 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 15557763, itt = 15557852)
  1900. 12/14 5:35:25.733878 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 15557764, itt = 15557853)
  1901. 12/14 5:35:25.733887 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6158 has failed due to unidentified error.
  1902. 12/14 5:35:25.733895 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1903. 12/14 5:35:25.733901 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F981D0 synchronization task has been failed!
  1904. 12/14 5:35:25.733917 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7D68 has failed due to unidentified error.
  1905. 12/14 5:35:25.733924 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1906. 12/14 5:35:25.733929 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F984B0 synchronization task has been failed!
  1907. 12/14 5:35:25.733940 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8268 has failed due to unidentified error.
  1908. 12/14 5:35:25.733947 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1909. 12/14 5:35:25.733952 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F98790 synchronization task has been failed!
  1910. 12/14 5:35:25.733964 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86994E8 has failed due to unidentified error.
  1911. 12/14 5:35:25.733970 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1912. 12/14 5:35:25.733979 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F98A70 synchronization task has been failed!
  1913. 12/14 5:35:25.733991 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D85E8 has failed due to unidentified error.
  1914. 12/14 5:35:25.733998 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1915. 12/14 5:35:25.734003 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96DB0 synchronization task has been failed!
  1916. 12/14 5:35:25.734013 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BE08 has failed due to unidentified error.
  1917. 12/14 5:35:25.734019 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1918. 12/14 5:35:25.734024 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97C10 synchronization task has been failed!
  1919. 12/14 5:35:25.749375 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (15 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 15557773, expcmdsn = 15557771, maxcmdsn = 15558010.
  1920. 12/14 5:35:25.749420 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 15557765, itt = 15557854)
  1921. 12/14 5:35:25.749435 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 15557766, itt = 15557855)
  1922. 12/14 5:35:25.749448 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 15557767, itt = 15557856)
  1923. 12/14 5:35:25.749459 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 15557768, itt = 15557857)
  1924. 12/14 5:35:25.749476 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86991E8 has failed due to unidentified error.
  1925. 12/14 5:35:25.749492 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1926. 12/14 5:35:25.749502 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F953D0 synchronization task has been failed!
  1927. 12/14 5:35:25.749525 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EC4F39D8 has failed due to unidentified error.
  1928. 12/14 5:35:25.749535 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1929. 12/14 5:35:25.749541 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F967F0 synchronization task has been failed!
  1930. 12/14 5:35:25.749556 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE60D8 has failed due to unidentified error.
  1931. 12/14 5:35:25.749564 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1932. 12/14 5:35:25.749571 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96510 synchronization task has been failed!
  1933. 12/14 5:35:25.749583 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6258 has failed due to unidentified error.
  1934. 12/14 5:35:25.749592 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1935. 12/14 5:35:25.749599 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97090 synchronization task has been failed!
  1936. 12/14 5:35:25.765389 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (16 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 15557773, expcmdsn = 15557771, maxcmdsn = 15558010.
  1937. 12/14 5:35:25.765438 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 15557769, itt = 15557858)
  1938. 12/14 5:35:25.765449 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 15557770, itt = 15557859)
  1939. 12/14 5:35:25.765458 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 15557771, itt = 15557860)
  1940. 12/14 5:35:25.765468 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86993E8 has failed due to unidentified error.
  1941. 12/14 5:35:25.765476 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1942. 12/14 5:35:25.765483 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97370 synchronization task has been failed!
  1943. 12/14 5:35:25.765501 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6758 has failed due to unidentified error.
  1944. 12/14 5:35:25.765508 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1945. 12/14 5:35:25.765513 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F956B0 synchronization task has been failed!
  1946. 12/14 5:35:25.765525 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86999E8 has failed due to unidentified error.
  1947. 12/14 5:35:25.765532 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  1948. 12/14 5:35:25.765537 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97650 synchronization task has been failed!
  1949. 12/14 5:35:30.566813 5d98 HA: CHAPartnerNode::AfterSynchronization: Event: Full synchronization has been completed (error: 1237)
  1950. 12/14 5:35:30.570059 5378 func: >>> HANode::reportEvent
  1951. 12/14 5:35:30.570106 5378 func: >>> iScsiServer::sendNotification
  1952. 12/14 5:35:30.570121 5378 func: >>> CEventDataBase::AddRecord
  1953. 12/14 5:35:30.570135 5378 EventDB: CEventDataBase::AddRecord: Code 783, severity 1, additional strings: 2
  1954. 12/14 5:35:30.570195 5378 func: <<< CEventDataBase::AddRecord
  1955. 12/14 5:35:30.570209 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 783).
  1956. 12/14 5:35:30.571032 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  1957. 12/14 5:35:30.571068 5378 func: <<< iScsiServer::sendNotification
  1958. 12/14 5:35:30.571083 5378 func: <<< HANode::reportEvent
  1959. 12/14 5:35:30.571097 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_COMPLETE_WITH_ERROR, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'
  1960. 12/14 5:35:30.581986 38d8 func: >>> HANode::onScsiOp_CustomControl_HeartbeatPing
  1961. 12/14 5:35:30.582146 38d8 Common: CStarWindStorageDevice::getLastPerfDegEventElapsedTimeMs: perfDegEventElapsedTime = -1
  1962. 12/14 5:35:30.582165 38d8 func: <<< HANode::onScsiOp_CustomControl_HeartbeatPing
  1963. 12/14 5:35:30.583307 38d8 func: >>> CHAPartnerNode::ShutdownNode
  1964. 12/14 5:35:30.583344 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  1965. 12/14 5:35:30.583364 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  1966. 12/14 5:35:30.583385 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  1967. 12/14 5:35:30.583403 38d8 func: <<< CHAPartnerNode::ShutdownNode
  1968. 12/14 5:35:33.328260 38d8 func: >>> CHAPartnerNode::ShutdownNode
  1969. 12/14 5:35:33.328305 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  1970. 12/14 5:35:33.328317 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  1971. 12/14 5:35:33.328328 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  1972. 12/14 5:35:33.328337 38d8 func: <<< CHAPartnerNode::ShutdownNode
  1973. 12/14 5:35:33.333321 2d00 HA: HANode::onScsiOp_CustomControl_SynchronizationRequest: The partner iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03 has requested the 1 synchronization type.
  1974. 12/14 5:35:33.335470 2d00 HA: CHAPartnerNode::Synchronize: ENTERed with params: p_SyncType = 1, p_ulSnapshotID = 0(0x0), p_pStorageDevice = 0x000002C7ECEDC4B0, p_pSyncBarrier = 0x000002C7ECEDC858, p_puliStartSector = 0x000000D2528FF118 (value is 0x0), p_puliEndSector = 0x000000D2528FF110 (value is 0xAEFFFFF), p_ulIOBlockSizeInSectors = 64, p_ulSectorSizeInUCHARs = 4096, p_bContinue = no, p_bVaaiWriteSameSupport = yes
  1975. 12/14 5:35:33.335542 2d00 HA: HANode::updatePartnerStatus: Statuses is already actual, we don't need to update them.
  1976. 12/14 5:35:33.335571 2d00 func: >>> CHAPartnerNode::SetSyncStatus
  1977. 12/14 5:35:33.335590 2d00 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  1978. 12/14 5:35:33.335609 2d00 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 2 from 3. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  1979. 12/14 5:35:33.335627 2d00 func: >>> HANode::reportEvent
  1980. 12/14 5:35:33.335649 2d00 func: >>> iScsiServer::sendNotification
  1981. 12/14 5:35:33.335667 2d00 func: >>> CEventDataBase::AddRecord
  1982. 12/14 5:35:33.335683 2d00 EventDB: CEventDataBase::AddRecord: Code 770, severity 2, additional strings: 2
  1983. 12/14 5:35:33.335764 2d00 func: <<< CEventDataBase::AddRecord
  1984. 12/14 5:35:33.335781 2d00 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 770).
  1985. 12/14 5:35:33.336598 2d00 Srv: iScsiServer::sendNotification: Send reaction finish.
  1986. 12/14 5:35:33.336639 2d00 func: <<< iScsiServer::sendNotification
  1987. 12/14 5:35:33.336658 2d00 func: <<< HANode::reportEvent
  1988. 12/14 5:35:33.336732 2d00 func: <<< CHAPartnerNode::SetSyncStatus
  1989. 12/14 5:35:33.337038 2bc8 func: >>> CHAPartnerNode::SetSyncStatus
  1990. 12/14 5:35:33.337074 2bc8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  1991. 12/14 5:35:33.337084 2bc8 func: <<< CHAPartnerNode::SetSyncStatus
  1992. 12/14 5:35:33.341760 5378 func: >>> HANode::reportEvent
  1993. 12/14 5:35:33.341820 5378 func: >>> iScsiServer::sendNotification
  1994. 12/14 5:35:33.341841 5378 func: >>> CEventDataBase::AddRecord
  1995. 12/14 5:35:33.341860 5378 EventDB: CEventDataBase::AddRecord: Code 780, severity 1, additional strings: 3
  1996. 12/14 5:35:33.341933 5378 func: <<< CEventDataBase::AddRecord
  1997. 12/14 5:35:33.341950 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 780).
  1998. 12/14 5:35:33.342794 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  1999. 12/14 5:35:33.342835 5378 func: <<< iScsiServer::sendNotification
  2000. 12/14 5:35:33.342862 5378 func: <<< HANode::reportEvent
  2001. 12/14 5:35:33.342880 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_STARTED, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03', Synchronization type: 'Full'
  2002. 12/14 5:58:08.420759 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (9953 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 16017670, expcmdsn = 16017667, maxcmdsn = 16017908.
  2003. 12/14 5:58:08.420816 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16017654, itt = 16017743)
  2004. 12/14 5:58:08.420832 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16017655, itt = 16017744)
  2005. 12/14 5:58:08.420842 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16017656, itt = 16017745)
  2006. 12/14 5:58:08.420853 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E8699A68 has failed due to unidentified error.
  2007. 12/14 5:58:08.420860 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2008. 12/14 5:58:08.420867 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F95F50 synchronization task has been failed!
  2009. 12/14 5:58:08.420896 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE60D8 has failed due to unidentified error.
  2010. 12/14 5:58:08.420903 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2011. 12/14 5:58:08.420908 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F981D0 synchronization task has been failed!
  2012. 12/14 5:58:08.420916 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86994E8 has failed due to unidentified error.
  2013. 12/14 5:58:08.420922 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2014. 12/14 5:58:08.420926 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96DB0 synchronization task has been failed!
  2015. 12/14 5:58:08.421090 2bc8 func: >>> CHAPartnerNode::SetSyncStatus
  2016. 12/14 5:58:08.421153 2bc8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  2017. 12/14 5:58:08.421177 2bc8 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 3 from 2. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  2018. 12/14 5:58:08.421199 2bc8 func: >>> HANode::reportEvent
  2019. 12/14 5:58:08.421234 2bc8 func: >>> iScsiServer::sendNotification
  2020. 12/14 5:58:08.421255 2bc8 func: >>> CEventDataBase::AddRecord
  2021. 12/14 5:58:08.421277 2bc8 EventDB: CEventDataBase::AddRecord: Code 771, severity 3, additional strings: 2
  2022. 12/14 5:58:08.421354 2bc8 func: <<< CEventDataBase::AddRecord
  2023. 12/14 5:58:08.421373 2bc8 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 771).
  2024. 12/14 5:58:08.422404 2bc8 Srv: iScsiServer::sendNotification: Send reaction finish.
  2025. 12/14 5:58:08.422447 2bc8 func: <<< iScsiServer::sendNotification
  2026. 12/14 5:58:08.422467 2bc8 func: <<< HANode::reportEvent
  2027. 12/14 5:58:08.422484 2bc8 func: <<< CHAPartnerNode::SetSyncStatus
  2028. 12/14 5:58:08.440734 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (16 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 16017671, expcmdsn = 16017667, maxcmdsn = 16017908.
  2029. 12/14 5:58:08.440775 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16017657, itt = 16017746)
  2030. 12/14 5:58:08.440789 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8268 has failed due to unidentified error.
  2031. 12/14 5:58:08.440796 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2032. 12/14 5:58:08.440803 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F950F0 synchronization task has been failed!
  2033. 12/14 5:58:08.457746 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (16 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 16017671, expcmdsn = 16017667, maxcmdsn = 16017908.
  2034. 12/14 5:58:08.457802 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16017658, itt = 16017747)
  2035. 12/14 5:58:08.457824 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7BE8 has failed due to unidentified error.
  2036. 12/14 5:58:08.457836 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2037. 12/14 5:58:08.457846 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F953D0 synchronization task has been failed!
  2038. 12/14 5:58:08.472803 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (15 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 16017671, expcmdsn = 16017667, maxcmdsn = 16017908.
  2039. 12/14 5:58:08.472889 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16017659, itt = 16017748)
  2040. 12/14 5:58:08.472916 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16017660, itt = 16017749)
  2041. 12/14 5:58:08.472937 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16017661, itt = 16017750)
  2042. 12/14 5:58:08.472958 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16017662, itt = 16017751)
  2043. 12/14 5:58:08.472979 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16017663, itt = 16017752)
  2044. 12/14 5:58:08.472999 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16017664, itt = 16017753)
  2045. 12/14 5:58:08.473018 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16017665, itt = 16017754)
  2046. 12/14 5:58:08.473038 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16017666, itt = 16017755)
  2047. 12/14 5:58:08.473057 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16017667, itt = 16017756)
  2048. 12/14 5:58:08.473076 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16017668, itt = 16017757)
  2049. 12/14 5:58:08.473099 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16017669, itt = 16017758)
  2050. 12/14 5:58:08.473122 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6758 has failed due to unidentified error.
  2051. 12/14 5:58:08.473143 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2052. 12/14 5:58:08.473160 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F956B0 synchronization task has been failed!
  2053. 12/14 5:58:08.473197 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6258 has failed due to unidentified error.
  2054. 12/14 5:58:08.473217 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2055. 12/14 5:58:08.473234 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97370 synchronization task has been failed!
  2056. 12/14 5:58:08.473260 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86999E8 has failed due to unidentified error.
  2057. 12/14 5:58:08.473280 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2058. 12/14 5:58:08.473293 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96510 synchronization task has been failed!
  2059. 12/14 5:58:08.473320 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BE08 has failed due to unidentified error.
  2060. 12/14 5:58:08.473339 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2061. 12/14 5:58:08.473353 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96230 synchronization task has been failed!
  2062. 12/14 5:58:08.473379 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8568 has failed due to unidentified error.
  2063. 12/14 5:58:08.473399 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2064. 12/14 5:58:08.473412 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F984B0 synchronization task has been failed!
  2065. 12/14 5:58:08.473438 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86991E8 has failed due to unidentified error.
  2066. 12/14 5:58:08.473458 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2067. 12/14 5:58:08.473476 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F95990 synchronization task has been failed!
  2068. 12/14 5:58:08.473506 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86993E8 has failed due to unidentified error.
  2069. 12/14 5:58:08.473526 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2070. 12/14 5:58:08.473539 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97EF0 synchronization task has been failed!
  2071. 12/14 5:58:08.473565 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D85E8 has failed due to unidentified error.
  2072. 12/14 5:58:08.473584 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2073. 12/14 5:58:08.473598 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F95C70 synchronization task has been failed!
  2074. 12/14 5:58:08.473623 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EC4F39D8 has failed due to unidentified error.
  2075. 12/14 5:58:08.473643 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2076. 12/14 5:58:08.473656 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F967F0 synchronization task has been failed!
  2077. 12/14 5:58:08.473682 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6158 has failed due to unidentified error.
  2078. 12/14 5:58:08.473710 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2079. 12/14 5:58:08.473728 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97650 synchronization task has been failed!
  2080. 12/14 5:58:08.473755 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BF88 has failed due to unidentified error.
  2081. 12/14 5:58:08.473774 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2082. 12/14 5:58:08.473788 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97090 synchronization task has been failed!
  2083. 12/14 5:58:13.192415 2bc8 HA: CHAPartnerNode::AfterSynchronization: Event: Full synchronization has been completed (error: 1237)
  2084. 12/14 5:58:13.197400 5378 func: >>> HANode::reportEvent
  2085. 12/14 5:58:13.197456 5378 func: >>> iScsiServer::sendNotification
  2086. 12/14 5:58:13.197477 5378 func: >>> CEventDataBase::AddRecord
  2087. 12/14 5:58:13.197493 5378 EventDB: CEventDataBase::AddRecord: Code 783, severity 1, additional strings: 2
  2088. 12/14 5:58:13.197546 38d8 func: >>> HANode::onScsiOp_CustomControl_HeartbeatPing
  2089. 12/14 5:58:13.197579 5378 func: <<< CEventDataBase::AddRecord
  2090. 12/14 5:58:13.197740 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 783).
  2091. 12/14 5:58:13.197862 38d8 Common: CStarWindStorageDevice::getLastPerfDegEventElapsedTimeMs: perfDegEventElapsedTime = -1
  2092. 12/14 5:58:13.197899 38d8 func: <<< HANode::onScsiOp_CustomControl_HeartbeatPing
  2093. 12/14 5:58:13.198759 38d8 func: >>> CHAPartnerNode::ShutdownNode
  2094. 12/14 5:58:13.198808 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  2095. 12/14 5:58:13.198831 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  2096. 12/14 5:58:13.198852 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  2097. 12/14 5:58:13.198879 38d8 func: <<< CHAPartnerNode::ShutdownNode
  2098. 12/14 5:58:13.198857 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  2099. 12/14 5:58:13.198957 5378 func: <<< iScsiServer::sendNotification
  2100. 12/14 5:58:13.198974 5378 func: <<< HANode::reportEvent
  2101. 12/14 5:58:13.199009 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_COMPLETE_WITH_ERROR, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'
  2102. 12/14 5:58:15.688783 38d8 func: >>> CHAPartnerNode::ShutdownNode
  2103. 12/14 5:58:15.688828 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  2104. 12/14 5:58:15.688835 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  2105. 12/14 5:58:15.688843 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  2106. 12/14 5:58:15.688849 38d8 func: <<< CHAPartnerNode::ShutdownNode
  2107. 12/14 5:58:15.693237 2d00 HA: HANode::onScsiOp_CustomControl_SynchronizationRequest: The partner iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03 has requested the 1 synchronization type.
  2108. 12/14 5:58:15.694390 2d00 HA: CHAPartnerNode::Synchronize: ENTERed with params: p_SyncType = 1, p_ulSnapshotID = 0(0x0), p_pStorageDevice = 0x000002C7ECEDC4B0, p_pSyncBarrier = 0x000002C7ECEDC858, p_puliStartSector = 0x000000D2528FF118 (value is 0x0), p_puliEndSector = 0x000000D2528FF110 (value is 0xAEFFFFF), p_ulIOBlockSizeInSectors = 64, p_ulSectorSizeInUCHARs = 4096, p_bContinue = no, p_bVaaiWriteSameSupport = yes
  2109. 12/14 5:58:15.694445 2d00 HA: HANode::updatePartnerStatus: Statuses is already actual, we don't need to update them.
  2110. 12/14 5:58:15.694475 2d00 func: >>> CHAPartnerNode::SetSyncStatus
  2111. 12/14 5:58:15.694496 2d00 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  2112. 12/14 5:58:15.694516 2d00 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 2 from 3. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  2113. 12/14 5:58:15.694542 2d00 func: >>> HANode::reportEvent
  2114. 12/14 5:58:15.694589 2d00 func: >>> iScsiServer::sendNotification
  2115. 12/14 5:58:15.694613 2d00 func: >>> CEventDataBase::AddRecord
  2116. 12/14 5:58:15.694632 2d00 EventDB: CEventDataBase::AddRecord: Code 770, severity 2, additional strings: 2
  2117. 12/14 5:58:15.694708 2d00 func: <<< CEventDataBase::AddRecord
  2118. 12/14 5:58:15.694727 2d00 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 770).
  2119. 12/14 5:58:15.695538 2d00 Srv: iScsiServer::sendNotification: Send reaction finish.
  2120. 12/14 5:58:15.695610 2d00 func: <<< iScsiServer::sendNotification
  2121. 12/14 5:58:15.695630 2d00 func: <<< HANode::reportEvent
  2122. 12/14 5:58:15.695645 2d00 func: <<< CHAPartnerNode::SetSyncStatus
  2123. 12/14 5:58:15.695992 6958 func: >>> CHAPartnerNode::SetSyncStatus
  2124. 12/14 5:58:15.696047 6958 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  2125. 12/14 5:58:15.696069 6958 func: <<< CHAPartnerNode::SetSyncStatus
  2126. 12/14 5:58:15.699151 5378 func: >>> HANode::reportEvent
  2127. 12/14 5:58:15.699214 5378 func: >>> iScsiServer::sendNotification
  2128. 12/14 5:58:15.699227 5378 func: >>> CEventDataBase::AddRecord
  2129. 12/14 5:58:15.699237 5378 EventDB: CEventDataBase::AddRecord: Code 780, severity 1, additional strings: 3
  2130. 12/14 5:58:15.699281 5378 func: <<< CEventDataBase::AddRecord
  2131. 12/14 5:58:15.699290 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 780).
  2132. 12/14 5:58:15.699860 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  2133. 12/14 5:58:15.699885 5378 func: <<< iScsiServer::sendNotification
  2134. 12/14 5:58:15.699896 5378 func: <<< HANode::reportEvent
  2135. 12/14 5:58:15.699906 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_STARTED, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03', Synchronization type: 'Full'
  2136. 12/14 6:20:50.810251 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (9953 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 16477590, expcmdsn = 16477589, maxcmdsn = 16477828.
  2137. 12/14 6:20:50.810431 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16477574, itt = 16477663)
  2138. 12/14 6:20:50.810473 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16477575, itt = 16477664)
  2139. 12/14 6:20:50.810776 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86999E8 has failed due to unidentified error.
  2140. 12/14 6:20:50.810803 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2141. 12/14 6:20:50.810821 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F95F50 synchronization task has been failed!
  2142. 12/14 6:20:50.810860 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8268 has failed due to unidentified error.
  2143. 12/14 6:20:50.810883 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2144. 12/14 6:20:50.810897 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96230 synchronization task has been failed!
  2145. 12/14 6:20:50.810911 6958 func: >>> CHAPartnerNode::SetSyncStatus
  2146. 12/14 6:20:50.810980 6958 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  2147. 12/14 6:20:50.811006 6958 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 3 from 2. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  2148. 12/14 6:20:50.811030 6958 func: >>> HANode::reportEvent
  2149. 12/14 6:20:50.811068 6958 func: >>> iScsiServer::sendNotification
  2150. 12/14 6:20:50.811090 6958 func: >>> CEventDataBase::AddRecord
  2151. 12/14 6:20:50.811113 6958 EventDB: CEventDataBase::AddRecord: Code 771, severity 3, additional strings: 2
  2152. 12/14 6:20:50.811364 6958 func: <<< CEventDataBase::AddRecord
  2153. 12/14 6:20:50.811390 6958 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 771).
  2154. 12/14 6:20:50.812494 6958 Srv: iScsiServer::sendNotification: Send reaction finish.
  2155. 12/14 6:20:50.812541 6958 func: <<< iScsiServer::sendNotification
  2156. 12/14 6:20:50.812561 6958 func: <<< HANode::reportEvent
  2157. 12/14 6:20:50.812579 6958 func: <<< CHAPartnerNode::SetSyncStatus
  2158. 12/14 6:20:50.814190 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16477576, itt = 16477665)
  2159. 12/14 6:20:50.814252 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16477577, itt = 16477666)
  2160. 12/14 6:20:50.814274 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86993E8 has failed due to unidentified error.
  2161. 12/14 6:20:50.814290 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2162. 12/14 6:20:50.814303 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97370 synchronization task has been failed!
  2163. 12/14 6:20:50.814333 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6158 has failed due to unidentified error.
  2164. 12/14 6:20:50.814349 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2165. 12/14 6:20:50.814360 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96510 synchronization task has been failed!
  2166. 12/14 6:20:50.830161 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (15 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 16477591, expcmdsn = 16477589, maxcmdsn = 16477828.
  2167. 12/14 6:20:50.830217 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16477578, itt = 16477667)
  2168. 12/14 6:20:50.830238 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16477579, itt = 16477668)
  2169. 12/14 6:20:50.830256 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16477580, itt = 16477669)
  2170. 12/14 6:20:50.830287 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16477581, itt = 16477670)
  2171. 12/14 6:20:50.830305 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16477582, itt = 16477671)
  2172. 12/14 6:20:50.830330 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86991E8 has failed due to unidentified error.
  2173. 12/14 6:20:50.830388 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2174. 12/14 6:20:50.830402 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F967F0 synchronization task has been failed!
  2175. 12/14 6:20:50.830431 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BF88 has failed due to unidentified error.
  2176. 12/14 6:20:50.830448 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2177. 12/14 6:20:50.830459 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F981D0 synchronization task has been failed!
  2178. 12/14 6:20:50.830478 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6758 has failed due to unidentified error.
  2179. 12/14 6:20:50.830493 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2180. 12/14 6:20:50.830504 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F95990 synchronization task has been failed!
  2181. 12/14 6:20:50.830527 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8568 has failed due to unidentified error.
  2182. 12/14 6:20:50.830542 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2183. 12/14 6:20:50.830553 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F984B0 synchronization task has been failed!
  2184. 12/14 6:20:50.830580 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6258 has failed due to unidentified error.
  2185. 12/14 6:20:50.830595 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2186. 12/14 6:20:50.830606 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F956B0 synchronization task has been failed!
  2187. 12/14 6:20:50.847210 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (16 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 16477591, expcmdsn = 16477589, maxcmdsn = 16477828.
  2188. 12/14 6:20:50.847277 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16477583, itt = 16477672)
  2189. 12/14 6:20:50.847303 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16477584, itt = 16477673)
  2190. 12/14 6:20:50.847327 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16477585, itt = 16477674)
  2191. 12/14 6:20:50.847349 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16477586, itt = 16477675)
  2192. 12/14 6:20:50.847370 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16477587, itt = 16477676)
  2193. 12/14 6:20:50.847394 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16477588, itt = 16477677)
  2194. 12/14 6:20:50.847414 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16477589, itt = 16477678)
  2195. 12/14 6:20:50.847440 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D85E8 has failed due to unidentified error.
  2196. 12/14 6:20:50.847461 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2197. 12/14 6:20:50.847486 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97EF0 synchronization task has been failed!
  2198. 12/14 6:20:50.847531 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE60D8 has failed due to unidentified error.
  2199. 12/14 6:20:50.847552 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2200. 12/14 6:20:50.847567 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97C10 synchronization task has been failed!
  2201. 12/14 6:20:50.847596 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EC4F39D8 has failed due to unidentified error.
  2202. 12/14 6:20:50.847616 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2203. 12/14 6:20:50.847631 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97930 synchronization task has been failed!
  2204. 12/14 6:20:50.847659 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86994E8 has failed due to unidentified error.
  2205. 12/14 6:20:50.847679 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2206. 12/14 6:20:50.847694 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F953D0 synchronization task has been failed!
  2207. 12/14 6:20:50.847728 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BE08 has failed due to unidentified error.
  2208. 12/14 6:20:50.847749 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2209. 12/14 6:20:50.847763 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F98A70 synchronization task has been failed!
  2210. 12/14 6:20:50.847790 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7BE8 has failed due to unidentified error.
  2211. 12/14 6:20:50.847810 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2212. 12/14 6:20:50.847825 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F950F0 synchronization task has been failed!
  2213. 12/14 6:20:50.847849 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E8699A68 has failed due to unidentified error.
  2214. 12/14 6:20:50.847868 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2215. 12/14 6:20:50.847883 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97650 synchronization task has been failed!
  2216. 12/14 6:20:55.565378 6958 HA: CHAPartnerNode::AfterSynchronization: Event: Full synchronization has been completed (error: 1237)
  2217. 12/14 6:20:55.568532 5378 func: >>> HANode::reportEvent
  2218. 12/14 6:20:55.568571 5378 func: >>> iScsiServer::sendNotification
  2219. 12/14 6:20:55.568585 5378 func: >>> CEventDataBase::AddRecord
  2220. 12/14 6:20:55.568597 5378 EventDB: CEventDataBase::AddRecord: Code 783, severity 1, additional strings: 2
  2221. 12/14 6:20:55.568650 5378 func: <<< CEventDataBase::AddRecord
  2222. 12/14 6:20:55.568661 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 783).
  2223. 12/14 6:20:55.569395 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  2224. 12/14 6:20:55.569425 5378 func: <<< iScsiServer::sendNotification
  2225. 12/14 6:20:55.569438 5378 func: <<< HANode::reportEvent
  2226. 12/14 6:20:55.569449 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_COMPLETE_WITH_ERROR, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'
  2227. 12/14 6:20:55.575238 38d8 func: >>> HANode::onScsiOp_CustomControl_HeartbeatPing
  2228. 12/14 6:20:55.575386 38d8 Common: CStarWindStorageDevice::getLastPerfDegEventElapsedTimeMs: perfDegEventElapsedTime = -1
  2229. 12/14 6:20:55.575406 38d8 func: <<< HANode::onScsiOp_CustomControl_HeartbeatPing
  2230. 12/14 6:20:55.576297 38d8 func: >>> CHAPartnerNode::ShutdownNode
  2231. 12/14 6:20:55.576321 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  2232. 12/14 6:20:55.576331 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  2233. 12/14 6:20:55.576339 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  2234. 12/14 6:20:55.576346 38d8 func: <<< CHAPartnerNode::ShutdownNode
  2235. 12/14 6:20:58.237855 38d8 func: >>> CHAPartnerNode::ShutdownNode
  2236. 12/14 6:20:58.237920 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  2237. 12/14 6:20:58.237943 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  2238. 12/14 6:20:58.237964 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  2239. 12/14 6:20:58.237983 38d8 func: <<< CHAPartnerNode::ShutdownNode
  2240. 12/14 6:20:58.242338 2d00 HA: HANode::onScsiOp_CustomControl_SynchronizationRequest: The partner iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03 has requested the 1 synchronization type.
  2241. 12/14 6:20:58.244359 2d00 HA: CHAPartnerNode::Synchronize: ENTERed with params: p_SyncType = 1, p_ulSnapshotID = 0(0x0), p_pStorageDevice = 0x000002C7ECEDC4B0, p_pSyncBarrier = 0x000002C7ECEDC858, p_puliStartSector = 0x000000D2528FF118 (value is 0x0), p_puliEndSector = 0x000000D2528FF110 (value is 0xAEFFFFF), p_ulIOBlockSizeInSectors = 64, p_ulSectorSizeInUCHARs = 4096, p_bContinue = no, p_bVaaiWriteSameSupport = yes
  2242. 12/14 6:20:58.244395 2d00 HA: HANode::updatePartnerStatus: Statuses is already actual, we don't need to update them.
  2243. 12/14 6:20:58.244416 2d00 func: >>> CHAPartnerNode::SetSyncStatus
  2244. 12/14 6:20:58.244430 2d00 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  2245. 12/14 6:20:58.244444 2d00 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 2 from 3. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  2246. 12/14 6:20:58.244457 2d00 func: >>> HANode::reportEvent
  2247. 12/14 6:20:58.244515 2d00 func: >>> iScsiServer::sendNotification
  2248. 12/14 6:20:58.244531 2d00 func: >>> CEventDataBase::AddRecord
  2249. 12/14 6:20:58.244544 2d00 EventDB: CEventDataBase::AddRecord: Code 770, severity 2, additional strings: 2
  2250. 12/14 6:20:58.244604 2d00 func: <<< CEventDataBase::AddRecord
  2251. 12/14 6:20:58.244616 2d00 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 770).
  2252. 12/14 6:20:58.245209 2d00 Srv: iScsiServer::sendNotification: Send reaction finish.
  2253. 12/14 6:20:58.245238 2d00 func: <<< iScsiServer::sendNotification
  2254. 12/14 6:20:58.245251 2d00 func: <<< HANode::reportEvent
  2255. 12/14 6:20:58.245262 2d00 func: <<< CHAPartnerNode::SetSyncStatus
  2256. 12/14 6:20:58.245579 1558 func: >>> CHAPartnerNode::SetSyncStatus
  2257. 12/14 6:20:58.245629 1558 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  2258. 12/14 6:20:58.245650 1558 func: <<< CHAPartnerNode::SetSyncStatus
  2259. 12/14 6:20:58.248450 5378 func: >>> HANode::reportEvent
  2260. 12/14 6:20:58.248493 5378 func: >>> iScsiServer::sendNotification
  2261. 12/14 6:20:58.248504 5378 func: >>> CEventDataBase::AddRecord
  2262. 12/14 6:20:58.248514 5378 EventDB: CEventDataBase::AddRecord: Code 780, severity 1, additional strings: 3
  2263. 12/14 6:20:58.248614 5378 func: <<< CEventDataBase::AddRecord
  2264. 12/14 6:20:58.248624 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 780).
  2265. 12/14 6:20:58.249129 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  2266. 12/14 6:20:58.249151 5378 func: <<< iScsiServer::sendNotification
  2267. 12/14 6:20:58.249161 5378 func: <<< HANode::reportEvent
  2268. 12/14 6:20:58.249170 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_STARTED, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03', Synchronization type: 'Full'
  2269. 12/14 6:43:33.329379 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (9953 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 16937706, expcmdsn = 16937704, maxcmdsn = 16937944.
  2270. 12/14 6:43:33.329455 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16937690, itt = 16937779)
  2271. 12/14 6:43:33.329471 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16937691, itt = 16937780)
  2272. 12/14 6:43:33.329486 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16937692, itt = 16937781)
  2273. 12/14 6:43:33.329498 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16937693, itt = 16937782)
  2274. 12/14 6:43:33.329516 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8568 has failed due to unidentified error.
  2275. 12/14 6:43:33.329528 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2276. 12/14 6:43:33.329538 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F953D0 synchronization task has been failed!
  2277. 12/14 6:43:33.329567 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8268 has failed due to unidentified error.
  2278. 12/14 6:43:33.329578 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2279. 12/14 6:43:33.329586 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F950F0 synchronization task has been failed!
  2280. 12/14 6:43:33.329600 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86991E8 has failed due to unidentified error.
  2281. 12/14 6:43:33.329612 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2282. 12/14 6:43:33.329620 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F98D50 synchronization task has been failed!
  2283. 12/14 6:43:33.329635 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE60D8 has failed due to unidentified error.
  2284. 12/14 6:43:33.329633 1558 func: >>> CHAPartnerNode::SetSyncStatus
  2285. 12/14 6:43:33.329710 1558 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  2286. 12/14 6:43:33.329739 1558 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 3 from 2. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  2287. 12/14 6:43:33.329767 1558 func: >>> HANode::reportEvent
  2288. 12/14 6:43:33.329647 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2289. 12/14 6:43:33.329804 1558 func: >>> iScsiServer::sendNotification
  2290. 12/14 6:43:33.329836 1558 func: >>> CEventDataBase::AddRecord
  2291. 12/14 6:43:33.329865 1558 EventDB: CEventDataBase::AddRecord: Code 771, severity 3, additional strings: 2
  2292. 12/14 6:43:33.329812 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F95990 synchronization task has been failed!
  2293. 12/14 6:43:33.330147 1558 func: <<< CEventDataBase::AddRecord
  2294. 12/14 6:43:33.330172 1558 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 771).
  2295. 12/14 6:43:33.331686 1558 Srv: iScsiServer::sendNotification: Send reaction finish.
  2296. 12/14 6:43:33.331734 1558 func: <<< iScsiServer::sendNotification
  2297. 12/14 6:43:33.331754 1558 func: <<< HANode::reportEvent
  2298. 12/14 6:43:33.331772 1558 func: <<< CHAPartnerNode::SetSyncStatus
  2299. 12/14 6:43:33.348349 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (16 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 16937707, expcmdsn = 16937704, maxcmdsn = 16937944.
  2300. 12/14 6:43:33.348411 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16937694, itt = 16937783)
  2301. 12/14 6:43:33.348427 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16937695, itt = 16937784)
  2302. 12/14 6:43:33.348440 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16937696, itt = 16937785)
  2303. 12/14 6:43:33.348451 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16937697, itt = 16937786)
  2304. 12/14 6:43:33.348463 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16937698, itt = 16937787)
  2305. 12/14 6:43:33.348481 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BF88 has failed due to unidentified error.
  2306. 12/14 6:43:33.348494 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2307. 12/14 6:43:33.348504 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96DB0 synchronization task has been failed!
  2308. 12/14 6:43:33.348527 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EC4F39D8 has failed due to unidentified error.
  2309. 12/14 6:43:33.348538 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2310. 12/14 6:43:33.348546 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F98790 synchronization task has been failed!
  2311. 12/14 6:43:33.348560 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86994E8 has failed due to unidentified error.
  2312. 12/14 6:43:33.348571 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2313. 12/14 6:43:33.348578 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97650 synchronization task has been failed!
  2314. 12/14 6:43:33.348595 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6158 has failed due to unidentified error.
  2315. 12/14 6:43:33.348606 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2316. 12/14 6:43:33.348613 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F956B0 synchronization task has been failed!
  2317. 12/14 6:43:33.348626 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7BE8 has failed due to unidentified error.
  2318. 12/14 6:43:33.348636 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2319. 12/14 6:43:33.348644 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F98A70 synchronization task has been failed!
  2320. 12/14 6:43:33.364315 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (15 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 16937707, expcmdsn = 16937704, maxcmdsn = 16937944.
  2321. 12/14 6:43:33.364378 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16937699, itt = 16937788)
  2322. 12/14 6:43:33.364401 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6758 has failed due to unidentified error.
  2323. 12/14 6:43:33.364416 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2324. 12/14 6:43:33.364429 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97090 synchronization task has been failed!
  2325. 12/14 6:43:33.381342 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (16 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 16937707, expcmdsn = 16937704, maxcmdsn = 16937944.
  2326. 12/14 6:43:33.381408 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16937700, itt = 16937789)
  2327. 12/14 6:43:33.381423 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16937701, itt = 16937790)
  2328. 12/14 6:43:33.381436 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16937702, itt = 16937791)
  2329. 12/14 6:43:33.381447 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16937703, itt = 16937792)
  2330. 12/14 6:43:33.381463 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16937704, itt = 16937793)
  2331. 12/14 6:43:33.381474 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 16937705, itt = 16937794)
  2332. 12/14 6:43:33.381488 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BE08 has failed due to unidentified error.
  2333. 12/14 6:43:33.381500 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2334. 12/14 6:43:33.381510 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96AD0 synchronization task has been failed!
  2335. 12/14 6:43:33.381535 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7D68 has failed due to unidentified error.
  2336. 12/14 6:43:33.381545 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2337. 12/14 6:43:33.381553 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97C10 synchronization task has been failed!
  2338. 12/14 6:43:33.381568 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86993E8 has failed due to unidentified error.
  2339. 12/14 6:43:33.381578 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2340. 12/14 6:43:33.381585 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96510 synchronization task has been failed!
  2341. 12/14 6:43:33.381600 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86999E8 has failed due to unidentified error.
  2342. 12/14 6:43:33.381610 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2343. 12/14 6:43:33.381617 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96230 synchronization task has been failed!
  2344. 12/14 6:43:33.381631 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E8699A68 has failed due to unidentified error.
  2345. 12/14 6:43:33.381640 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2346. 12/14 6:43:33.381647 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F95F50 synchronization task has been failed!
  2347. 12/14 6:43:33.381662 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D85E8 has failed due to unidentified error.
  2348. 12/14 6:43:33.381672 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2349. 12/14 6:43:33.381680 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F95C70 synchronization task has been failed!
  2350. 12/14 6:43:38.081009 1558 HA: CHAPartnerNode::AfterSynchronization: Event: Full synchronization has been completed (error: 1237)
  2351. 12/14 6:43:38.084524 38d8 func: >>> HANode::onScsiOp_CustomControl_HeartbeatPing
  2352. 12/14 6:43:38.084617 38d8 Common: CStarWindStorageDevice::getLastPerfDegEventElapsedTimeMs: perfDegEventElapsedTime = -1
  2353. 12/14 6:43:38.084633 38d8 func: <<< HANode::onScsiOp_CustomControl_HeartbeatPing
  2354. 12/14 6:43:38.085148 5378 func: >>> HANode::reportEvent
  2355. 12/14 6:43:38.085208 5378 func: >>> iScsiServer::sendNotification
  2356. 12/14 6:43:38.085229 5378 func: >>> CEventDataBase::AddRecord
  2357. 12/14 6:43:38.085249 5378 EventDB: CEventDataBase::AddRecord: Code 783, severity 1, additional strings: 2
  2358. 12/14 6:43:38.085330 5378 func: <<< CEventDataBase::AddRecord
  2359. 12/14 6:43:38.085350 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 783).
  2360. 12/14 6:43:38.085569 38d8 func: >>> CHAPartnerNode::ShutdownNode
  2361. 12/14 6:43:38.085601 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  2362. 12/14 6:43:38.085610 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  2363. 12/14 6:43:38.085619 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  2364. 12/14 6:43:38.085626 38d8 func: <<< CHAPartnerNode::ShutdownNode
  2365. 12/14 6:43:38.086519 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  2366. 12/14 6:43:38.086570 5378 func: <<< iScsiServer::sendNotification
  2367. 12/14 6:43:38.086591 5378 func: <<< HANode::reportEvent
  2368. 12/14 6:43:38.086611 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_COMPLETE_WITH_ERROR, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'
  2369. 12/14 6:43:40.898871 38d8 func: >>> CHAPartnerNode::ShutdownNode
  2370. 12/14 6:43:40.898922 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  2371. 12/14 6:43:40.898940 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  2372. 12/14 6:43:40.898958 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  2373. 12/14 6:43:40.898972 38d8 func: <<< CHAPartnerNode::ShutdownNode
  2374. 12/14 6:43:40.903498 2d00 HA: HANode::onScsiOp_CustomControl_SynchronizationRequest: The partner iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03 has requested the 1 synchronization type.
  2375. 12/14 6:43:40.905221 2d00 HA: CHAPartnerNode::Synchronize: ENTERed with params: p_SyncType = 1, p_ulSnapshotID = 0(0x0), p_pStorageDevice = 0x000002C7ECEDC4B0, p_pSyncBarrier = 0x000002C7ECEDC858, p_puliStartSector = 0x000000D2528FF118 (value is 0x0), p_puliEndSector = 0x000000D2528FF110 (value is 0xAEFFFFF), p_ulIOBlockSizeInSectors = 64, p_ulSectorSizeInUCHARs = 4096, p_bContinue = no, p_bVaaiWriteSameSupport = yes
  2376. 12/14 6:43:40.905277 2d00 HA: HANode::updatePartnerStatus: Statuses is already actual, we don't need to update them.
  2377. 12/14 6:43:40.905306 2d00 func: >>> CHAPartnerNode::SetSyncStatus
  2378. 12/14 6:43:40.905326 2d00 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  2379. 12/14 6:43:40.905346 2d00 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 2 from 3. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  2380. 12/14 6:43:40.905365 2d00 func: >>> HANode::reportEvent
  2381. 12/14 6:43:40.905385 2d00 func: >>> iScsiServer::sendNotification
  2382. 12/14 6:43:40.905404 2d00 func: >>> CEventDataBase::AddRecord
  2383. 12/14 6:43:40.905427 2d00 EventDB: CEventDataBase::AddRecord: Code 770, severity 2, additional strings: 2
  2384. 12/14 6:43:40.905506 2d00 func: <<< CEventDataBase::AddRecord
  2385. 12/14 6:43:40.905524 2d00 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 770).
  2386. 12/14 6:43:40.906289 2d00 Srv: iScsiServer::sendNotification: Send reaction finish.
  2387. 12/14 6:43:40.906328 2d00 func: <<< iScsiServer::sendNotification
  2388. 12/14 6:43:40.906348 2d00 func: <<< HANode::reportEvent
  2389. 12/14 6:43:40.906366 2d00 func: <<< CHAPartnerNode::SetSyncStatus
  2390. 12/14 6:43:40.906720 54b4 func: >>> CHAPartnerNode::SetSyncStatus
  2391. 12/14 6:43:40.906765 54b4 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  2392. 12/14 6:43:40.906787 54b4 func: <<< CHAPartnerNode::SetSyncStatus
  2393. 12/14 6:43:40.909959 5378 func: >>> HANode::reportEvent
  2394. 12/14 6:43:40.909991 5378 func: >>> iScsiServer::sendNotification
  2395. 12/14 6:43:40.910000 5378 func: >>> CEventDataBase::AddRecord
  2396. 12/14 6:43:40.910009 5378 EventDB: CEventDataBase::AddRecord: Code 780, severity 1, additional strings: 3
  2397. 12/14 6:43:40.910045 5378 func: <<< CEventDataBase::AddRecord
  2398. 12/14 6:43:40.910052 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 780).
  2399. 12/14 6:43:40.910522 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  2400. 12/14 6:43:40.910541 5378 func: <<< iScsiServer::sendNotification
  2401. 12/14 6:43:40.910549 5378 func: <<< HANode::reportEvent
  2402. 12/14 6:43:40.910556 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_STARTED, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03', Synchronization type: 'Full'
  2403. 12/14 7:06:15.943300 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (9953 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 17397651, expcmdsn = 17397650, maxcmdsn = 17397889.
  2404. 12/14 7:06:15.943407 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 17397635, itt = 17397724)
  2405. 12/14 7:06:15.943420 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 17397636, itt = 17397725)
  2406. 12/14 7:06:15.943430 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 17397637, itt = 17397726)
  2407. 12/14 7:06:15.943443 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6158 has failed due to unidentified error.
  2408. 12/14 7:06:15.943454 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2409. 12/14 7:06:15.943463 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96510 synchronization task has been failed!
  2410. 12/14 7:06:15.943486 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6258 has failed due to unidentified error.
  2411. 12/14 7:06:15.943493 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2412. 12/14 7:06:15.943498 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97370 synchronization task has been failed!
  2413. 12/14 7:06:15.943507 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8568 has failed due to unidentified error.
  2414. 12/14 7:06:15.943514 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2415. 12/14 7:06:15.943519 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97650 synchronization task has been failed!
  2416. 12/14 7:06:15.943525 54b4 func: >>> CHAPartnerNode::SetSyncStatus
  2417. 12/14 7:06:15.943561 54b4 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  2418. 12/14 7:06:15.943571 54b4 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 3 from 2. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  2419. 12/14 7:06:15.943579 54b4 func: >>> HANode::reportEvent
  2420. 12/14 7:06:15.943598 54b4 func: >>> iScsiServer::sendNotification
  2421. 12/14 7:06:15.943606 54b4 func: >>> CEventDataBase::AddRecord
  2422. 12/14 7:06:15.943613 54b4 EventDB: CEventDataBase::AddRecord: Code 771, severity 3, additional strings: 2
  2423. 12/14 7:06:15.943718 54b4 func: <<< CEventDataBase::AddRecord
  2424. 12/14 7:06:15.943727 54b4 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 771).
  2425. 12/14 7:06:15.944369 54b4 Srv: iScsiServer::sendNotification: Send reaction finish.
  2426. 12/14 7:06:15.944390 54b4 func: <<< iScsiServer::sendNotification
  2427. 12/14 7:06:15.944397 54b4 func: <<< HANode::reportEvent
  2428. 12/14 7:06:15.944413 54b4 func: <<< CHAPartnerNode::SetSyncStatus
  2429. 12/14 7:06:15.963287 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (16 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 17397652, expcmdsn = 17397650, maxcmdsn = 17397889.
  2430. 12/14 7:06:15.963389 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 17397638, itt = 17397727)
  2431. 12/14 7:06:15.963405 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 17397639, itt = 17397728)
  2432. 12/14 7:06:15.963413 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 17397640, itt = 17397729)
  2433. 12/14 7:06:15.963420 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 17397641, itt = 17397730)
  2434. 12/14 7:06:15.963435 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E8699A68 has failed due to unidentified error.
  2435. 12/14 7:06:15.963443 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2436. 12/14 7:06:15.963450 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F95F50 synchronization task has been failed!
  2437. 12/14 7:06:15.963494 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7D68 has failed due to unidentified error.
  2438. 12/14 7:06:15.963501 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2439. 12/14 7:06:15.963506 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F95990 synchronization task has been failed!
  2440. 12/14 7:06:15.963517 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86991E8 has failed due to unidentified error.
  2441. 12/14 7:06:15.963523 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2442. 12/14 7:06:15.963528 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96DB0 synchronization task has been failed!
  2443. 12/14 7:06:15.963539 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6758 has failed due to unidentified error.
  2444. 12/14 7:06:15.963545 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2445. 12/14 7:06:15.963550 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96230 synchronization task has been failed!
  2446. 12/14 7:06:15.978932 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (15 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 17397652, expcmdsn = 17397650, maxcmdsn = 17397889.
  2447. 12/14 7:06:15.978980 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 17397642, itt = 17397731)
  2448. 12/14 7:06:15.978990 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 17397643, itt = 17397732)
  2449. 12/14 7:06:15.978998 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 17397644, itt = 17397733)
  2450. 12/14 7:06:15.979005 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 17397645, itt = 17397734)
  2451. 12/14 7:06:15.979013 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 17397646, itt = 17397735)
  2452. 12/14 7:06:15.979020 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 17397647, itt = 17397736)
  2453. 12/14 7:06:15.979026 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 17397648, itt = 17397737)
  2454. 12/14 7:06:15.979036 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86993E8 has failed due to unidentified error.
  2455. 12/14 7:06:15.979044 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2456. 12/14 7:06:15.979056 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F984B0 synchronization task has been failed!
  2457. 12/14 7:06:15.979078 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D85E8 has failed due to unidentified error.
  2458. 12/14 7:06:15.979084 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2459. 12/14 7:06:15.979089 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F956B0 synchronization task has been failed!
  2460. 12/14 7:06:15.979099 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86994E8 has failed due to unidentified error.
  2461. 12/14 7:06:15.979105 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2462. 12/14 7:06:15.979110 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F95C70 synchronization task has been failed!
  2463. 12/14 7:06:15.979120 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BE08 has failed due to unidentified error.
  2464. 12/14 7:06:15.979191 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2465. 12/14 7:06:15.979198 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F950F0 synchronization task has been failed!
  2466. 12/14 7:06:15.979210 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8268 has failed due to unidentified error.
  2467. 12/14 7:06:15.979216 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2468. 12/14 7:06:15.979221 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F98A70 synchronization task has been failed!
  2469. 12/14 7:06:15.979253 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BF88 has failed due to unidentified error.
  2470. 12/14 7:06:15.979259 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2471. 12/14 7:06:15.979264 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F967F0 synchronization task has been failed!
  2472. 12/14 7:06:15.979275 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7BE8 has failed due to unidentified error.
  2473. 12/14 7:06:15.979281 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2474. 12/14 7:06:15.979286 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96AD0 synchronization task has been failed!
  2475. 12/14 7:06:15.994929 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (16 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 17397652, expcmdsn = 17397650, maxcmdsn = 17397889.
  2476. 12/14 7:06:15.994977 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 17397649, itt = 17397738)
  2477. 12/14 7:06:15.994987 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 17397650, itt = 17397739)
  2478. 12/14 7:06:15.994998 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE60D8 has failed due to unidentified error.
  2479. 12/14 7:06:15.995006 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2480. 12/14 7:06:15.995013 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97930 synchronization task has been failed!
  2481. 12/14 7:06:15.995033 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86999E8 has failed due to unidentified error.
  2482. 12/14 7:06:15.995156 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2483. 12/14 7:06:15.995162 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97090 synchronization task has been failed!
  2484. 12/14 7:06:20.688306 54b4 HA: CHAPartnerNode::AfterSynchronization: Event: Full synchronization has been completed (error: 1237)
  2485. 12/14 7:06:20.690733 5378 func: >>> HANode::reportEvent
  2486. 12/14 7:06:20.690774 5378 func: >>> iScsiServer::sendNotification
  2487. 12/14 7:06:20.690782 5378 func: >>> CEventDataBase::AddRecord
  2488. 12/14 7:06:20.690790 5378 EventDB: CEventDataBase::AddRecord: Code 783, severity 1, additional strings: 2
  2489. 12/14 7:06:20.690835 5378 func: <<< CEventDataBase::AddRecord
  2490. 12/14 7:06:20.690842 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 783).
  2491. 12/14 7:06:20.691398 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  2492. 12/14 7:06:20.691424 5378 func: <<< iScsiServer::sendNotification
  2493. 12/14 7:06:20.691435 5378 func: <<< HANode::reportEvent
  2494. 12/14 7:06:20.691445 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_COMPLETE_WITH_ERROR, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'
  2495. 12/14 7:06:20.694214 38d8 func: >>> HANode::onScsiOp_CustomControl_HeartbeatPing
  2496. 12/14 7:06:20.694295 38d8 Common: CStarWindStorageDevice::getLastPerfDegEventElapsedTimeMs: perfDegEventElapsedTime = -1
  2497. 12/14 7:06:20.694302 38d8 func: <<< HANode::onScsiOp_CustomControl_HeartbeatPing
  2498. 12/14 7:06:20.694954 38d8 func: >>> CHAPartnerNode::ShutdownNode
  2499. 12/14 7:06:20.694986 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  2500. 12/14 7:06:20.695005 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  2501. 12/14 7:06:20.695039 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  2502. 12/14 7:06:20.695058 38d8 func: <<< CHAPartnerNode::ShutdownNode
  2503. 12/14 7:06:23.585517 38d8 func: >>> CHAPartnerNode::ShutdownNode
  2504. 12/14 7:06:23.585572 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  2505. 12/14 7:06:23.585589 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  2506. 12/14 7:06:23.585604 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  2507. 12/14 7:06:23.585617 38d8 func: <<< CHAPartnerNode::ShutdownNode
  2508. 12/14 7:06:23.591204 2d00 HA: HANode::onScsiOp_CustomControl_SynchronizationRequest: The partner iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03 has requested the 1 synchronization type.
  2509. 12/14 7:06:23.591985 2d00 HA: CHAPartnerNode::Synchronize: ENTERed with params: p_SyncType = 1, p_ulSnapshotID = 0(0x0), p_pStorageDevice = 0x000002C7ECEDC4B0, p_pSyncBarrier = 0x000002C7ECEDC858, p_puliStartSector = 0x000000D2528FF118 (value is 0x0), p_puliEndSector = 0x000000D2528FF110 (value is 0xAEFFFFF), p_ulIOBlockSizeInSectors = 64, p_ulSectorSizeInUCHARs = 4096, p_bContinue = no, p_bVaaiWriteSameSupport = yes
  2510. 12/14 7:06:23.592020 2d00 HA: HANode::updatePartnerStatus: Statuses is already actual, we don't need to update them.
  2511. 12/14 7:06:23.592039 2d00 func: >>> CHAPartnerNode::SetSyncStatus
  2512. 12/14 7:06:23.592050 2d00 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  2513. 12/14 7:06:23.592060 2d00 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 2 from 3. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  2514. 12/14 7:06:23.592071 2d00 func: >>> HANode::reportEvent
  2515. 12/14 7:06:23.592082 2d00 func: >>> iScsiServer::sendNotification
  2516. 12/14 7:06:23.592092 2d00 func: >>> CEventDataBase::AddRecord
  2517. 12/14 7:06:23.592102 2d00 EventDB: CEventDataBase::AddRecord: Code 770, severity 2, additional strings: 2
  2518. 12/14 7:06:23.592156 2d00 func: <<< CEventDataBase::AddRecord
  2519. 12/14 7:06:23.592166 2d00 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 770).
  2520. 12/14 7:06:23.592674 2d00 Srv: iScsiServer::sendNotification: Send reaction finish.
  2521. 12/14 7:06:23.592698 2d00 func: <<< iScsiServer::sendNotification
  2522. 12/14 7:06:23.592709 2d00 func: <<< HANode::reportEvent
  2523. 12/14 7:06:23.592719 2d00 func: <<< CHAPartnerNode::SetSyncStatus
  2524. 12/14 7:06:23.593555 50a4 func: >>> CHAPartnerNode::SetSyncStatus
  2525. 12/14 7:06:23.593592 50a4 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  2526. 12/14 7:06:23.593608 50a4 func: <<< CHAPartnerNode::SetSyncStatus
  2527. 12/14 7:06:23.596120 5378 func: >>> HANode::reportEvent
  2528. 12/14 7:06:23.596152 5378 func: >>> iScsiServer::sendNotification
  2529. 12/14 7:06:23.596161 5378 func: >>> CEventDataBase::AddRecord
  2530. 12/14 7:06:23.596168 5378 EventDB: CEventDataBase::AddRecord: Code 780, severity 1, additional strings: 3
  2531. 12/14 7:06:23.596210 5378 func: <<< CEventDataBase::AddRecord
  2532. 12/14 7:06:23.596220 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 780).
  2533. 12/14 7:06:23.596791 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  2534. 12/14 7:06:23.596819 5378 func: <<< iScsiServer::sendNotification
  2535. 12/14 7:06:23.596827 5378 func: <<< HANode::reportEvent
  2536. 12/14 7:06:23.596834 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_STARTED, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03', Synchronization type: 'Full'
  2537. 12/14 7:28:59.707935 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (9969 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 17857985, expcmdsn = 17857982, maxcmdsn = 17858223.
  2538. 12/14 7:28:59.708017 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 17857969, itt = 17858058)
  2539. 12/14 7:28:59.708050 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E8699A68 has failed due to unidentified error.
  2540. 12/14 7:28:59.708071 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2541. 12/14 7:28:59.708088 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96DB0 synchronization task has been failed!
  2542. 12/14 7:28:59.708177 50a4 func: >>> CHAPartnerNode::SetSyncStatus
  2543. 12/14 7:28:59.708243 50a4 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  2544. 12/14 7:28:59.708270 50a4 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 3 from 2. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  2545. 12/14 7:28:59.708294 50a4 func: >>> HANode::reportEvent
  2546. 12/14 7:28:59.708330 50a4 func: >>> iScsiServer::sendNotification
  2547. 12/14 7:28:59.708351 50a4 func: >>> CEventDataBase::AddRecord
  2548. 12/14 7:28:59.708374 50a4 EventDB: CEventDataBase::AddRecord: Code 771, severity 3, additional strings: 2
  2549. 12/14 7:28:59.708452 50a4 func: <<< CEventDataBase::AddRecord
  2550. 12/14 7:28:59.708472 50a4 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 771).
  2551. 12/14 7:28:59.709930 50a4 Srv: iScsiServer::sendNotification: Send reaction finish.
  2552. 12/14 7:28:59.709983 50a4 func: <<< iScsiServer::sendNotification
  2553. 12/14 7:28:59.710004 50a4 func: <<< HANode::reportEvent
  2554. 12/14 7:28:59.710022 50a4 func: <<< CHAPartnerNode::SetSyncStatus
  2555. 12/14 7:28:59.726920 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (15 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 17857986, expcmdsn = 17857982, maxcmdsn = 17858223.
  2556. 12/14 7:28:59.726992 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 17857970, itt = 17858059)
  2557. 12/14 7:28:59.727019 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 17857971, itt = 17858060)
  2558. 12/14 7:28:59.727032 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 17857972, itt = 17858061)
  2559. 12/14 7:28:59.727043 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 17857973, itt = 17858062)
  2560. 12/14 7:28:59.727052 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 17857974, itt = 17858063)
  2561. 12/14 7:28:59.727062 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 17857975, itt = 17858064)
  2562. 12/14 7:28:59.727069 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 17857976, itt = 17858065)
  2563. 12/14 7:28:59.727080 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86993E8 has failed due to unidentified error.
  2564. 12/14 7:28:59.727087 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2565. 12/14 7:28:59.727094 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F953D0 synchronization task has been failed!
  2566. 12/14 7:28:59.727109 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6758 has failed due to unidentified error.
  2567. 12/14 7:28:59.727116 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2568. 12/14 7:28:59.727120 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96AD0 synchronization task has been failed!
  2569. 12/14 7:28:59.727129 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BE08 has failed due to unidentified error.
  2570. 12/14 7:28:59.727138 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2571. 12/14 7:28:59.727145 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F98D50 synchronization task has been failed!
  2572. 12/14 7:28:59.727161 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86991E8 has failed due to unidentified error.
  2573. 12/14 7:28:59.727167 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2574. 12/14 7:28:59.727172 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F98790 synchronization task has been failed!
  2575. 12/14 7:28:59.727184 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8568 has failed due to unidentified error.
  2576. 12/14 7:28:59.727190 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2577. 12/14 7:28:59.727197 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97EF0 synchronization task has been failed!
  2578. 12/14 7:28:59.727208 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86994E8 has failed due to unidentified error.
  2579. 12/14 7:28:59.727215 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2580. 12/14 7:28:59.727220 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F984B0 synchronization task has been failed!
  2581. 12/14 7:28:59.727232 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D85E8 has failed due to unidentified error.
  2582. 12/14 7:28:59.727240 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2583. 12/14 7:28:59.727245 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97930 synchronization task has been failed!
  2584. 12/14 7:28:59.743506 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (16 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 17857986, expcmdsn = 17857982, maxcmdsn = 17858223.
  2585. 12/14 7:28:59.743545 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 17857977, itt = 17858066)
  2586. 12/14 7:28:59.743556 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 17857978, itt = 17858067)
  2587. 12/14 7:28:59.743564 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 17857979, itt = 17858068)
  2588. 12/14 7:28:59.743571 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 17857980, itt = 17858069)
  2589. 12/14 7:28:59.743579 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 17857981, itt = 17858070)
  2590. 12/14 7:28:59.743587 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 17857982, itt = 17858071)
  2591. 12/14 7:28:59.743595 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 17857983, itt = 17858072)
  2592. 12/14 7:28:59.743605 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 17857984, itt = 17858073)
  2593. 12/14 7:28:59.743616 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6258 has failed due to unidentified error.
  2594. 12/14 7:28:59.743626 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2595. 12/14 7:28:59.743635 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F956B0 synchronization task has been failed!
  2596. 12/14 7:28:59.743656 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6158 has failed due to unidentified error.
  2597. 12/14 7:28:59.743665 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2598. 12/14 7:28:59.743672 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F98A70 synchronization task has been failed!
  2599. 12/14 7:28:59.743685 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7BE8 has failed due to unidentified error.
  2600. 12/14 7:28:59.743694 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2601. 12/14 7:28:59.743702 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F95F50 synchronization task has been failed!
  2602. 12/14 7:28:59.743716 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BF88 has failed due to unidentified error.
  2603. 12/14 7:28:59.743724 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2604. 12/14 7:28:59.743730 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F95C70 synchronization task has been failed!
  2605. 12/14 7:28:59.743742 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7D68 has failed due to unidentified error.
  2606. 12/14 7:28:59.743750 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2607. 12/14 7:28:59.743755 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97C10 synchronization task has been failed!
  2608. 12/14 7:28:59.743766 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86999E8 has failed due to unidentified error.
  2609. 12/14 7:28:59.743773 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2610. 12/14 7:28:59.743777 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F950F0 synchronization task has been failed!
  2611. 12/14 7:28:59.743786 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8268 has failed due to unidentified error.
  2612. 12/14 7:28:59.743799 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2613. 12/14 7:28:59.743806 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F981D0 synchronization task has been failed!
  2614. 12/14 7:28:59.743817 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE60D8 has failed due to unidentified error.
  2615. 12/14 7:28:59.743826 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2616. 12/14 7:28:59.743833 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97090 synchronization task has been failed!
  2617. 12/14 7:29:04.423790 50a4 HA: CHAPartnerNode::AfterSynchronization: Event: Full synchronization has been completed (error: 1237)
  2618. 12/14 7:29:04.425981 5378 func: >>> HANode::reportEvent
  2619. 12/14 7:29:04.426013 5378 func: >>> iScsiServer::sendNotification
  2620. 12/14 7:29:04.426022 5378 func: >>> CEventDataBase::AddRecord
  2621. 12/14 7:29:04.426029 5378 EventDB: CEventDataBase::AddRecord: Code 783, severity 1, additional strings: 2
  2622. 12/14 7:29:04.426069 5378 func: <<< CEventDataBase::AddRecord
  2623. 12/14 7:29:04.426075 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 783).
  2624. 12/14 7:29:04.426597 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  2625. 12/14 7:29:04.426620 5378 func: <<< iScsiServer::sendNotification
  2626. 12/14 7:29:04.426628 5378 func: <<< HANode::reportEvent
  2627. 12/14 7:29:04.426634 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_COMPLETE_WITH_ERROR, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'
  2628. 12/14 7:29:04.427500 38d8 func: >>> HANode::onScsiOp_CustomControl_HeartbeatPing
  2629. 12/14 7:29:04.427663 38d8 Common: CStarWindStorageDevice::getLastPerfDegEventElapsedTimeMs: perfDegEventElapsedTime = -1
  2630. 12/14 7:29:04.427687 38d8 func: <<< HANode::onScsiOp_CustomControl_HeartbeatPing
  2631. 12/14 7:29:04.428594 38d8 func: >>> CHAPartnerNode::ShutdownNode
  2632. 12/14 7:29:04.428752 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  2633. 12/14 7:29:04.428793 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  2634. 12/14 7:29:04.428817 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  2635. 12/14 7:29:04.428839 38d8 func: <<< CHAPartnerNode::ShutdownNode
  2636. 12/14 7:29:06.155282 38d8 func: >>> CHAPartnerNode::ShutdownNode
  2637. 12/14 7:29:06.155352 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  2638. 12/14 7:29:06.155375 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  2639. 12/14 7:29:06.155396 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  2640. 12/14 7:29:06.155414 38d8 func: <<< CHAPartnerNode::ShutdownNode
  2641. 12/14 7:29:06.160027 2d00 HA: HANode::onScsiOp_CustomControl_SynchronizationRequest: The partner iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03 has requested the 1 synchronization type.
  2642. 12/14 7:29:06.162081 2d00 HA: CHAPartnerNode::Synchronize: ENTERed with params: p_SyncType = 1, p_ulSnapshotID = 0(0x0), p_pStorageDevice = 0x000002C7ECEDC4B0, p_pSyncBarrier = 0x000002C7ECEDC858, p_puliStartSector = 0x000000D2528FF118 (value is 0x0), p_puliEndSector = 0x000000D2528FF110 (value is 0xAEFFFFF), p_ulIOBlockSizeInSectors = 64, p_ulSectorSizeInUCHARs = 4096, p_bContinue = no, p_bVaaiWriteSameSupport = yes
  2643. 12/14 7:29:06.162129 2d00 HA: HANode::updatePartnerStatus: Statuses is already actual, we don't need to update them.
  2644. 12/14 7:29:06.162156 2d00 func: >>> CHAPartnerNode::SetSyncStatus
  2645. 12/14 7:29:06.162175 2d00 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  2646. 12/14 7:29:06.162209 2d00 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 2 from 3. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  2647. 12/14 7:29:06.162229 2d00 func: >>> HANode::reportEvent
  2648. 12/14 7:29:06.162249 2d00 func: >>> iScsiServer::sendNotification
  2649. 12/14 7:29:06.162268 2d00 func: >>> CEventDataBase::AddRecord
  2650. 12/14 7:29:06.162286 2d00 EventDB: CEventDataBase::AddRecord: Code 770, severity 2, additional strings: 2
  2651. 12/14 7:29:06.162362 2d00 func: <<< CEventDataBase::AddRecord
  2652. 12/14 7:29:06.162381 2d00 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 770).
  2653. 12/14 7:29:06.163176 2d00 Srv: iScsiServer::sendNotification: Send reaction finish.
  2654. 12/14 7:29:06.163215 2d00 func: <<< iScsiServer::sendNotification
  2655. 12/14 7:29:06.163234 2d00 func: <<< HANode::reportEvent
  2656. 12/14 7:29:06.163252 2d00 func: <<< CHAPartnerNode::SetSyncStatus
  2657. 12/14 7:29:06.163583 68c0 func: >>> CHAPartnerNode::SetSyncStatus
  2658. 12/14 7:29:06.163631 68c0 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  2659. 12/14 7:29:06.163652 68c0 func: <<< CHAPartnerNode::SetSyncStatus
  2660. 12/14 7:29:06.165689 5378 func: >>> HANode::reportEvent
  2661. 12/14 7:29:06.165720 5378 func: >>> iScsiServer::sendNotification
  2662. 12/14 7:29:06.165728 5378 func: >>> CEventDataBase::AddRecord
  2663. 12/14 7:29:06.165735 5378 EventDB: CEventDataBase::AddRecord: Code 780, severity 1, additional strings: 3
  2664. 12/14 7:29:06.165770 5378 func: <<< CEventDataBase::AddRecord
  2665. 12/14 7:29:06.165776 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 780).
  2666. 12/14 7:29:06.166252 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  2667. 12/14 7:29:06.166276 5378 func: <<< iScsiServer::sendNotification
  2668. 12/14 7:29:06.166285 5378 func: <<< HANode::reportEvent
  2669. 12/14 7:29:06.166294 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_STARTED, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03', Synchronization type: 'Full'
  2670. 12/14 7:51:42.466708 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (9953 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 18318068, expcmdsn = 18318067, maxcmdsn = 18318306.
  2671. 12/14 7:51:42.466822 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 18318052, itt = 18318141)
  2672. 12/14 7:51:42.466840 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 18318053, itt = 18318142)
  2673. 12/14 7:51:42.466863 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EC4F39D8 has failed due to unidentified error.
  2674. 12/14 7:51:42.466876 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2675. 12/14 7:51:42.466887 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F984B0 synchronization task has been failed!
  2676. 12/14 7:51:42.466911 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE60D8 has failed due to unidentified error.
  2677. 12/14 7:51:42.466923 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2678. 12/14 7:51:42.466932 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F98D50 synchronization task has been failed!
  2679. 12/14 7:51:42.467224 68c0 func: >>> CHAPartnerNode::SetSyncStatus
  2680. 12/14 7:51:42.467285 68c0 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  2681. 12/14 7:51:42.467308 68c0 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 3 from 2. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  2682. 12/14 7:51:42.467330 68c0 func: >>> HANode::reportEvent
  2683. 12/14 7:51:42.467389 68c0 func: >>> iScsiServer::sendNotification
  2684. 12/14 7:51:42.467410 68c0 func: >>> CEventDataBase::AddRecord
  2685. 12/14 7:51:42.467427 68c0 EventDB: CEventDataBase::AddRecord: Code 771, severity 3, additional strings: 2
  2686. 12/14 7:51:42.467852 68c0 func: <<< CEventDataBase::AddRecord
  2687. 12/14 7:51:42.467957 68c0 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 771).
  2688. 12/14 7:51:42.469133 68c0 Srv: iScsiServer::sendNotification: Send reaction finish.
  2689. 12/14 7:51:42.469186 68c0 func: <<< iScsiServer::sendNotification
  2690. 12/14 7:51:42.469207 68c0 func: <<< HANode::reportEvent
  2691. 12/14 7:51:42.469226 68c0 func: <<< CHAPartnerNode::SetSyncStatus
  2692. 12/14 7:51:42.485873 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (16 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 18318069, expcmdsn = 18318067, maxcmdsn = 18318306.
  2693. 12/14 7:51:42.485912 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 18318054, itt = 18318143)
  2694. 12/14 7:51:42.485924 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 18318055, itt = 18318144)
  2695. 12/14 7:51:42.485933 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 18318056, itt = 18318145)
  2696. 12/14 7:51:42.485941 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 18318057, itt = 18318146)
  2697. 12/14 7:51:42.485950 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 18318058, itt = 18318147)
  2698. 12/14 7:51:42.485960 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 18318059, itt = 18318148)
  2699. 12/14 7:51:42.485968 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 18318060, itt = 18318149)
  2700. 12/14 7:51:42.485977 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 18318061, itt = 18318150)
  2701. 12/14 7:51:42.485989 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86994E8 has failed due to unidentified error.
  2702. 12/14 7:51:42.485997 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2703. 12/14 7:51:42.486005 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97930 synchronization task has been failed!
  2704. 12/14 7:51:42.486023 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D85E8 has failed due to unidentified error.
  2705. 12/14 7:51:42.486030 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2706. 12/14 7:51:42.486036 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F95990 synchronization task has been failed!
  2707. 12/14 7:51:42.486047 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6758 has failed due to unidentified error.
  2708. 12/14 7:51:42.486055 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2709. 12/14 7:51:42.486060 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F981D0 synchronization task has been failed!
  2710. 12/14 7:51:42.486071 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8568 has failed due to unidentified error.
  2711. 12/14 7:51:42.486079 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2712. 12/14 7:51:42.486084 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F950F0 synchronization task has been failed!
  2713. 12/14 7:51:42.486096 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7BE8 has failed due to unidentified error.
  2714. 12/14 7:51:42.486103 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2715. 12/14 7:51:42.486113 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97370 synchronization task has been failed!
  2716. 12/14 7:51:42.486128 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86999E8 has failed due to unidentified error.
  2717. 12/14 7:51:42.486135 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2718. 12/14 7:51:42.486140 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96AD0 synchronization task has been failed!
  2719. 12/14 7:51:42.486151 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6258 has failed due to unidentified error.
  2720. 12/14 7:51:42.486158 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2721. 12/14 7:51:42.486163 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F98790 synchronization task has been failed!
  2722. 12/14 7:51:42.486173 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6158 has failed due to unidentified error.
  2723. 12/14 7:51:42.486180 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2724. 12/14 7:51:42.486185 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F956B0 synchronization task has been failed!
  2725. 12/14 7:51:42.502949 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (16 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 18318069, expcmdsn = 18318067, maxcmdsn = 18318306.
  2726. 12/14 7:51:42.503033 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 18318062, itt = 18318151)
  2727. 12/14 7:51:42.503051 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 18318063, itt = 18318152)
  2728. 12/14 7:51:42.503064 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 18318064, itt = 18318153)
  2729. 12/14 7:51:42.503109 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 18318065, itt = 18318154)
  2730. 12/14 7:51:42.503129 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 18318066, itt = 18318155)
  2731. 12/14 7:51:42.503155 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 18318067, itt = 18318156)
  2732. 12/14 7:51:42.503181 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7D68 has failed due to unidentified error.
  2733. 12/14 7:51:42.503201 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2734. 12/14 7:51:42.503212 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96510 synchronization task has been failed!
  2735. 12/14 7:51:42.503235 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86993E8 has failed due to unidentified error.
  2736. 12/14 7:51:42.503247 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2737. 12/14 7:51:42.503255 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97EF0 synchronization task has been failed!
  2738. 12/14 7:51:42.503273 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BE08 has failed due to unidentified error.
  2739. 12/14 7:51:42.503284 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2740. 12/14 7:51:42.503292 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97C10 synchronization task has been failed!
  2741. 12/14 7:51:42.503316 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86991E8 has failed due to unidentified error.
  2742. 12/14 7:51:42.503327 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2743. 12/14 7:51:42.503335 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97650 synchronization task has been failed!
  2744. 12/14 7:51:42.503348 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BF88 has failed due to unidentified error.
  2745. 12/14 7:51:42.503359 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2746. 12/14 7:51:42.503367 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F953D0 synchronization task has been failed!
  2747. 12/14 7:51:42.503383 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E8699A68 has failed due to unidentified error.
  2748. 12/14 7:51:42.503394 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2749. 12/14 7:51:42.503402 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97090 synchronization task has been failed!
  2750. 12/14 7:51:47.101396 38d8 func: >>> HANode::onScsiOp_CustomControl_HeartbeatPing
  2751. 12/14 7:51:47.101581 38d8 Common: CStarWindStorageDevice::getLastPerfDegEventElapsedTimeMs: perfDegEventElapsedTime = -1
  2752. 12/14 7:51:47.101603 38d8 func: <<< HANode::onScsiOp_CustomControl_HeartbeatPing
  2753. 12/14 7:51:47.102118 68c0 HA: CHAPartnerNode::AfterSynchronization: Event: Full synchronization has been completed (error: 1237)
  2754. 12/14 7:51:47.102514 38d8 func: >>> CHAPartnerNode::ShutdownNode
  2755. 12/14 7:51:47.102548 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  2756. 12/14 7:51:47.102568 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  2757. 12/14 7:51:47.102593 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  2758. 12/14 7:51:47.102611 38d8 func: <<< CHAPartnerNode::ShutdownNode
  2759. 12/14 7:51:47.104851 5378 func: >>> HANode::reportEvent
  2760. 12/14 7:51:47.104891 5378 func: >>> iScsiServer::sendNotification
  2761. 12/14 7:51:47.104903 5378 func: >>> CEventDataBase::AddRecord
  2762. 12/14 7:51:47.104913 5378 EventDB: CEventDataBase::AddRecord: Code 783, severity 1, additional strings: 2
  2763. 12/14 7:51:47.104962 5378 func: <<< CEventDataBase::AddRecord
  2764. 12/14 7:51:47.104972 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 783).
  2765. 12/14 7:51:47.105594 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  2766. 12/14 7:51:47.105617 5378 func: <<< iScsiServer::sendNotification
  2767. 12/14 7:51:47.105628 5378 func: <<< HANode::reportEvent
  2768. 12/14 7:51:47.105637 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_COMPLETE_WITH_ERROR, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'
  2769. 12/14 7:51:47.108382 38d8 Common: *** sw_common::Sw_Disk_Header::backup: Asynchronous header 'C:\Program Files\StarWind Software\StarWind\headers\datastore03\datastore03_HA.swdsk' operation in progress!
  2770. 12/14 7:51:48.801462 38d8 func: >>> CHAPartnerNode::ShutdownNode
  2771. 12/14 7:51:48.801539 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  2772. 12/14 7:51:48.801563 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  2773. 12/14 7:51:48.801583 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  2774. 12/14 7:51:48.801601 38d8 func: <<< CHAPartnerNode::ShutdownNode
  2775. 12/14 7:51:48.807068 2d00 HA: HANode::onScsiOp_CustomControl_SynchronizationRequest: The partner iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03 has requested the 1 synchronization type.
  2776. 12/14 7:51:48.809198 2d00 HA: CHAPartnerNode::Synchronize: ENTERed with params: p_SyncType = 1, p_ulSnapshotID = 0(0x0), p_pStorageDevice = 0x000002C7ECEDC4B0, p_pSyncBarrier = 0x000002C7ECEDC858, p_puliStartSector = 0x000000D2528FF118 (value is 0x0), p_puliEndSector = 0x000000D2528FF110 (value is 0xAEFFFFF), p_ulIOBlockSizeInSectors = 64, p_ulSectorSizeInUCHARs = 4096, p_bContinue = no, p_bVaaiWriteSameSupport = yes
  2777. 12/14 7:51:48.809262 2d00 HA: HANode::updatePartnerStatus: Statuses is already actual, we don't need to update them.
  2778. 12/14 7:51:48.809296 2d00 func: >>> CHAPartnerNode::SetSyncStatus
  2779. 12/14 7:51:48.809316 2d00 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  2780. 12/14 7:51:48.809338 2d00 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 2 from 3. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  2781. 12/14 7:51:48.809357 2d00 func: >>> HANode::reportEvent
  2782. 12/14 7:51:48.809377 2d00 func: >>> iScsiServer::sendNotification
  2783. 12/14 7:51:48.809396 2d00 func: >>> CEventDataBase::AddRecord
  2784. 12/14 7:51:48.809414 2d00 EventDB: CEventDataBase::AddRecord: Code 770, severity 2, additional strings: 2
  2785. 12/14 7:51:48.809510 2d00 func: <<< CEventDataBase::AddRecord
  2786. 12/14 7:51:48.809529 2d00 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 770).
  2787. 12/14 7:51:48.810168 2d00 Srv: iScsiServer::sendNotification: Send reaction finish.
  2788. 12/14 7:51:48.810196 2d00 func: <<< iScsiServer::sendNotification
  2789. 12/14 7:51:48.810209 2d00 func: <<< HANode::reportEvent
  2790. 12/14 7:51:48.810221 2d00 func: <<< CHAPartnerNode::SetSyncStatus
  2791. 12/14 7:51:48.810910 65cc func: >>> CHAPartnerNode::SetSyncStatus
  2792. 12/14 7:51:48.810949 65cc HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  2793. 12/14 7:51:48.810965 65cc func: <<< CHAPartnerNode::SetSyncStatus
  2794. 12/14 7:51:48.813444 5378 func: >>> HANode::reportEvent
  2795. 12/14 7:51:48.813473 5378 func: >>> iScsiServer::sendNotification
  2796. 12/14 7:51:48.813481 5378 func: >>> CEventDataBase::AddRecord
  2797. 12/14 7:51:48.813487 5378 EventDB: CEventDataBase::AddRecord: Code 780, severity 1, additional strings: 3
  2798. 12/14 7:51:48.813526 5378 func: <<< CEventDataBase::AddRecord
  2799. 12/14 7:51:48.813533 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 780).
  2800. 12/14 7:51:48.814025 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  2801. 12/14 7:51:48.814046 5378 func: <<< iScsiServer::sendNotification
  2802. 12/14 7:51:48.814054 5378 func: <<< HANode::reportEvent
  2803. 12/14 7:51:48.814060 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_STARTED, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03', Synchronization type: 'Full'
  2804. 12/14 8:14:24.848907 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (9953 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 18778181, expcmdsn = 18778180, maxcmdsn = 18778419.
  2805. 12/14 8:14:24.848981 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 18778165, itt = 18778254)
  2806. 12/14 8:14:24.849012 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86994E8 has failed due to unidentified error.
  2807. 12/14 8:14:24.849030 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2808. 12/14 8:14:24.849045 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97090 synchronization task has been failed!
  2809. 12/14 8:14:24.849126 65cc func: >>> CHAPartnerNode::SetSyncStatus
  2810. 12/14 8:14:24.849187 65cc HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  2811. 12/14 8:14:24.849214 65cc HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 3 from 2. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  2812. 12/14 8:14:24.849262 65cc func: >>> HANode::reportEvent
  2813. 12/14 8:14:24.849305 65cc func: >>> iScsiServer::sendNotification
  2814. 12/14 8:14:24.849340 65cc func: >>> CEventDataBase::AddRecord
  2815. 12/14 8:14:24.849367 65cc EventDB: CEventDataBase::AddRecord: Code 771, severity 3, additional strings: 2
  2816. 12/14 8:14:24.849449 65cc func: <<< CEventDataBase::AddRecord
  2817. 12/14 8:14:24.849469 65cc Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 771).
  2818. 12/14 8:14:24.850591 65cc Srv: iScsiServer::sendNotification: Send reaction finish.
  2819. 12/14 8:14:24.850634 65cc func: <<< iScsiServer::sendNotification
  2820. 12/14 8:14:24.850655 65cc func: <<< HANode::reportEvent
  2821. 12/14 8:14:24.850674 65cc func: <<< CHAPartnerNode::SetSyncStatus
  2822. 12/14 8:14:24.867958 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (16 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 18778182, expcmdsn = 18778180, maxcmdsn = 18778419.
  2823. 12/14 8:14:24.868028 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 18778166, itt = 18778255)
  2824. 12/14 8:14:24.868055 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 18778167, itt = 18778256)
  2825. 12/14 8:14:24.868077 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 18778168, itt = 18778257)
  2826. 12/14 8:14:24.868118 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 18778169, itt = 18778258)
  2827. 12/14 8:14:24.868126 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 18778170, itt = 18778259)
  2828. 12/14 8:14:24.868133 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 18778171, itt = 18778260)
  2829. 12/14 8:14:24.868139 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 18778172, itt = 18778261)
  2830. 12/14 8:14:24.868147 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 18778173, itt = 18778262)
  2831. 12/14 8:14:24.868154 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 18778174, itt = 18778263)
  2832. 12/14 8:14:24.868160 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 18778175, itt = 18778264)
  2833. 12/14 8:14:24.868169 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 18778176, itt = 18778265)
  2834. 12/14 8:14:24.868176 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 18778177, itt = 18778266)
  2835. 12/14 8:14:24.868185 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86999E8 has failed due to unidentified error.
  2836. 12/14 8:14:24.868192 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2837. 12/14 8:14:24.868203 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F950F0 synchronization task has been failed!
  2838. 12/14 8:14:24.868220 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EC4F39D8 has failed due to unidentified error.
  2839. 12/14 8:14:24.868226 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2840. 12/14 8:14:24.868231 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97C10 synchronization task has been failed!
  2841. 12/14 8:14:24.868239 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7D68 has failed due to unidentified error.
  2842. 12/14 8:14:24.868246 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2843. 12/14 8:14:24.868251 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F981D0 synchronization task has been failed!
  2844. 12/14 8:14:24.868259 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6158 has failed due to unidentified error.
  2845. 12/14 8:14:24.868273 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2846. 12/14 8:14:24.868278 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F953D0 synchronization task has been failed!
  2847. 12/14 8:14:24.868288 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8568 has failed due to unidentified error.
  2848. 12/14 8:14:24.868295 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2849. 12/14 8:14:24.868300 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F95990 synchronization task has been failed!
  2850. 12/14 8:14:24.868307 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86991E8 has failed due to unidentified error.
  2851. 12/14 8:14:24.868314 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2852. 12/14 8:14:24.868318 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96AD0 synchronization task has been failed!
  2853. 12/14 8:14:24.868326 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D85E8 has failed due to unidentified error.
  2854. 12/14 8:14:24.868332 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2855. 12/14 8:14:24.868337 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F956B0 synchronization task has been failed!
  2856. 12/14 8:14:24.868346 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E8699A68 has failed due to unidentified error.
  2857. 12/14 8:14:24.868353 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2858. 12/14 8:14:24.868358 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F98790 synchronization task has been failed!
  2859. 12/14 8:14:24.868404 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6258 has failed due to unidentified error.
  2860. 12/14 8:14:24.868417 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2861. 12/14 8:14:24.868426 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F984B0 synchronization task has been failed!
  2862. 12/14 8:14:24.868435 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BE08 has failed due to unidentified error.
  2863. 12/14 8:14:24.868442 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2864. 12/14 8:14:24.868446 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F95C70 synchronization task has been failed!
  2865. 12/14 8:14:24.868459 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7BE8 has failed due to unidentified error.
  2866. 12/14 8:14:24.868468 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2867. 12/14 8:14:24.868475 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97930 synchronization task has been failed!
  2868. 12/14 8:14:24.868487 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86993E8 has failed due to unidentified error.
  2869. 12/14 8:14:24.868495 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2870. 12/14 8:14:24.868500 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97650 synchronization task has been failed!
  2871. 12/14 8:14:24.884903 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (16 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 18778182, expcmdsn = 18778180, maxcmdsn = 18778419.
  2872. 12/14 8:14:24.884960 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 18778178, itt = 18778267)
  2873. 12/14 8:14:24.884982 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 18778179, itt = 18778268)
  2874. 12/14 8:14:24.885000 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 18778180, itt = 18778269)
  2875. 12/14 8:14:24.885019 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BF88 has failed due to unidentified error.
  2876. 12/14 8:14:24.885037 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2877. 12/14 8:14:24.885050 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97EF0 synchronization task has been failed!
  2878. 12/14 8:14:24.885082 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8268 has failed due to unidentified error.
  2879. 12/14 8:14:24.885098 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2880. 12/14 8:14:24.885110 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97370 synchronization task has been failed!
  2881. 12/14 8:14:24.885133 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6758 has failed due to unidentified error.
  2882. 12/14 8:14:24.885150 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2883. 12/14 8:14:24.885161 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96DB0 synchronization task has been failed!
  2884. 12/14 8:14:29.695498 65cc HA: CHAPartnerNode::AfterSynchronization: Event: Full synchronization has been completed (error: 1237)
  2885. 12/14 8:14:29.698809 38d8 func: >>> HANode::onScsiOp_CustomControl_HeartbeatPing
  2886. 12/14 8:14:29.698983 38d8 Common: CStarWindStorageDevice::getLastPerfDegEventElapsedTimeMs: perfDegEventElapsedTime = -1
  2887. 12/14 8:14:29.699006 38d8 func: <<< HANode::onScsiOp_CustomControl_HeartbeatPing
  2888. 12/14 8:14:29.699407 5378 func: >>> HANode::reportEvent
  2889. 12/14 8:14:29.699467 5378 func: >>> iScsiServer::sendNotification
  2890. 12/14 8:14:29.699488 5378 func: >>> CEventDataBase::AddRecord
  2891. 12/14 8:14:29.699507 5378 EventDB: CEventDataBase::AddRecord: Code 783, severity 1, additional strings: 2
  2892. 12/14 8:14:29.699680 5378 func: <<< CEventDataBase::AddRecord
  2893. 12/14 8:14:29.699706 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 783).
  2894. 12/14 8:14:29.700715 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  2895. 12/14 8:14:29.700764 5378 func: <<< iScsiServer::sendNotification
  2896. 12/14 8:14:29.700786 5378 func: <<< HANode::reportEvent
  2897. 12/14 8:14:29.700804 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_COMPLETE_WITH_ERROR, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'
  2898. 12/14 8:14:29.702185 38d8 func: >>> CHAPartnerNode::ShutdownNode
  2899. 12/14 8:14:29.702248 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  2900. 12/14 8:14:29.702270 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  2901. 12/14 8:14:29.702291 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  2902. 12/14 8:14:29.702309 38d8 func: <<< CHAPartnerNode::ShutdownNode
  2903. 12/14 8:14:31.334112 38d8 func: >>> CHAPartnerNode::ShutdownNode
  2904. 12/14 8:14:31.334167 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  2905. 12/14 8:14:31.334178 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  2906. 12/14 8:14:31.334198 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  2907. 12/14 8:14:31.334207 38d8 func: <<< CHAPartnerNode::ShutdownNode
  2908. 12/14 8:14:31.339079 2d00 HA: HANode::onScsiOp_CustomControl_SynchronizationRequest: The partner iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03 has requested the 1 synchronization type.
  2909. 12/14 8:14:31.340977 2d00 HA: CHAPartnerNode::Synchronize: ENTERed with params: p_SyncType = 1, p_ulSnapshotID = 0(0x0), p_pStorageDevice = 0x000002C7ECEDC4B0, p_pSyncBarrier = 0x000002C7ECEDC858, p_puliStartSector = 0x000000D2528FF118 (value is 0x0), p_puliEndSector = 0x000000D2528FF110 (value is 0xAEFFFFF), p_ulIOBlockSizeInSectors = 64, p_ulSectorSizeInUCHARs = 4096, p_bContinue = no, p_bVaaiWriteSameSupport = yes
  2910. 12/14 8:14:31.341031 2d00 HA: HANode::updatePartnerStatus: Statuses is already actual, we don't need to update them.
  2911. 12/14 8:14:31.341059 2d00 func: >>> CHAPartnerNode::SetSyncStatus
  2912. 12/14 8:14:31.341078 2d00 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  2913. 12/14 8:14:31.341097 2d00 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 2 from 3. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  2914. 12/14 8:14:31.341115 2d00 func: >>> HANode::reportEvent
  2915. 12/14 8:14:31.341135 2d00 func: >>> iScsiServer::sendNotification
  2916. 12/14 8:14:31.341154 2d00 func: >>> CEventDataBase::AddRecord
  2917. 12/14 8:14:31.341171 2d00 EventDB: CEventDataBase::AddRecord: Code 770, severity 2, additional strings: 2
  2918. 12/14 8:14:31.341244 2d00 func: <<< CEventDataBase::AddRecord
  2919. 12/14 8:14:31.341262 2d00 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 770).
  2920. 12/14 8:14:31.341989 2d00 Srv: iScsiServer::sendNotification: Send reaction finish.
  2921. 12/14 8:14:31.342027 2d00 func: <<< iScsiServer::sendNotification
  2922. 12/14 8:14:31.342045 2d00 func: <<< HANode::reportEvent
  2923. 12/14 8:14:31.342062 2d00 func: <<< CHAPartnerNode::SetSyncStatus
  2924. 12/14 8:14:31.342331 2690 func: >>> CHAPartnerNode::SetSyncStatus
  2925. 12/14 8:14:31.342356 2690 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  2926. 12/14 8:14:31.342365 2690 func: <<< CHAPartnerNode::SetSyncStatus
  2927. 12/14 8:14:31.346541 5378 func: >>> HANode::reportEvent
  2928. 12/14 8:14:31.346601 5378 func: >>> iScsiServer::sendNotification
  2929. 12/14 8:14:31.346623 5378 func: >>> CEventDataBase::AddRecord
  2930. 12/14 8:14:31.346643 5378 EventDB: CEventDataBase::AddRecord: Code 780, severity 1, additional strings: 3
  2931. 12/14 8:14:31.346721 5378 func: <<< CEventDataBase::AddRecord
  2932. 12/14 8:14:31.346741 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 780).
  2933. 12/14 8:14:31.347616 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  2934. 12/14 8:14:31.347662 5378 func: <<< iScsiServer::sendNotification
  2935. 12/14 8:14:31.347684 5378 func: <<< HANode::reportEvent
  2936. 12/14 8:14:31.347703 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_STARTED, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03', Synchronization type: 'Full'
  2937. 12/14 8:37:06.891523 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (9969 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 19238304, expcmdsn = 19238303, maxcmdsn = 19238542.
  2938. 12/14 8:37:06.891656 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 19238288, itt = 19238377)
  2939. 12/14 8:37:06.891679 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 19238289, itt = 19238378)
  2940. 12/14 8:37:06.891695 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 19238290, itt = 19238379)
  2941. 12/14 8:37:06.891708 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 19238291, itt = 19238380)
  2942. 12/14 8:37:06.891734 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 19238292, itt = 19238381)
  2943. 12/14 8:37:06.891748 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 19238293, itt = 19238382)
  2944. 12/14 8:37:06.891765 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6758 has failed due to unidentified error.
  2945. 12/14 8:37:06.891780 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2946. 12/14 8:37:06.891792 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F956B0 synchronization task has been failed!
  2947. 12/14 8:37:06.891823 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7BE8 has failed due to unidentified error.
  2948. 12/14 8:37:06.891836 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2949. 12/14 8:37:06.891845 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F984B0 synchronization task has been failed!
  2950. 12/14 8:37:06.891862 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D85E8 has failed due to unidentified error.
  2951. 12/14 8:37:06.891874 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2952. 12/14 8:37:06.891884 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97C10 synchronization task has been failed!
  2953. 12/14 8:37:06.891882 2690 func: >>> CHAPartnerNode::SetSyncStatus
  2954. 12/14 8:37:06.891959 2690 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  2955. 12/14 8:37:06.891987 2690 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 3 from 2. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  2956. 12/14 8:37:06.891900 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6258 has failed due to unidentified error.
  2957. 12/14 8:37:06.892042 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2958. 12/14 8:37:06.892055 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F95990 synchronization task has been failed!
  2959. 12/14 8:37:06.892009 2690 func: >>> HANode::reportEvent
  2960. 12/14 8:37:06.892077 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8268 has failed due to unidentified error.
  2961. 12/14 8:37:06.892123 2690 func: >>> iScsiServer::sendNotification
  2962. 12/14 8:37:06.892176 2690 func: >>> CEventDataBase::AddRecord
  2963. 12/14 8:37:06.892199 2690 EventDB: CEventDataBase::AddRecord: Code 771, severity 3, additional strings: 2
  2964. 12/14 8:37:06.892144 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2965. 12/14 8:37:06.892238 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96DB0 synchronization task has been failed!
  2966. 12/14 8:37:06.892261 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86993E8 has failed due to unidentified error.
  2967. 12/14 8:37:06.892275 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2968. 12/14 8:37:06.892285 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F95F50 synchronization task has been failed!
  2969. 12/14 8:37:06.892429 2690 func: <<< CEventDataBase::AddRecord
  2970. 12/14 8:37:06.892458 2690 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 771).
  2971. 12/14 8:37:06.893563 2690 Srv: iScsiServer::sendNotification: Send reaction finish.
  2972. 12/14 8:37:06.893606 2690 func: <<< iScsiServer::sendNotification
  2973. 12/14 8:37:06.893639 2690 func: <<< HANode::reportEvent
  2974. 12/14 8:37:06.893658 2690 func: <<< CHAPartnerNode::SetSyncStatus
  2975. 12/14 8:37:06.895164 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 19238294, itt = 19238383)
  2976. 12/14 8:37:06.895259 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 19238295, itt = 19238384)
  2977. 12/14 8:37:06.895283 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 19238296, itt = 19238385)
  2978. 12/14 8:37:06.895303 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 19238297, itt = 19238386)
  2979. 12/14 8:37:06.895323 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 19238298, itt = 19238387)
  2980. 12/14 8:37:06.895342 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 19238299, itt = 19238388)
  2981. 12/14 8:37:06.895361 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 19238300, itt = 19238389)
  2982. 12/14 8:37:06.895385 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE60D8 has failed due to unidentified error.
  2983. 12/14 8:37:06.895405 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2984. 12/14 8:37:06.895421 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F98D50 synchronization task has been failed!
  2985. 12/14 8:37:06.895457 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BF88 has failed due to unidentified error.
  2986. 12/14 8:37:06.895476 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2987. 12/14 8:37:06.895491 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96AD0 synchronization task has been failed!
  2988. 12/14 8:37:06.895518 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EC4F39D8 has failed due to unidentified error.
  2989. 12/14 8:37:06.895537 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2990. 12/14 8:37:06.895551 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97370 synchronization task has been failed!
  2991. 12/14 8:37:06.895577 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8568 has failed due to unidentified error.
  2992. 12/14 8:37:06.895596 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2993. 12/14 8:37:06.895615 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97090 synchronization task has been failed!
  2994. 12/14 8:37:06.895641 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86994E8 has failed due to unidentified error.
  2995. 12/14 8:37:06.895669 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2996. 12/14 8:37:06.895686 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96230 synchronization task has been failed!
  2997. 12/14 8:37:06.895720 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7D68 has failed due to unidentified error.
  2998. 12/14 8:37:06.895741 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  2999. 12/14 8:37:06.895755 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F953D0 synchronization task has been failed!
  3000. 12/14 8:37:06.895781 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86999E8 has failed due to unidentified error.
  3001. 12/14 8:37:06.895800 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3002. 12/14 8:37:06.896094 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F967F0 synchronization task has been failed!
  3003. 12/14 8:37:06.911016 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (15 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 19238305, expcmdsn = 19238303, maxcmdsn = 19238542.
  3004. 12/14 8:37:06.911085 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 19238301, itt = 19238390)
  3005. 12/14 8:37:06.911120 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 19238302, itt = 19238391)
  3006. 12/14 8:37:06.911143 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 19238303, itt = 19238392)
  3007. 12/14 8:37:06.911167 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BE08 has failed due to unidentified error.
  3008. 12/14 8:37:06.911188 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3009. 12/14 8:37:06.911204 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F95C70 synchronization task has been failed!
  3010. 12/14 8:37:06.911242 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86991E8 has failed due to unidentified error.
  3011. 12/14 8:37:06.911261 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3012. 12/14 8:37:06.911275 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96510 synchronization task has been failed!
  3013. 12/14 8:37:06.911302 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6158 has failed due to unidentified error.
  3014. 12/14 8:37:06.911322 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3015. 12/14 8:37:06.911336 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F950F0 synchronization task has been failed!
  3016. 12/14 8:37:11.660964 2690 HA: CHAPartnerNode::AfterSynchronization: Event: Full synchronization has been completed (error: 1237)
  3017. 12/14 8:37:11.662727 38d8 func: >>> HANode::onScsiOp_CustomControl_HeartbeatPing
  3018. 12/14 8:37:11.662905 38d8 Common: CStarWindStorageDevice::getLastPerfDegEventElapsedTimeMs: perfDegEventElapsedTime = -1
  3019. 12/14 8:37:11.662928 38d8 func: <<< HANode::onScsiOp_CustomControl_HeartbeatPing
  3020. 12/14 8:37:11.665094 5378 func: >>> HANode::reportEvent
  3021. 12/14 8:37:11.665155 5378 func: >>> iScsiServer::sendNotification
  3022. 12/14 8:37:11.665176 5378 func: >>> CEventDataBase::AddRecord
  3023. 12/14 8:37:11.665193 5378 EventDB: CEventDataBase::AddRecord: Code 783, severity 1, additional strings: 2
  3024. 12/14 8:37:11.665270 5378 func: <<< CEventDataBase::AddRecord
  3025. 12/14 8:37:11.665290 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 783).
  3026. 12/14 8:37:11.667965 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  3027. 12/14 8:37:11.668008 5378 func: <<< iScsiServer::sendNotification
  3028. 12/14 8:37:11.668024 5378 func: <<< HANode::reportEvent
  3029. 12/14 8:37:11.668037 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_COMPLETE_WITH_ERROR, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'
  3030. 12/14 8:37:12.669697 38d8 func: >>> CHAPartnerNode::ShutdownNode
  3031. 12/14 8:37:12.669759 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  3032. 12/14 8:37:12.669780 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  3033. 12/14 8:37:12.669799 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  3034. 12/14 8:37:12.669815 38d8 func: <<< CHAPartnerNode::ShutdownNode
  3035. 12/14 8:37:13.750624 38d8 func: >>> CHAPartnerNode::ShutdownNode
  3036. 12/14 8:37:13.750694 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  3037. 12/14 8:37:13.750723 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  3038. 12/14 8:37:13.750745 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  3039. 12/14 8:37:13.750763 38d8 func: <<< CHAPartnerNode::ShutdownNode
  3040. 12/14 8:37:13.756137 2d00 HA: HANode::onScsiOp_CustomControl_SynchronizationRequest: The partner iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03 has requested the 1 synchronization type.
  3041. 12/14 8:37:13.758004 2d00 HA: CHAPartnerNode::Synchronize: ENTERed with params: p_SyncType = 1, p_ulSnapshotID = 0(0x0), p_pStorageDevice = 0x000002C7ECEDC4B0, p_pSyncBarrier = 0x000002C7ECEDC858, p_puliStartSector = 0x000000D2528FF118 (value is 0x0), p_puliEndSector = 0x000000D2528FF110 (value is 0xAEFFFFF), p_ulIOBlockSizeInSectors = 64, p_ulSectorSizeInUCHARs = 4096, p_bContinue = no, p_bVaaiWriteSameSupport = yes
  3042. 12/14 8:37:13.758040 2d00 HA: HANode::updatePartnerStatus: Statuses is already actual, we don't need to update them.
  3043. 12/14 8:37:13.758120 2d00 func: >>> CHAPartnerNode::SetSyncStatus
  3044. 12/14 8:37:13.758133 2d00 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  3045. 12/14 8:37:13.758144 2d00 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 2 from 3. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  3046. 12/14 8:37:13.758156 2d00 func: >>> HANode::reportEvent
  3047. 12/14 8:37:13.758169 2d00 func: >>> iScsiServer::sendNotification
  3048. 12/14 8:37:13.758181 2d00 func: >>> CEventDataBase::AddRecord
  3049. 12/14 8:37:13.758191 2d00 EventDB: CEventDataBase::AddRecord: Code 770, severity 2, additional strings: 2
  3050. 12/14 8:37:13.758244 2d00 func: <<< CEventDataBase::AddRecord
  3051. 12/14 8:37:13.758255 2d00 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 770).
  3052. 12/14 8:37:13.758817 2d00 Srv: iScsiServer::sendNotification: Send reaction finish.
  3053. 12/14 8:37:13.758844 2d00 func: <<< iScsiServer::sendNotification
  3054. 12/14 8:37:13.758856 2d00 func: <<< HANode::reportEvent
  3055. 12/14 8:37:13.758867 2d00 func: <<< CHAPartnerNode::SetSyncStatus
  3056. 12/14 8:37:13.759214 590c func: >>> CHAPartnerNode::SetSyncStatus
  3057. 12/14 8:37:13.759280 590c HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  3058. 12/14 8:37:13.759302 590c func: <<< CHAPartnerNode::SetSyncStatus
  3059. 12/14 8:37:13.767114 5378 func: >>> HANode::reportEvent
  3060. 12/14 8:37:13.767155 5378 func: >>> iScsiServer::sendNotification
  3061. 12/14 8:37:13.767166 5378 func: >>> CEventDataBase::AddRecord
  3062. 12/14 8:37:13.767174 5378 EventDB: CEventDataBase::AddRecord: Code 780, severity 1, additional strings: 3
  3063. 12/14 8:37:13.767221 5378 func: <<< CEventDataBase::AddRecord
  3064. 12/14 8:37:13.767230 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 780).
  3065. 12/14 8:37:13.767791 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  3066. 12/14 8:37:13.767816 5378 func: <<< iScsiServer::sendNotification
  3067. 12/14 8:37:13.767825 5378 func: <<< HANode::reportEvent
  3068. 12/14 8:37:13.767834 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_STARTED, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03', Synchronization type: 'Full'
  3069. 12/14 8:59:52.113451 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (9968 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 19699166, expcmdsn = 19699165, maxcmdsn = 19699404.
  3070. 12/14 8:59:52.113528 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 19699150, itt = 19699239)
  3071. 12/14 8:59:52.113554 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 19699151, itt = 19699240)
  3072. 12/14 8:59:52.113576 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 19699152, itt = 19699241)
  3073. 12/14 8:59:52.113612 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 19699153, itt = 19699242)
  3074. 12/14 8:59:52.113632 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 19699154, itt = 19699243)
  3075. 12/14 8:59:52.113659 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8268 has failed due to unidentified error.
  3076. 12/14 8:59:52.113680 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3077. 12/14 8:59:52.113698 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F981D0 synchronization task has been failed!
  3078. 12/14 8:59:52.113740 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D85E8 has failed due to unidentified error.
  3079. 12/14 8:59:52.113759 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3080. 12/14 8:59:52.113773 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F95C70 synchronization task has been failed!
  3081. 12/14 8:59:52.113797 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE60D8 has failed due to unidentified error.
  3082. 12/14 8:59:52.113801 590c func: >>> CHAPartnerNode::SetSyncStatus
  3083. 12/14 8:59:52.113880 590c HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  3084. 12/14 8:59:52.113909 590c HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 3 from 2. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  3085. 12/14 8:59:52.113934 590c func: >>> HANode::reportEvent
  3086. 12/14 8:59:52.113817 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3087. 12/14 8:59:52.113983 590c func: >>> iScsiServer::sendNotification
  3088. 12/14 8:59:52.114027 590c func: >>> CEventDataBase::AddRecord
  3089. 12/14 8:59:52.114055 590c EventDB: CEventDataBase::AddRecord: Code 771, severity 3, additional strings: 2
  3090. 12/14 8:59:52.113995 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97090 synchronization task has been failed!
  3091. 12/14 8:59:52.114119 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E8699A68 has failed due to unidentified error.
  3092. 12/14 8:59:52.114136 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3093. 12/14 8:59:52.114141 590c func: <<< CEventDataBase::AddRecord
  3094. 12/14 8:59:52.114170 590c Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 771).
  3095. 12/14 8:59:52.114147 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97930 synchronization task has been failed!
  3096. 12/14 8:59:52.114214 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6158 has failed due to unidentified error.
  3097. 12/14 8:59:52.114226 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3098. 12/14 8:59:52.114235 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97C10 synchronization task has been failed!
  3099. 12/14 8:59:52.115172 590c Srv: iScsiServer::sendNotification: Send reaction finish.
  3100. 12/14 8:59:52.115214 590c func: <<< iScsiServer::sendNotification
  3101. 12/14 8:59:52.115234 590c func: <<< HANode::reportEvent
  3102. 12/14 8:59:52.115252 590c func: <<< CHAPartnerNode::SetSyncStatus
  3103. 12/14 8:59:52.133123 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (16 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 19699167, expcmdsn = 19699165, maxcmdsn = 19699404.
  3104. 12/14 8:59:52.133183 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 19699155, itt = 19699244)
  3105. 12/14 8:59:52.133198 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 19699156, itt = 19699245)
  3106. 12/14 8:59:52.133210 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 19699157, itt = 19699246)
  3107. 12/14 8:59:52.133220 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 19699158, itt = 19699247)
  3108. 12/14 8:59:52.133231 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 19699159, itt = 19699248)
  3109. 12/14 8:59:52.133240 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 19699160, itt = 19699249)
  3110. 12/14 8:59:52.133250 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 19699161, itt = 19699250)
  3111. 12/14 8:59:52.133261 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 19699162, itt = 19699251)
  3112. 12/14 8:59:52.133271 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 19699163, itt = 19699252)
  3113. 12/14 8:59:52.133284 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86993E8 has failed due to unidentified error.
  3114. 12/14 8:59:52.133294 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3115. 12/14 8:59:52.133303 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97650 synchronization task has been failed!
  3116. 12/14 8:59:52.133326 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86994E8 has failed due to unidentified error.
  3117. 12/14 8:59:52.133335 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3118. 12/14 8:59:52.133343 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96230 synchronization task has been failed!
  3119. 12/14 8:59:52.133357 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7D68 has failed due to unidentified error.
  3120. 12/14 8:59:52.133366 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3121. 12/14 8:59:52.133374 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F967F0 synchronization task has been failed!
  3122. 12/14 8:59:52.133388 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EC4F39D8 has failed due to unidentified error.
  3123. 12/14 8:59:52.133397 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3124. 12/14 8:59:52.133404 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96510 synchronization task has been failed!
  3125. 12/14 8:59:52.133418 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8568 has failed due to unidentified error.
  3126. 12/14 8:59:52.133427 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3127. 12/14 8:59:52.133433 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96AD0 synchronization task has been failed!
  3128. 12/14 8:59:52.133446 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6258 has failed due to unidentified error.
  3129. 12/14 8:59:52.133455 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3130. 12/14 8:59:52.133462 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F95990 synchronization task has been failed!
  3131. 12/14 8:59:52.133476 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BE08 has failed due to unidentified error.
  3132. 12/14 8:59:52.133488 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3133. 12/14 8:59:52.133495 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F953D0 synchronization task has been failed!
  3134. 12/14 8:59:52.133509 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6758 has failed due to unidentified error.
  3135. 12/14 8:59:52.133518 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3136. 12/14 8:59:52.133525 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F95F50 synchronization task has been failed!
  3137. 12/14 8:59:52.133538 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86999E8 has failed due to unidentified error.
  3138. 12/14 8:59:52.133547 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3139. 12/14 8:59:52.133553 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97EF0 synchronization task has been failed!
  3140. 12/14 8:59:52.150168 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (16 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 19699167, expcmdsn = 19699165, maxcmdsn = 19699404.
  3141. 12/14 8:59:52.150266 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 19699164, itt = 19699253)
  3142. 12/14 8:59:52.150292 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 19699165, itt = 19699254)
  3143. 12/14 8:59:52.150318 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7BE8 has failed due to unidentified error.
  3144. 12/14 8:59:52.150340 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3145. 12/14 8:59:52.150357 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F98D50 synchronization task has been failed!
  3146. 12/14 8:59:52.150397 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BF88 has failed due to unidentified error.
  3147. 12/14 8:59:52.150416 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3148. 12/14 8:59:52.150431 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F956B0 synchronization task has been failed!
  3149. 12/14 8:59:56.828576 590c HA: CHAPartnerNode::AfterSynchronization: Event: Full synchronization has been completed (error: 1237)
  3150. 12/14 8:59:56.831475 5378 func: >>> HANode::reportEvent
  3151. 12/14 8:59:56.831532 5378 func: >>> iScsiServer::sendNotification
  3152. 12/14 8:59:56.831554 5378 func: >>> CEventDataBase::AddRecord
  3153. 12/14 8:59:56.831573 5378 EventDB: CEventDataBase::AddRecord: Code 783, severity 1, additional strings: 2
  3154. 12/14 8:59:56.831654 5378 func: <<< CEventDataBase::AddRecord
  3155. 12/14 8:59:56.831673 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 783).
  3156. 12/14 8:59:56.832669 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  3157. 12/14 8:59:56.832713 5378 func: <<< iScsiServer::sendNotification
  3158. 12/14 8:59:56.832734 5378 func: <<< HANode::reportEvent
  3159. 12/14 8:59:56.832752 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_COMPLETE_WITH_ERROR, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'
  3160. 12/14 8:59:56.836271 38d8 func: >>> HANode::onScsiOp_CustomControl_HeartbeatPing
  3161. 12/14 8:59:56.836385 38d8 Common: CStarWindStorageDevice::getLastPerfDegEventElapsedTimeMs: perfDegEventElapsedTime = -1
  3162. 12/14 8:59:56.836397 38d8 func: <<< HANode::onScsiOp_CustomControl_HeartbeatPing
  3163. 12/14 8:59:56.837189 38d8 func: >>> CHAPartnerNode::ShutdownNode
  3164. 12/14 8:59:56.837226 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  3165. 12/14 8:59:56.837242 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  3166. 12/14 8:59:56.837254 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  3167. 12/14 8:59:56.837264 38d8 func: <<< CHAPartnerNode::ShutdownNode
  3168. 12/14 9:00:01.251614 38d8 func: >>> CHAPartnerNode::ShutdownNode
  3169. 12/14 9:00:01.251695 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  3170. 12/14 9:00:01.251725 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  3171. 12/14 9:00:01.251746 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  3172. 12/14 9:00:01.251764 38d8 func: <<< CHAPartnerNode::ShutdownNode
  3173. 12/14 9:00:01.256414 2d00 HA: HANode::onScsiOp_CustomControl_SynchronizationRequest: The partner iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03 has requested the 1 synchronization type.
  3174. 12/14 9:00:01.258945 2d00 HA: CHAPartnerNode::Synchronize: ENTERed with params: p_SyncType = 1, p_ulSnapshotID = 0(0x0), p_pStorageDevice = 0x000002C7ECEDC4B0, p_pSyncBarrier = 0x000002C7ECEDC858, p_puliStartSector = 0x000000D2528FF118 (value is 0x0), p_puliEndSector = 0x000000D2528FF110 (value is 0xAEFFFFF), p_ulIOBlockSizeInSectors = 64, p_ulSectorSizeInUCHARs = 4096, p_bContinue = no, p_bVaaiWriteSameSupport = yes
  3175. 12/14 9:00:01.258999 2d00 HA: HANode::updatePartnerStatus: Statuses is already actual, we don't need to update them.
  3176. 12/14 9:00:01.259027 2d00 func: >>> CHAPartnerNode::SetSyncStatus
  3177. 12/14 9:00:01.259045 2d00 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  3178. 12/14 9:00:01.259063 2d00 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 2 from 3. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  3179. 12/14 9:00:01.259081 2d00 func: >>> HANode::reportEvent
  3180. 12/14 9:00:01.259100 2d00 func: >>> iScsiServer::sendNotification
  3181. 12/14 9:00:01.259117 2d00 func: >>> CEventDataBase::AddRecord
  3182. 12/14 9:00:01.259133 2d00 EventDB: CEventDataBase::AddRecord: Code 770, severity 2, additional strings: 2
  3183. 12/14 9:00:01.259204 2d00 func: <<< CEventDataBase::AddRecord
  3184. 12/14 9:00:01.259221 2d00 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 770).
  3185. 12/14 9:00:01.260089 2d00 Srv: iScsiServer::sendNotification: Send reaction finish.
  3186. 12/14 9:00:01.260128 2d00 func: <<< iScsiServer::sendNotification
  3187. 12/14 9:00:01.260146 2d00 func: <<< HANode::reportEvent
  3188. 12/14 9:00:01.260162 2d00 func: <<< CHAPartnerNode::SetSyncStatus
  3189. 12/14 9:00:01.260472 29a8 func: >>> CHAPartnerNode::SetSyncStatus
  3190. 12/14 9:00:01.260504 29a8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  3191. 12/14 9:00:01.260518 29a8 func: <<< CHAPartnerNode::SetSyncStatus
  3192. 12/14 9:00:01.263380 5378 func: >>> HANode::reportEvent
  3193. 12/14 9:00:01.263417 5378 func: >>> iScsiServer::sendNotification
  3194. 12/14 9:00:01.263429 5378 func: >>> CEventDataBase::AddRecord
  3195. 12/14 9:00:01.263439 5378 EventDB: CEventDataBase::AddRecord: Code 780, severity 1, additional strings: 3
  3196. 12/14 9:00:01.263487 5378 func: <<< CEventDataBase::AddRecord
  3197. 12/14 9:00:01.263497 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 780).
  3198. 12/14 9:00:01.264371 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  3199. 12/14 9:00:01.264395 5378 func: <<< iScsiServer::sendNotification
  3200. 12/14 9:00:01.264406 5378 func: <<< HANode::reportEvent
  3201. 12/14 9:00:01.264415 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_STARTED, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03', Synchronization type: 'Full'
  3202. 12/14 9:22:33.521276 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (9953 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 20157622, expcmdsn = 20157621, maxcmdsn = 20157860.
  3203. 12/14 9:22:33.521370 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 20157606, itt = 20157696)
  3204. 12/14 9:22:33.521381 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 20157607, itt = 20157697)
  3205. 12/14 9:22:33.521389 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 20157608, itt = 20157698)
  3206. 12/14 9:22:33.521397 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 20157609, itt = 20157699)
  3207. 12/14 9:22:33.521405 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 20157610, itt = 20157700)
  3208. 12/14 9:22:33.521419 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86994E8 has failed due to unidentified error.
  3209. 12/14 9:22:33.521427 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3210. 12/14 9:22:33.521434 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96510 synchronization task has been failed!
  3211. 12/14 9:22:33.521454 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E8699A68 has failed due to unidentified error.
  3212. 12/14 9:22:33.521461 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3213. 12/14 9:22:33.521466 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F953D0 synchronization task has been failed!
  3214. 12/14 9:22:33.521475 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BF88 has failed due to unidentified error.
  3215. 12/14 9:22:33.521482 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3216. 12/14 9:22:33.521487 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F967F0 synchronization task has been failed!
  3217. 12/14 9:22:33.521496 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8568 has failed due to unidentified error.
  3218. 12/14 9:22:33.521492 29a8 func: >>> CHAPartnerNode::SetSyncStatus
  3219. 12/14 9:22:33.521546 29a8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  3220. 12/14 9:22:33.521568 29a8 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 3 from 2. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  3221. 12/14 9:22:33.521589 29a8 func: >>> HANode::reportEvent
  3222. 12/14 9:22:33.521503 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3223. 12/14 9:22:33.521618 29a8 func: >>> iScsiServer::sendNotification
  3224. 12/14 9:22:33.521643 29a8 func: >>> CEventDataBase::AddRecord
  3225. 12/14 9:22:33.521660 29a8 EventDB: CEventDataBase::AddRecord: Code 771, severity 3, additional strings: 2
  3226. 12/14 9:22:33.521626 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97370 synchronization task has been failed!
  3227. 12/14 9:22:33.521702 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86991E8 has failed due to unidentified error.
  3228. 12/14 9:22:33.521711 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3229. 12/14 9:22:33.521717 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97EF0 synchronization task has been failed!
  3230. 12/14 9:22:33.521826 29a8 func: <<< CEventDataBase::AddRecord
  3231. 12/14 9:22:33.521843 29a8 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 771).
  3232. 12/14 9:22:33.522894 29a8 Srv: iScsiServer::sendNotification: Send reaction finish.
  3233. 12/14 9:22:33.522938 29a8 func: <<< iScsiServer::sendNotification
  3234. 12/14 9:22:33.522959 29a8 func: <<< HANode::reportEvent
  3235. 12/14 9:22:33.522977 29a8 func: <<< CHAPartnerNode::SetSyncStatus
  3236. 12/14 9:22:33.541927 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (16 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 20157623, expcmdsn = 20157621, maxcmdsn = 20157860.
  3237. 12/14 9:22:33.541961 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 20157611, itt = 20157701)
  3238. 12/14 9:22:33.541970 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 20157612, itt = 20157702)
  3239. 12/14 9:22:33.541978 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 20157613, itt = 20157703)
  3240. 12/14 9:22:33.541985 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 20157614, itt = 20157704)
  3241. 12/14 9:22:33.541993 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 20157615, itt = 20157705)
  3242. 12/14 9:22:33.542004 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE60D8 has failed due to unidentified error.
  3243. 12/14 9:22:33.542011 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3244. 12/14 9:22:33.542017 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97090 synchronization task has been failed!
  3245. 12/14 9:22:33.542035 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6758 has failed due to unidentified error.
  3246. 12/14 9:22:33.542044 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3247. 12/14 9:22:33.542049 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F984B0 synchronization task has been failed!
  3248. 12/14 9:22:33.542060 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EC4F39D8 has failed due to unidentified error.
  3249. 12/14 9:22:33.542067 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3250. 12/14 9:22:33.542071 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96230 synchronization task has been failed!
  3251. 12/14 9:22:33.542082 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6258 has failed due to unidentified error.
  3252. 12/14 9:22:33.542089 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3253. 12/14 9:22:33.542093 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F95F50 synchronization task has been failed!
  3254. 12/14 9:22:33.542103 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BE08 has failed due to unidentified error.
  3255. 12/14 9:22:33.542109 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3256. 12/14 9:22:33.542114 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F981D0 synchronization task has been failed!
  3257. 12/14 9:22:33.557012 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (15 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 20157623, expcmdsn = 20157621, maxcmdsn = 20157860.
  3258. 12/14 9:22:33.557047 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 20157616, itt = 20157706)
  3259. 12/14 9:22:33.557057 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 20157617, itt = 20157707)
  3260. 12/14 9:22:33.557066 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 20157618, itt = 20157708)
  3261. 12/14 9:22:33.557082 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 20157619, itt = 20157709)
  3262. 12/14 9:22:33.557090 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 20157620, itt = 20157710)
  3263. 12/14 9:22:33.557101 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7BE8 has failed due to unidentified error.
  3264. 12/14 9:22:33.557109 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3265. 12/14 9:22:33.557115 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F95C70 synchronization task has been failed!
  3266. 12/14 9:22:33.557130 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6158 has failed due to unidentified error.
  3267. 12/14 9:22:33.557137 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3268. 12/14 9:22:33.557143 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96AD0 synchronization task has been failed!
  3269. 12/14 9:22:33.557154 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86999E8 has failed due to unidentified error.
  3270. 12/14 9:22:33.557161 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3271. 12/14 9:22:33.557166 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F95990 synchronization task has been failed!
  3272. 12/14 9:22:33.557177 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7D68 has failed due to unidentified error.
  3273. 12/14 9:22:33.557183 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3274. 12/14 9:22:33.557188 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F956B0 synchronization task has been failed!
  3275. 12/14 9:22:33.557199 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86993E8 has failed due to unidentified error.
  3276. 12/14 9:22:33.557206 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3277. 12/14 9:22:33.557211 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96DB0 synchronization task has been failed!
  3278. 12/14 9:22:33.573045 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (16 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 20157623, expcmdsn = 20157621, maxcmdsn = 20157860.
  3279. 12/14 9:22:33.573092 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 20157621, itt = 20157711)
  3280. 12/14 9:22:33.573111 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8268 has failed due to unidentified error.
  3281. 12/14 9:22:33.573125 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3282. 12/14 9:22:33.573136 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F98790 synchronization task has been failed!
  3283. 12/14 9:22:38.273345 29a8 HA: CHAPartnerNode::AfterSynchronization: Event: Full synchronization has been completed (error: 1237)
  3284. 12/14 9:22:38.275805 5378 func: >>> HANode::reportEvent
  3285. 12/14 9:22:38.275836 5378 func: >>> iScsiServer::sendNotification
  3286. 12/14 9:22:38.275844 5378 func: >>> CEventDataBase::AddRecord
  3287. 12/14 9:22:38.275851 5378 EventDB: CEventDataBase::AddRecord: Code 783, severity 1, additional strings: 2
  3288. 12/14 9:22:38.275890 5378 func: <<< CEventDataBase::AddRecord
  3289. 12/14 9:22:38.275897 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 783).
  3290. 12/14 9:22:38.276497 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  3291. 12/14 9:22:38.276525 5378 func: <<< iScsiServer::sendNotification
  3292. 12/14 9:22:38.276533 5378 func: <<< HANode::reportEvent
  3293. 12/14 9:22:38.276539 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_COMPLETE_WITH_ERROR, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'
  3294. 12/14 9:22:38.287804 38d8 func: >>> HANode::onScsiOp_CustomControl_HeartbeatPing
  3295. 12/14 9:22:38.287902 38d8 Common: CStarWindStorageDevice::getLastPerfDegEventElapsedTimeMs: perfDegEventElapsedTime = -1
  3296. 12/14 9:22:38.287914 38d8 func: <<< HANode::onScsiOp_CustomControl_HeartbeatPing
  3297. 12/14 9:22:38.288673 38d8 func: >>> CHAPartnerNode::ShutdownNode
  3298. 12/14 9:22:38.288694 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  3299. 12/14 9:22:38.288703 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  3300. 12/14 9:22:38.288713 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  3301. 12/14 9:22:38.288722 38d8 func: <<< CHAPartnerNode::ShutdownNode
  3302. 12/14 9:22:38.874305 38d8 func: >>> CHAPartnerNode::ShutdownNode
  3303. 12/14 9:22:38.874368 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  3304. 12/14 9:22:38.874390 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  3305. 12/14 9:22:38.874411 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  3306. 12/14 9:22:38.874429 38d8 func: <<< CHAPartnerNode::ShutdownNode
  3307. 12/14 9:22:38.879276 2d00 HA: HANode::onScsiOp_CustomControl_SynchronizationRequest: The partner iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03 has requested the 1 synchronization type.
  3308. 12/14 9:22:38.881436 2d00 HA: CHAPartnerNode::Synchronize: ENTERed with params: p_SyncType = 1, p_ulSnapshotID = 0(0x0), p_pStorageDevice = 0x000002C7ECEDC4B0, p_pSyncBarrier = 0x000002C7ECEDC858, p_puliStartSector = 0x000000D2528FF118 (value is 0x0), p_puliEndSector = 0x000000D2528FF110 (value is 0xAEFFFFF), p_ulIOBlockSizeInSectors = 64, p_ulSectorSizeInUCHARs = 4096, p_bContinue = no, p_bVaaiWriteSameSupport = yes
  3309. 12/14 9:22:38.881486 2d00 HA: HANode::updatePartnerStatus: Statuses is already actual, we don't need to update them.
  3310. 12/14 9:22:38.881499 2d00 func: >>> CHAPartnerNode::SetSyncStatus
  3311. 12/14 9:22:38.881506 2d00 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  3312. 12/14 9:22:38.881513 2d00 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 2 from 3. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  3313. 12/14 9:22:38.881520 2d00 func: >>> HANode::reportEvent
  3314. 12/14 9:22:38.881553 2d00 func: >>> iScsiServer::sendNotification
  3315. 12/14 9:22:38.881561 2d00 func: >>> CEventDataBase::AddRecord
  3316. 12/14 9:22:38.881568 2d00 EventDB: CEventDataBase::AddRecord: Code 770, severity 2, additional strings: 2
  3317. 12/14 9:22:38.881605 2d00 func: <<< CEventDataBase::AddRecord
  3318. 12/14 9:22:38.881637 2d00 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 770).
  3319. 12/14 9:22:38.882202 2d00 Srv: iScsiServer::sendNotification: Send reaction finish.
  3320. 12/14 9:22:38.882281 2d00 func: <<< iScsiServer::sendNotification
  3321. 12/14 9:22:38.882290 2d00 func: <<< HANode::reportEvent
  3322. 12/14 9:22:38.882296 2d00 func: <<< CHAPartnerNode::SetSyncStatus
  3323. 12/14 9:22:38.882620 35e0 func: >>> CHAPartnerNode::SetSyncStatus
  3324. 12/14 9:22:38.882667 35e0 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  3325. 12/14 9:22:38.882688 35e0 func: <<< CHAPartnerNode::SetSyncStatus
  3326. 12/14 9:22:38.886146 5378 func: >>> HANode::reportEvent
  3327. 12/14 9:22:38.886184 5378 func: >>> iScsiServer::sendNotification
  3328. 12/14 9:22:38.886197 5378 func: >>> CEventDataBase::AddRecord
  3329. 12/14 9:22:38.886206 5378 EventDB: CEventDataBase::AddRecord: Code 780, severity 1, additional strings: 3
  3330. 12/14 9:22:38.886262 5378 func: <<< CEventDataBase::AddRecord
  3331. 12/14 9:22:38.886272 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 780).
  3332. 12/14 9:22:38.886850 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  3333. 12/14 9:22:38.886877 5378 func: <<< iScsiServer::sendNotification
  3334. 12/14 9:22:38.886888 5378 func: <<< HANode::reportEvent
  3335. 12/14 9:22:38.886897 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_STARTED, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03', Synchronization type: 'Full'
  3336. 12/14 9:45:15.511137 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (9875 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 20616276, expcmdsn = 20616265, maxcmdsn = 20616514.
  3337. 12/14 9:45:15.511222 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 20616260, itt = 20616350)
  3338. 12/14 9:45:15.511247 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 20616261, itt = 20616351)
  3339. 12/14 9:45:15.511269 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 20616262, itt = 20616352)
  3340. 12/14 9:45:15.511296 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86991E8 has failed due to unidentified error.
  3341. 12/14 9:45:15.511317 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3342. 12/14 9:45:15.511335 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97650 synchronization task has been failed!
  3343. 12/14 9:45:15.511375 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E8699A68 has failed due to unidentified error.
  3344. 12/14 9:45:15.511395 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3345. 12/14 9:45:15.511409 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97930 synchronization task has been failed!
  3346. 12/14 9:45:15.511433 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE60D8 has failed due to unidentified error.
  3347. 12/14 9:45:15.511460 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3348. 12/14 9:45:15.511427 35e0 func: >>> CHAPartnerNode::SetSyncStatus
  3349. 12/14 9:45:15.511541 35e0 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  3350. 12/14 9:45:15.511569 35e0 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 3 from 2. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  3351. 12/14 9:45:15.511595 35e0 func: >>> HANode::reportEvent
  3352. 12/14 9:45:15.511475 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F95990 synchronization task has been failed!
  3353. 12/14 9:45:15.511631 35e0 func: >>> iScsiServer::sendNotification
  3354. 12/14 9:45:15.511659 35e0 func: >>> CEventDataBase::AddRecord
  3355. 12/14 9:45:15.511677 35e0 EventDB: CEventDataBase::AddRecord: Code 771, severity 3, additional strings: 2
  3356. 12/14 9:45:15.511764 35e0 func: <<< CEventDataBase::AddRecord
  3357. 12/14 9:45:15.511783 35e0 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 771).
  3358. 12/14 9:45:15.512870 35e0 Srv: iScsiServer::sendNotification: Send reaction finish.
  3359. 12/14 9:45:15.512912 35e0 func: <<< iScsiServer::sendNotification
  3360. 12/14 9:45:15.512932 35e0 func: <<< HANode::reportEvent
  3361. 12/14 9:45:15.512950 35e0 func: <<< CHAPartnerNode::SetSyncStatus
  3362. 12/14 9:45:15.531257 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (16 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 20616277, expcmdsn = 20616265, maxcmdsn = 20616514.
  3363. 12/14 9:45:15.531360 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 20616263, itt = 20616353)
  3364. 12/14 9:45:15.531387 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 20616264, itt = 20616354)
  3365. 12/14 9:45:15.531413 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EC4F39D8 has failed due to unidentified error.
  3366. 12/14 9:45:15.531432 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3367. 12/14 9:45:15.531448 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96510 synchronization task has been failed!
  3368. 12/14 9:45:15.531484 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6258 has failed due to unidentified error.
  3369. 12/14 9:45:15.531502 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3370. 12/14 9:45:15.531516 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F950F0 synchronization task has been failed!
  3371. 12/14 9:45:15.625979 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (94 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 20616277, expcmdsn = 20616265, maxcmdsn = 20616514.
  3372. 12/14 9:45:15.626088 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 20616265, itt = 20616355)
  3373. 12/14 9:45:15.626118 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 20616266, itt = 20616356)
  3374. 12/14 9:45:15.626141 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 20616267, itt = 20616357)
  3375. 12/14 9:45:15.626161 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 20616268, itt = 20616358)
  3376. 12/14 9:45:15.626182 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 20616269, itt = 20616359)
  3377. 12/14 9:45:15.626202 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 20616270, itt = 20616360)
  3378. 12/14 9:45:15.626222 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 20616271, itt = 20616361)
  3379. 12/14 9:45:15.626247 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BF88 has failed due to unidentified error.
  3380. 12/14 9:45:15.626269 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3381. 12/14 9:45:15.626287 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97EF0 synchronization task has been failed!
  3382. 12/14 9:45:15.626328 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86994E8 has failed due to unidentified error.
  3383. 12/14 9:45:15.626348 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3384. 12/14 9:45:15.626369 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F956B0 synchronization task has been failed!
  3385. 12/14 9:45:15.626406 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8268 has failed due to unidentified error.
  3386. 12/14 9:45:15.626427 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3387. 12/14 9:45:15.626449 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97090 synchronization task has been failed!
  3388. 12/14 9:45:15.626484 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BE08 has failed due to unidentified error.
  3389. 12/14 9:45:15.626504 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3390. 12/14 9:45:15.626528 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F95F50 synchronization task has been failed!
  3391. 12/14 9:45:15.626561 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6758 has failed due to unidentified error.
  3392. 12/14 9:45:15.626581 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3393. 12/14 9:45:15.626594 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F98A70 synchronization task has been failed!
  3394. 12/14 9:45:15.626626 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86999E8 has failed due to unidentified error.
  3395. 12/14 9:45:15.626645 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3396. 12/14 9:45:15.626659 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F981D0 synchronization task has been failed!
  3397. 12/14 9:45:15.626685 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86993E8 has failed due to unidentified error.
  3398. 12/14 9:45:15.626704 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3399. 12/14 9:45:15.626720 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96DB0 synchronization task has been failed!
  3400. 12/14 9:45:15.641963 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (15 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 20616277, expcmdsn = 20616265, maxcmdsn = 20616514.
  3401. 12/14 9:45:15.642024 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 20616272, itt = 20616362)
  3402. 12/14 9:45:15.642043 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 20616273, itt = 20616363)
  3403. 12/14 9:45:15.642059 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 20616274, itt = 20616364)
  3404. 12/14 9:45:15.642073 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 20616275, itt = 20616365)
  3405. 12/14 9:45:15.642091 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7BE8 has failed due to unidentified error.
  3406. 12/14 9:45:15.642107 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3407. 12/14 9:45:15.642119 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96AD0 synchronization task has been failed!
  3408. 12/14 9:45:15.642148 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D85E8 has failed due to unidentified error.
  3409. 12/14 9:45:15.642162 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3410. 12/14 9:45:15.642172 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97C10 synchronization task has been failed!
  3411. 12/14 9:45:15.642192 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8568 has failed due to unidentified error.
  3412. 12/14 9:45:15.642205 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3413. 12/14 9:45:15.642216 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F967F0 synchronization task has been failed!
  3414. 12/14 9:45:15.642233 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6158 has failed due to unidentified error.
  3415. 12/14 9:45:15.642246 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3416. 12/14 9:45:15.642263 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F95C70 synchronization task has been failed!
  3417. 12/14 9:45:20.235389 35e0 HA: CHAPartnerNode::AfterSynchronization: Event: Full synchronization has been completed (error: 1237)
  3418. 12/14 9:45:20.237930 38d8 func: >>> HANode::onScsiOp_CustomControl_HeartbeatPing
  3419. 12/14 9:45:20.238103 38d8 Common: CStarWindStorageDevice::getLastPerfDegEventElapsedTimeMs: perfDegEventElapsedTime = -1
  3420. 12/14 9:45:20.238125 38d8 func: <<< HANode::onScsiOp_CustomControl_HeartbeatPing
  3421. 12/14 9:45:20.238366 5378 func: >>> HANode::reportEvent
  3422. 12/14 9:45:20.238413 5378 func: >>> iScsiServer::sendNotification
  3423. 12/14 9:45:20.238425 5378 func: >>> CEventDataBase::AddRecord
  3424. 12/14 9:45:20.238435 5378 EventDB: CEventDataBase::AddRecord: Code 783, severity 1, additional strings: 2
  3425. 12/14 9:45:20.238487 5378 func: <<< CEventDataBase::AddRecord
  3426. 12/14 9:45:20.238497 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 783).
  3427. 12/14 9:45:20.239017 38d8 func: >>> CHAPartnerNode::ShutdownNode
  3428. 12/14 9:45:20.239066 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  3429. 12/14 9:45:20.239088 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  3430. 12/14 9:45:20.239109 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  3431. 12/14 9:45:20.239127 38d8 func: <<< CHAPartnerNode::ShutdownNode
  3432. 12/14 9:45:20.239269 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  3433. 12/14 9:45:20.239304 5378 func: <<< iScsiServer::sendNotification
  3434. 12/14 9:45:20.239316 5378 func: <<< HANode::reportEvent
  3435. 12/14 9:45:20.239325 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_COMPLETE_WITH_ERROR, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'
  3436. 12/14 9:45:21.247049 38d8 func: >>> CHAPartnerNode::ShutdownNode
  3437. 12/14 9:45:21.247115 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  3438. 12/14 9:45:21.247141 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  3439. 12/14 9:45:21.247162 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  3440. 12/14 9:45:21.247181 38d8 func: <<< CHAPartnerNode::ShutdownNode
  3441. 12/14 9:45:21.251741 2d00 HA: HANode::onScsiOp_CustomControl_SynchronizationRequest: The partner iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03 has requested the 1 synchronization type.
  3442. 12/14 9:45:21.253428 2d00 HA: CHAPartnerNode::Synchronize: ENTERed with params: p_SyncType = 1, p_ulSnapshotID = 0(0x0), p_pStorageDevice = 0x000002C7ECEDC4B0, p_pSyncBarrier = 0x000002C7ECEDC858, p_puliStartSector = 0x000000D2528FF118 (value is 0x0), p_puliEndSector = 0x000000D2528FF110 (value is 0xAEFFFFF), p_ulIOBlockSizeInSectors = 64, p_ulSectorSizeInUCHARs = 4096, p_bContinue = no, p_bVaaiWriteSameSupport = yes
  3443. 12/14 9:45:21.253457 2d00 HA: HANode::updatePartnerStatus: Statuses is already actual, we don't need to update them.
  3444. 12/14 9:45:21.253473 2d00 func: >>> CHAPartnerNode::SetSyncStatus
  3445. 12/14 9:45:21.253483 2d00 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  3446. 12/14 9:45:21.253492 2d00 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 2 from 3. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  3447. 12/14 9:45:21.253502 2d00 func: >>> HANode::reportEvent
  3448. 12/14 9:45:21.253512 2d00 func: >>> iScsiServer::sendNotification
  3449. 12/14 9:45:21.253522 2d00 func: >>> CEventDataBase::AddRecord
  3450. 12/14 9:45:21.253530 2d00 EventDB: CEventDataBase::AddRecord: Code 770, severity 2, additional strings: 2
  3451. 12/14 9:45:21.253597 2d00 func: <<< CEventDataBase::AddRecord
  3452. 12/14 9:45:21.253607 2d00 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 770).
  3453. 12/14 9:45:21.255557 2d00 Srv: iScsiServer::sendNotification: Send reaction finish.
  3454. 12/14 9:45:21.255594 2d00 func: <<< iScsiServer::sendNotification
  3455. 12/14 9:45:21.255604 2d00 func: <<< HANode::reportEvent
  3456. 12/14 9:45:21.255613 2d00 func: <<< CHAPartnerNode::SetSyncStatus
  3457. 12/14 9:45:21.255909 6ec0 func: >>> CHAPartnerNode::SetSyncStatus
  3458. 12/14 9:45:21.255957 6ec0 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  3459. 12/14 9:45:21.255978 6ec0 func: <<< CHAPartnerNode::SetSyncStatus
  3460. 12/14 9:45:21.260368 5378 func: >>> HANode::reportEvent
  3461. 12/14 9:45:21.260409 5378 func: >>> iScsiServer::sendNotification
  3462. 12/14 9:45:21.260423 5378 func: >>> CEventDataBase::AddRecord
  3463. 12/14 9:45:21.260435 5378 EventDB: CEventDataBase::AddRecord: Code 780, severity 1, additional strings: 3
  3464. 12/14 9:45:21.260498 5378 func: <<< CEventDataBase::AddRecord
  3465. 12/14 9:45:21.260527 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 780).
  3466. 12/14 9:45:21.261158 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  3467. 12/14 9:45:21.261196 5378 func: <<< iScsiServer::sendNotification
  3468. 12/14 9:45:21.261209 5378 func: <<< HANode::reportEvent
  3469. 12/14 9:45:21.261221 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_STARTED, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03', Synchronization type: 'Full'
  3470. 12/14 10:07:55.525340 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (9860 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 21075223, expcmdsn = 21075219, maxcmdsn = 21075461.
  3471. 12/14 10:07:55.525462 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 21075207, itt = 21075297)
  3472. 12/14 10:07:55.525480 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 21075208, itt = 21075298)
  3473. 12/14 10:07:55.525492 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 21075209, itt = 21075299)
  3474. 12/14 10:07:55.525502 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 21075210, itt = 21075300)
  3475. 12/14 10:07:55.525512 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 21075211, itt = 21075301)
  3476. 12/14 10:07:55.525527 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86999E8 has failed due to unidentified error.
  3477. 12/14 10:07:55.525537 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3478. 12/14 10:07:55.525553 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F967F0 synchronization task has been failed!
  3479. 12/14 10:07:55.525583 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E8699A68 has failed due to unidentified error.
  3480. 12/14 10:07:55.525592 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3481. 12/14 10:07:55.525607 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96230 synchronization task has been failed!
  3482. 12/14 10:07:55.525619 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6758 has failed due to unidentified error.
  3483. 12/14 10:07:55.525628 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3484. 12/14 10:07:55.525625 6ec0 func: >>> CHAPartnerNode::SetSyncStatus
  3485. 12/14 10:07:55.525687 6ec0 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  3486. 12/14 10:07:55.525642 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F95C70 synchronization task has been failed!
  3487. 12/14 10:07:55.525751 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8268 has failed due to unidentified error.
  3488. 12/14 10:07:55.525773 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3489. 12/14 10:07:55.525790 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96DB0 synchronization task has been failed!
  3490. 12/14 10:07:55.525709 6ec0 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 3 from 2. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  3491. 12/14 10:07:55.525801 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7D68 has failed due to unidentified error.
  3492. 12/14 10:07:55.525824 6ec0 func: >>> HANode::reportEvent
  3493. 12/14 10:07:55.525837 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3494. 12/14 10:07:55.525895 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97650 synchronization task has been failed!
  3495. 12/14 10:07:55.525865 6ec0 func: >>> iScsiServer::sendNotification
  3496. 12/14 10:07:55.526112 6ec0 func: >>> CEventDataBase::AddRecord
  3497. 12/14 10:07:55.526131 6ec0 EventDB: CEventDataBase::AddRecord: Code 771, severity 3, additional strings: 2
  3498. 12/14 10:07:55.526397 6ec0 func: <<< CEventDataBase::AddRecord
  3499. 12/14 10:07:55.526417 6ec0 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 771).
  3500. 12/14 10:07:55.527488 6ec0 Srv: iScsiServer::sendNotification: Send reaction finish.
  3501. 12/14 10:07:55.527517 6ec0 func: <<< iScsiServer::sendNotification
  3502. 12/14 10:07:55.527528 6ec0 func: <<< HANode::reportEvent
  3503. 12/14 10:07:55.527537 6ec0 func: <<< CHAPartnerNode::SetSyncStatus
  3504. 12/14 10:07:55.544342 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (15 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 21075224, expcmdsn = 21075219, maxcmdsn = 21075461.
  3505. 12/14 10:07:55.544386 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 21075212, itt = 21075302)
  3506. 12/14 10:07:55.544397 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 21075213, itt = 21075303)
  3507. 12/14 10:07:55.544413 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6258 has failed due to unidentified error.
  3508. 12/14 10:07:55.544419 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3509. 12/14 10:07:55.544428 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F98A70 synchronization task has been failed!
  3510. 12/14 10:07:55.544446 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86993E8 has failed due to unidentified error.
  3511. 12/14 10:07:55.544450 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3512. 12/14 10:07:55.544458 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F98D50 synchronization task has been failed!
  3513. 12/14 10:07:55.561331 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (16 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 21075224, expcmdsn = 21075219, maxcmdsn = 21075461.
  3514. 12/14 10:07:55.561373 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 21075214, itt = 21075304)
  3515. 12/14 10:07:55.561382 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 21075215, itt = 21075305)
  3516. 12/14 10:07:55.561389 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 21075216, itt = 21075306)
  3517. 12/14 10:07:55.561394 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 21075217, itt = 21075307)
  3518. 12/14 10:07:55.561399 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 21075218, itt = 21075308)
  3519. 12/14 10:07:55.561583 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BF88 has failed due to unidentified error.
  3520. 12/14 10:07:55.561591 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3521. 12/14 10:07:55.561600 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96AD0 synchronization task has been failed!
  3522. 12/14 10:07:55.561619 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BE08 has failed due to unidentified error.
  3523. 12/14 10:07:55.561623 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3524. 12/14 10:07:55.561630 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F956B0 synchronization task has been failed!
  3525. 12/14 10:07:55.561636 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6158 has failed due to unidentified error.
  3526. 12/14 10:07:55.561640 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3527. 12/14 10:07:55.561647 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F95F50 synchronization task has been failed!
  3528. 12/14 10:07:55.561657 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE60D8 has failed due to unidentified error.
  3529. 12/14 10:07:55.561661 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3530. 12/14 10:07:55.561668 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97370 synchronization task has been failed!
  3531. 12/14 10:07:55.561675 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D85E8 has failed due to unidentified error.
  3532. 12/14 10:07:55.561679 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3533. 12/14 10:07:55.561686 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F95990 synchronization task has been failed!
  3534. 12/14 10:07:55.670389 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (109 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 21075224, expcmdsn = 21075219, maxcmdsn = 21075461.
  3535. 12/14 10:07:55.670429 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 21075219, itt = 21075309)
  3536. 12/14 10:07:55.670438 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 21075220, itt = 21075310)
  3537. 12/14 10:07:55.670445 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 21075221, itt = 21075311)
  3538. 12/14 10:07:55.670451 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 21075222, itt = 21075312)
  3539. 12/14 10:07:55.670463 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86994E8 has failed due to unidentified error.
  3540. 12/14 10:07:55.670470 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3541. 12/14 10:07:55.670480 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96510 synchronization task has been failed!
  3542. 12/14 10:07:55.670496 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7BE8 has failed due to unidentified error.
  3543. 12/14 10:07:55.670502 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3544. 12/14 10:07:55.670511 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F953D0 synchronization task has been failed!
  3545. 12/14 10:07:55.670619 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8568 has failed due to unidentified error.
  3546. 12/14 10:07:55.670628 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3547. 12/14 10:07:55.670635 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97090 synchronization task has been failed!
  3548. 12/14 10:07:55.670645 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EC4F39D8 has failed due to unidentified error.
  3549. 12/14 10:07:55.670650 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3550. 12/14 10:07:55.670657 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97930 synchronization task has been failed!
  3551. 12/14 10:08:00.139967 6ec0 HA: CHAPartnerNode::AfterSynchronization: Event: Full synchronization has been completed (error: 1237)
  3552. 12/14 10:08:00.142270 5378 func: >>> HANode::reportEvent
  3553. 12/14 10:08:00.142305 5378 func: >>> iScsiServer::sendNotification
  3554. 12/14 10:08:00.142315 5378 func: >>> CEventDataBase::AddRecord
  3555. 12/14 10:08:00.142323 5378 EventDB: CEventDataBase::AddRecord: Code 783, severity 1, additional strings: 2
  3556. 12/14 10:08:00.142364 5378 func: <<< CEventDataBase::AddRecord
  3557. 12/14 10:08:00.142369 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 783).
  3558. 12/14 10:08:00.143318 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  3559. 12/14 10:08:00.143340 5378 func: <<< iScsiServer::sendNotification
  3560. 12/14 10:08:00.143346 5378 func: <<< HANode::reportEvent
  3561. 12/14 10:08:00.143353 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_COMPLETE_WITH_ERROR, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'
  3562. 12/14 10:08:00.151426 38d8 func: >>> HANode::onScsiOp_CustomControl_HeartbeatPing
  3563. 12/14 10:08:00.151523 38d8 Common: CStarWindStorageDevice::getLastPerfDegEventElapsedTimeMs: perfDegEventElapsedTime = -1
  3564. 12/14 10:08:00.151534 38d8 func: <<< HANode::onScsiOp_CustomControl_HeartbeatPing
  3565. 12/14 10:08:00.152445 38d8 func: >>> CHAPartnerNode::ShutdownNode
  3566. 12/14 10:08:00.152483 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  3567. 12/14 10:08:00.152491 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  3568. 12/14 10:08:00.152500 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  3569. 12/14 10:08:00.152507 38d8 func: <<< CHAPartnerNode::ShutdownNode
  3570. 12/14 10:08:03.774218 38d8 func: >>> CHAPartnerNode::ShutdownNode
  3571. 12/14 10:08:03.774293 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  3572. 12/14 10:08:03.774307 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  3573. 12/14 10:08:03.774321 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  3574. 12/14 10:08:03.774339 38d8 func: <<< CHAPartnerNode::ShutdownNode
  3575. 12/14 10:08:03.779427 2d00 HA: HANode::onScsiOp_CustomControl_SynchronizationRequest: The partner iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03 has requested the 1 synchronization type.
  3576. 12/14 10:08:03.781703 2d00 HA: CHAPartnerNode::Synchronize: ENTERed with params: p_SyncType = 1, p_ulSnapshotID = 0(0x0), p_pStorageDevice = 0x000002C7ECEDC4B0, p_pSyncBarrier = 0x000002C7ECEDC858, p_puliStartSector = 0x000000D2528FF118 (value is 0x0), p_puliEndSector = 0x000000D2528FF110 (value is 0xAEFFFFF), p_ulIOBlockSizeInSectors = 64, p_ulSectorSizeInUCHARs = 4096, p_bContinue = no, p_bVaaiWriteSameSupport = yes
  3577. 12/14 10:08:03.781737 2d00 HA: HANode::updatePartnerStatus: Statuses is already actual, we don't need to update them.
  3578. 12/14 10:08:03.781755 2d00 func: >>> CHAPartnerNode::SetSyncStatus
  3579. 12/14 10:08:03.781762 2d00 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  3580. 12/14 10:08:03.781859 2d00 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 2 from 3. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  3581. 12/14 10:08:03.781867 2d00 func: >>> HANode::reportEvent
  3582. 12/14 10:08:03.781876 2d00 func: >>> iScsiServer::sendNotification
  3583. 12/14 10:08:03.781882 2d00 func: >>> CEventDataBase::AddRecord
  3584. 12/14 10:08:03.781886 2d00 EventDB: CEventDataBase::AddRecord: Code 770, severity 2, additional strings: 2
  3585. 12/14 10:08:03.781924 2d00 func: <<< CEventDataBase::AddRecord
  3586. 12/14 10:08:03.781929 2d00 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 770).
  3587. 12/14 10:08:03.782515 2d00 Srv: iScsiServer::sendNotification: Send reaction finish.
  3588. 12/14 10:08:03.782534 2d00 func: <<< iScsiServer::sendNotification
  3589. 12/14 10:08:03.782541 2d00 func: <<< HANode::reportEvent
  3590. 12/14 10:08:03.782546 2d00 func: <<< CHAPartnerNode::SetSyncStatus
  3591. 12/14 10:08:03.782836 6dac func: >>> CHAPartnerNode::SetSyncStatus
  3592. 12/14 10:08:03.782876 6dac HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  3593. 12/14 10:08:03.782891 6dac func: <<< CHAPartnerNode::SetSyncStatus
  3594. 12/14 10:08:03.785906 5378 func: >>> HANode::reportEvent
  3595. 12/14 10:08:03.785934 5378 func: >>> iScsiServer::sendNotification
  3596. 12/14 10:08:03.785941 5378 func: >>> CEventDataBase::AddRecord
  3597. 12/14 10:08:03.785947 5378 EventDB: CEventDataBase::AddRecord: Code 780, severity 1, additional strings: 3
  3598. 12/14 10:08:03.785981 5378 func: <<< CEventDataBase::AddRecord
  3599. 12/14 10:08:03.785986 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 780).
  3600. 12/14 10:08:03.786515 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  3601. 12/14 10:08:03.786534 5378 func: <<< iScsiServer::sendNotification
  3602. 12/14 10:08:03.786539 5378 func: <<< HANode::reportEvent
  3603. 12/14 10:08:03.786544 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_STARTED, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03', Synchronization type: 'Full'
  3604. 12/14 10:30:38.490716 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (9954 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 21533503, expcmdsn = 21533502, maxcmdsn = 21533741.
  3605. 12/14 10:30:38.490784 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 21533487, itt = 21533578)
  3606. 12/14 10:30:38.490801 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86999E8 has failed due to unidentified error.
  3607. 12/14 10:30:38.490807 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3608. 12/14 10:30:38.490817 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96230 synchronization task has been failed!
  3609. 12/14 10:30:38.490886 6dac func: >>> CHAPartnerNode::SetSyncStatus
  3610. 12/14 10:30:38.490935 6dac HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  3611. 12/14 10:30:38.490954 6dac HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 3 from 2. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  3612. 12/14 10:30:38.490968 6dac func: >>> HANode::reportEvent
  3613. 12/14 10:30:38.490997 6dac func: >>> iScsiServer::sendNotification
  3614. 12/14 10:30:38.491013 6dac func: >>> CEventDataBase::AddRecord
  3615. 12/14 10:30:38.491025 6dac EventDB: CEventDataBase::AddRecord: Code 771, severity 3, additional strings: 2
  3616. 12/14 10:30:38.491096 6dac func: <<< CEventDataBase::AddRecord
  3617. 12/14 10:30:38.491108 6dac Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 771).
  3618. 12/14 10:30:38.492325 6dac Srv: iScsiServer::sendNotification: Send reaction finish.
  3619. 12/14 10:30:38.492375 6dac func: <<< iScsiServer::sendNotification
  3620. 12/14 10:30:38.492390 6dac func: <<< HANode::reportEvent
  3621. 12/14 10:30:38.492402 6dac func: <<< CHAPartnerNode::SetSyncStatus
  3622. 12/14 10:30:38.494095 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 21533488, itt = 21533579)
  3623. 12/14 10:30:38.494134 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 21533489, itt = 21533580)
  3624. 12/14 10:30:38.494154 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 21533490, itt = 21533581)
  3625. 12/14 10:30:38.494164 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 21533491, itt = 21533582)
  3626. 12/14 10:30:38.494174 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 21533492, itt = 21533583)
  3627. 12/14 10:30:38.494185 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 21533493, itt = 21533584)
  3628. 12/14 10:30:38.494194 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 21533494, itt = 21533585)
  3629. 12/14 10:30:38.494207 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6258 has failed due to unidentified error.
  3630. 12/14 10:30:38.494217 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3631. 12/14 10:30:38.494232 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97C10 synchronization task has been failed!
  3632. 12/14 10:30:38.494254 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D85E8 has failed due to unidentified error.
  3633. 12/14 10:30:38.494265 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3634. 12/14 10:30:38.494279 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F98D50 synchronization task has been failed!
  3635. 12/14 10:30:38.494294 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8268 has failed due to unidentified error.
  3636. 12/14 10:30:38.494303 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3637. 12/14 10:30:38.494319 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96AD0 synchronization task has been failed!
  3638. 12/14 10:30:38.494334 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7BE8 has failed due to unidentified error.
  3639. 12/14 10:30:38.494342 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3640. 12/14 10:30:38.494356 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97930 synchronization task has been failed!
  3641. 12/14 10:30:38.494370 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E8699A68 has failed due to unidentified error.
  3642. 12/14 10:30:38.494378 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3643. 12/14 10:30:38.494392 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F953D0 synchronization task has been failed!
  3644. 12/14 10:30:38.494406 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BE08 has failed due to unidentified error.
  3645. 12/14 10:30:38.494415 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3646. 12/14 10:30:38.494428 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97650 synchronization task has been failed!
  3647. 12/14 10:30:38.494442 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EC4F39D8 has failed due to unidentified error.
  3648. 12/14 10:30:38.494459 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3649. 12/14 10:30:38.494472 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F95F50 synchronization task has been failed!
  3650. 12/14 10:30:38.511103 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (16 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 21533504, expcmdsn = 21533502, maxcmdsn = 21533741.
  3651. 12/14 10:30:38.511156 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 21533495, itt = 21533586)
  3652. 12/14 10:30:38.511175 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 21533496, itt = 21533587)
  3653. 12/14 10:30:38.511189 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 21533497, itt = 21533588)
  3654. 12/14 10:30:38.511206 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE60D8 has failed due to unidentified error.
  3655. 12/14 10:30:38.511219 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3656. 12/14 10:30:38.511240 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97EF0 synchronization task has been failed!
  3657. 12/14 10:30:38.511273 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86994E8 has failed due to unidentified error.
  3658. 12/14 10:30:38.511285 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3659. 12/14 10:30:38.511303 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F967F0 synchronization task has been failed!
  3660. 12/14 10:30:38.511318 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86991E8 has failed due to unidentified error.
  3661. 12/14 10:30:38.511330 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3662. 12/14 10:30:38.511347 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F981D0 synchronization task has been failed!
  3663. 12/14 10:30:38.526122 2564 Common: *** iscsi_tcp_dispatch: request execution timed out (15 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 21533504, expcmdsn = 21533502, maxcmdsn = 21533741.
  3664. 12/14 10:30:38.526167 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 21533498, itt = 21533589)
  3665. 12/14 10:30:38.526180 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 21533499, itt = 21533590)
  3666. 12/14 10:30:38.526189 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 21533500, itt = 21533591)
  3667. 12/14 10:30:38.526197 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 21533501, itt = 21533592)
  3668. 12/14 10:30:38.526205 2564 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 21533502, itt = 21533593)
  3669. 12/14 10:30:38.526217 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EDEE6758 has failed due to unidentified error.
  3670. 12/14 10:30:38.526226 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3671. 12/14 10:30:38.526251 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F95C70 synchronization task has been failed!
  3672. 12/14 10:30:38.526272 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D8568 has failed due to unidentified error.
  3673. 12/14 10:30:38.526280 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3674. 12/14 10:30:38.526300 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96510 synchronization task has been failed!
  3675. 12/14 10:30:38.526313 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E86993E8 has failed due to unidentified error.
  3676. 12/14 10:30:38.526320 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3677. 12/14 10:30:38.526331 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F97370 synchronization task has been failed!
  3678. 12/14 10:30:38.526344 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E91D7D68 has failed due to unidentified error.
  3679. 12/14 10:30:38.526351 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3680. 12/14 10:30:38.526362 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F96DB0 synchronization task has been failed!
  3681. 12/14 10:30:38.526374 2564 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EA05BF88 has failed due to unidentified error.
  3682. 12/14 10:30:38.526381 2564 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
  3683. 12/14 10:30:38.526392 2564 HA: CReadWriteSyncTask::complete_task: The 0x000002C780F95990 synchronization task has been failed!
  3684. 12/14 10:30:43.284256 38d8 func: >>> HANode::onScsiOp_CustomControl_HeartbeatPing
  3685. 12/14 10:30:43.284457 38d8 Common: CStarWindStorageDevice::getLastPerfDegEventElapsedTimeMs: perfDegEventElapsedTime = -1
  3686. 12/14 10:30:43.284487 38d8 func: <<< HANode::onScsiOp_CustomControl_HeartbeatPing
  3687. 12/14 10:30:43.285115 6dac HA: CHAPartnerNode::AfterSynchronization: Event: Full synchronization has been completed (error: 1237)
  3688. 12/14 10:30:43.285369 38d8 func: >>> CHAPartnerNode::ShutdownNode
  3689. 12/14 10:30:43.285407 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  3690. 12/14 10:30:43.285419 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  3691. 12/14 10:30:43.285434 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  3692. 12/14 10:30:43.285459 38d8 func: <<< CHAPartnerNode::ShutdownNode
  3693. 12/14 10:30:43.287936 5378 func: >>> HANode::reportEvent
  3694. 12/14 10:30:43.287971 5378 func: >>> iScsiServer::sendNotification
  3695. 12/14 10:30:43.287980 5378 func: >>> CEventDataBase::AddRecord
  3696. 12/14 10:30:43.287986 5378 EventDB: CEventDataBase::AddRecord: Code 783, severity 1, additional strings: 2
  3697. 12/14 10:30:43.288032 5378 func: <<< CEventDataBase::AddRecord
  3698. 12/14 10:30:43.288039 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 783).
  3699. 12/14 10:30:43.288758 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  3700. 12/14 10:30:43.288783 5378 func: <<< iScsiServer::sendNotification
  3701. 12/14 10:30:43.288791 5378 func: <<< HANode::reportEvent
  3702. 12/14 10:30:43.288809 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_COMPLETE_WITH_ERROR, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'
  3703. 12/14 10:30:43.291134 38d8 Common: *** sw_common::Sw_Disk_Header::backup: Asynchronous header 'C:\Program Files\StarWind Software\StarWind\headers\datastore03\datastore03_HA.swdsk' operation in progress!
  3704. 12/14 10:30:46.233415 38d8 func: >>> CHAPartnerNode::ShutdownNode
  3705. 12/14 10:30:46.233492 38d8 func: >>> CHAPartnerNode::SetSyncStatus
  3706. 12/14 10:30:46.233507 38d8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  3707. 12/14 10:30:46.233521 38d8 func: <<< CHAPartnerNode::SetSyncStatus
  3708. 12/14 10:30:46.233533 38d8 func: <<< CHAPartnerNode::ShutdownNode
  3709. 12/14 10:30:46.238506 2d00 HA: HANode::onScsiOp_CustomControl_SynchronizationRequest: The partner iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03 has requested the 1 synchronization type.
  3710. 12/14 10:30:46.239950 2d00 HA: CHAPartnerNode::Synchronize: ENTERed with params: p_SyncType = 1, p_ulSnapshotID = 0(0x0), p_pStorageDevice = 0x000002C7ECEDC4B0, p_pSyncBarrier = 0x000002C7ECEDC858, p_puliStartSector = 0x000000D2528FF118 (value is 0x0), p_puliEndSector = 0x000000D2528FF110 (value is 0xAEFFFFF), p_ulIOBlockSizeInSectors = 64, p_ulSectorSizeInUCHARs = 4096, p_bContinue = no, p_bVaaiWriteSameSupport = yes
  3711. 12/14 10:30:46.240007 2d00 HA: HANode::updatePartnerStatus: Statuses is already actual, we don't need to update them.
  3712. 12/14 10:30:46.240039 2d00 func: >>> CHAPartnerNode::SetSyncStatus
  3713. 12/14 10:30:46.240052 2d00 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  3714. 12/14 10:30:46.240064 2d00 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 2 from 3. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
  3715. 12/14 10:30:46.240078 2d00 func: >>> HANode::reportEvent
  3716. 12/14 10:30:46.240091 2d00 func: >>> iScsiServer::sendNotification
  3717. 12/14 10:30:46.240103 2d00 func: >>> CEventDataBase::AddRecord
  3718. 12/14 10:30:46.240115 2d00 EventDB: CEventDataBase::AddRecord: Code 770, severity 2, additional strings: 2
  3719. 12/14 10:30:46.240182 2d00 func: <<< CEventDataBase::AddRecord
  3720. 12/14 10:30:46.240194 2d00 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 770).
  3721. 12/14 10:30:46.240993 2d00 Srv: iScsiServer::sendNotification: Send reaction finish.
  3722. 12/14 10:30:46.241025 2d00 func: <<< iScsiServer::sendNotification
  3723. 12/14 10:30:46.241039 2d00 func: <<< HANode::reportEvent
  3724. 12/14 10:30:46.241051 2d00 func: <<< CHAPartnerNode::SetSyncStatus
  3725. 12/14 10:30:46.241422 6e68 func: >>> CHAPartnerNode::SetSyncStatus
  3726. 12/14 10:30:46.241467 6e68 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
  3727. 12/14 10:30:46.241489 6e68 func: <<< CHAPartnerNode::SetSyncStatus
  3728. 12/14 10:30:46.243748 5378 func: >>> HANode::reportEvent
  3729. 12/14 10:30:46.243774 5378 func: >>> iScsiServer::sendNotification
  3730. 12/14 10:30:46.243779 5378 func: >>> CEventDataBase::AddRecord
  3731. 12/14 10:30:46.243784 5378 EventDB: CEventDataBase::AddRecord: Code 780, severity 1, additional strings: 3
  3732. 12/14 10:30:46.243817 5378 func: <<< CEventDataBase::AddRecord
  3733. 12/14 10:30:46.243821 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 780).
  3734. 12/14 10:30:46.244194 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
  3735. 12/14 10:30:46.244209 5378 func: <<< iScsiServer::sendNotification
  3736. 12/14 10:30:46.244214 5378 func: <<< HANode::reportEvent
  3737. 12/14 10:30:46.244220 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_STARTED, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03', Synchronization type: 'Full'
  3738. 12/14 10:37:49.233290 5cf8 func: >>> CHAPartnerNode::ShutdownNode
  3739. 12/14 10:37:49.233425 5cf8 func: >>> CHAPartnerNode::SetSyncStatus
  3740. 12/14 10:37:49.233438 5cf8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore02'.
  3741. 12/14 10:37:49.233450 5cf8 func: <<< CHAPartnerNode::SetSyncStatus
  3742. 12/14 10:37:49.233459 5cf8 func: <<< CHAPartnerNode::ShutdownNode
  3743. 12/14 10:37:49.238656 2e18 HA: HANode::onScsiOp_CustomControl_SynchronizationRequest: The partner iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore02 has requested the 2 synchronization type.
  3744. 12/14 10:37:49.239819 2e18 HA: CHAPartnerNode::Synchronize: ENTERed with params: p_SyncType = 2, p_ulSnapshotID = 0(0x0), p_pStorageDevice = 0x000002C7E87440D0, p_pSyncBarrier = 0x000002C7E8744478, p_puliStartSector = 0x000000D24D6FEF08 (value is 0x0), p_puliEndSector = 0x000000D24D6FEF00 (value is 0xC7FFFFF), p_ulIOBlockSizeInSectors = 64, p_ulSectorSizeInUCHARs = 4096, p_bContinue = no, p_bVaaiWriteSameSupport = yes
  3745. 12/14 10:37:49.239896 2e18 HA: HANode::updatePartnerStatus: Statuses is already actual, we don't need to update them.
  3746. 12/14 10:37:49.239932 2e18 func: >>> CHAPartnerNode::SetSyncStatus
  3747. 12/14 10:37:49.239945 2e18 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore02'.
  3748. 12/14 10:37:49.239958 2e18 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 2 from 3. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore02')
  3749. 12/14 10:37:49.239971 2e18 func: >>> HANode::reportEvent
  3750. 12/14 10:37:49.239986 2e18 func: >>> iScsiServer::sendNotification
  3751. 12/14 10:37:49.240000 2e18 func: >>> CEventDataBase::AddRecord
  3752. 12/14 10:37:49.240011 2e18 EventDB: CEventDataBase::AddRecord: Code 770, severity 2, additional strings: 2
  3753. 12/14 10:37:49.240202 2e18 func: <<< CEventDataBase::AddRecord
  3754. 12/14 10:37:49.240223 2e18 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 770).
  3755. 12/14 10:37:49.241219 2e18 Srv: iScsiServer::sendNotification: Send reaction finish.
  3756. 12/14 10:37:49.241255 2e18 func: <<< iScsiServer::sendNotification
  3757. 12/14 10:37:49.241269 2e18 func: <<< HANode::reportEvent
  3758. 12/14 10:37:49.241281 2e18 func: <<< CHAPartnerNode::SetSyncStatus
  3759. 12/14 10:37:49.241650 2344 func: >>> CHAPartnerNode::SetSyncStatus
  3760. 12/14 10:37:49.241670 2344 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore02'.
  3761. 12/14 10:37:49.241676 2344 func: <<< CHAPartnerNode::SetSyncStatus
  3762. 12/14 10:37:49.261832 2698 func: >>> HANode::reportEvent
  3763. 12/14 10:37:49.261871 2698 func: >>> iScsiServer::sendNotification
  3764. 12/14 10:37:49.261881 2698 func: >>> CEventDataBase::AddRecord
  3765. 12/14 10:37:49.261888 2698 EventDB: CEventDataBase::AddRecord: Code 780, severity 1, additional strings: 3
  3766. 12/14 10:37:49.261933 2698 func: <<< CEventDataBase::AddRecord
  3767. 12/14 10:37:49.261940 2698 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 780).
  3768. 12/14 10:37:49.262615 2698 Srv: iScsiServer::sendNotification: Send reaction finish.
  3769. 12/14 10:37:49.262642 2698 func: <<< iScsiServer::sendNotification
  3770. 12/14 10:37:49.262651 2698 func: <<< HANode::reportEvent
  3771. 12/14 10:37:49.262659 2698 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_STARTED, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore02', Synchronization type: 'Fast'
  3772. 12/14 10:37:49.285912 2344 HA: CHAPartnerNode::FastSynchronizationThread: Sync partitions count is 0x0
  3773. 12/14 10:37:49.287747 2344 HA: CHAPartnerNode::FastSynchronizationThread: Changed regions count is 0x1B732
  3774. 12/14 10:37:50.475432 2344 HA: Blocks_Barrier::enter_barrier: Intersects exist:
  3775. 12/14 10:37:50.475493 2344 HA: Blocks_Barrier::enter_barrier: CurrentBlockID=0x000002C780F967F0, ullStartSector = 0x87530, ulLength = 16, bIsSerialize = 1
  3776. 12/14 10:37:50.475510 2344 HA: Blocks_Barrier::enter_barrier: OtherBlockID=0x000002C7F03124E0, ullStartSector = 0x8753E, ulLength = 1, bIsSerialize = 0
  3777. 12/14 10:37:51.258021 4be8 func: >>> CHAPartnerNode::ShutdownNode
  3778. 12/14 10:37:51.258096 4be8 func: >>> CHAPartnerNode::SetSyncStatus
  3779. 12/14 10:37:51.258110 4be8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore01'.
  3780. 12/14 10:37:51.258125 4be8 func: <<< CHAPartnerNode::SetSyncStatus
  3781. 12/14 10:37:51.258136 4be8 func: <<< CHAPartnerNode::ShutdownNode
  3782. 12/14 10:37:51.263182 5860 HA: HANode::onScsiOp_CustomControl_SynchronizationRequest: The partner iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore01 has requested the 2 synchronization type.
  3783. 12/14 10:37:51.264290 5860 HA: CHAPartnerNode::Synchronize: ENTERed with params: p_SyncType = 2, p_ulSnapshotID = 0(0x0), p_pStorageDevice = 0x000002C7F58922D0, p_pSyncBarrier = 0x000002C7F5892678, p_puliStartSector = 0x000000D24B17EE18 (value is 0x0), p_puliEndSector = 0x000000D24B17EE10 (value is 0x3FFFFFFF), p_ulIOBlockSizeInSectors = 64, p_ulSectorSizeInUCHARs = 4096, p_bContinue = no, p_bVaaiWriteSameSupport = yes
  3784. 12/14 10:37:51.264344 5860 HA: HANode::updatePartnerStatus: Statuses is already actual, we don't need to update them.
  3785. 12/14 10:37:51.264380 5860 HA: *** CHAPartnerNode::Synchronize: Fast synchronization is not available and will be changed to FULL or FULL_HASH synchronization!
  3786. 12/14 10:37:51.264401 5860 func: >>> CHAPartnerNode::SetSyncStatus
  3787. 12/14 10:37:51.264412 5860 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore01'.
  3788. 12/14 10:37:51.264425 5860 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 2 from 3. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore01')
  3789. 12/14 10:37:51.264437 5860 func: >>> HANode::reportEvent
  3790. 12/14 10:37:51.264477 5860 func: >>> iScsiServer::sendNotification
  3791. 12/14 10:37:51.264495 5860 func: >>> CEventDataBase::AddRecord
  3792. 12/14 10:37:51.264506 5860 EventDB: CEventDataBase::AddRecord: Code 770, severity 2, additional strings: 2
  3793. 12/14 10:37:51.264572 5860 func: <<< CEventDataBase::AddRecord
  3794. 12/14 10:37:51.264583 5860 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 770).
  3795. 12/14 10:37:51.265756 5860 Srv: iScsiServer::sendNotification: Send reaction finish.
  3796. 12/14 10:37:51.265791 5860 func: <<< iScsiServer::sendNotification
  3797. 12/14 10:37:51.265803 5860 func: <<< HANode::reportEvent
  3798. 12/14 10:37:51.265814 5860 func: <<< CHAPartnerNode::SetSyncStatus
  3799. 12/14 10:37:51.266120 4348 func: >>> CHAPartnerNode::SetSyncStatus
  3800. 12/14 10:37:51.266160 4348 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore01'.
  3801. 12/14 10:37:51.266173 4348 func: <<< CHAPartnerNode::SetSyncStatus
  3802. 12/14 10:37:51.281607 2bd4 func: >>> HANode::reportEvent
  3803. 12/14 10:37:51.281643 2bd4 func: >>> iScsiServer::sendNotification
  3804. 12/14 10:37:51.281650 2bd4 func: >>> CEventDataBase::AddRecord
  3805. 12/14 10:37:51.281654 2bd4 EventDB: CEventDataBase::AddRecord: Code 780, severity 1, additional strings: 3
  3806. 12/14 10:37:51.281691 2bd4 func: <<< CEventDataBase::AddRecord
  3807. 12/14 10:37:51.281696 2bd4 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 780).
  3808. 12/14 10:37:51.282624 2bd4 Srv: iScsiServer::sendNotification: Send reaction finish.
  3809. 12/14 10:37:51.282648 2bd4 func: <<< iScsiServer::sendNotification
  3810. 12/14 10:37:51.282654 2bd4 func: <<< HANode::reportEvent
  3811. 12/14 10:37:51.282658 2bd4 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_STARTED, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore01', Synchronization type: 'Full'
  3812. 12/14 10:38:14.033355 2344 HA: Blocks_Barrier::enter_barrier: Intersects exist:
  3813. 12/14 10:38:14.033401 2344 HA: Blocks_Barrier::enter_barrier: CurrentBlockID=0x000002C780F96230, ullStartSector = 0x112B300, ulLength = 16, bIsSerialize = 1
  3814. 12/14 10:38:14.033413 2344 HA: Blocks_Barrier::enter_barrier: OtherBlockID=0x000002C7F0555D80, ullStartSector = 0x112B30D, ulLength = 1, bIsSerialize = 0
  3815. 12/14 10:38:22.719501 2344 HA: CHAPartnerNode::FastSynchronizationThread: Changed region not found.
  3816. 12/14 10:38:22.719547 2344 HA: CHAPartnerNode::FastSynchronizationThread: Coping of unsync partner bitmap to synchronizing partner...
  3817. 12/14 10:38:22.719555 2344 HA: CHAPartnerNode::FastSynchronizationThread: Wait for all tasks complete...
  3818. 12/14 10:38:22.721065 2344 func: >>> CHAPartnerNode::SetSyncStatus
  3819. 12/14 10:38:22.721079 2344 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 1, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore02'.
  3820. 12/14 10:38:22.721084 2344 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 1 from 2. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore02')
  3821. 12/14 10:38:22.721095 2344 func: >>> HANode::reportEvent
  3822. 12/14 10:38:22.721113 2344 func: >>> iScsiServer::sendNotification
  3823. 12/14 10:38:22.721118 2344 func: >>> CEventDataBase::AddRecord
  3824. 12/14 10:38:22.721122 2344 EventDB: CEventDataBase::AddRecord: Code 769, severity 1, additional strings: 2
  3825. 12/14 10:38:22.721155 2344 func: <<< CEventDataBase::AddRecord
  3826. 12/14 10:38:22.721159 2344 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 769).
  3827. 12/14 10:38:22.721598 2344 Srv: iScsiServer::sendNotification: Send reaction finish.
  3828. 12/14 10:38:22.721613 2344 func: <<< iScsiServer::sendNotification
  3829. 12/14 10:38:22.721618 2344 func: <<< HANode::reportEvent
  3830. 12/14 10:38:22.721622 2344 func: <<< CHAPartnerNode::SetSyncStatus
  3831. 12/14 10:38:22.721669 2344 HA: CHAPartnerNode::FastSynchronizationThread: Event: Fast_Synchronization_Complete error:0
  3832. 12/14 10:38:22.724640 2698 func: >>> HANode::reportEvent
  3833. 12/14 10:38:22.724671 2698 func: >>> iScsiServer::sendNotification
  3834. 12/14 10:38:22.724679 2698 func: >>> CEventDataBase::AddRecord
  3835. 12/14 10:38:22.724685 2698 EventDB: CEventDataBase::AddRecord: Code 782, severity 1, additional strings: 2
  3836. 12/14 10:38:22.724724 2698 func: <<< CEventDataBase::AddRecord
  3837. 12/14 10:38:22.724730 2698 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 782).
  3838. 12/14 10:38:22.725354 2698 Srv: iScsiServer::sendNotification: Send reaction finish.
  3839. 12/14 10:38:22.725381 2698 func: <<< iScsiServer::sendNotification
  3840. 12/14 10:38:22.725394 2698 func: <<< HANode::reportEvent
  3841. 12/14 10:38:22.725403 2698 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_COMPLETE_SUCCESSFULLY, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore02'
  3842. 12/14 10:38:23.568545 2e18 HA: HAPerRes::ResLunReservationEx::exportState_CMD_V1: exporting reservations (5 regs)...
  3843. 12/14 10:38:23.568623 2e18 HA: HAPerRes::ResLunReservationEx::exportState_CMD_V1: export: iqn.2008-08.com.starwindsoftware:home-hypv-h01-datastore02 1 1 12 5 0 5 5F9CF57D0001734D 0
  3844. 12/14 10:38:23.568647 2e18 HA: HAPerRes::ResLunReservationEx::exportState_CMD_V1: export: iqn.1991-05.com.microsoft:home-hypv-h01.home.jorgedelgado.net,400001370000 3409 iqn.2008-08.com.starwindsoftware:home-hypv-h01-datastore02 1 5F9CF57D0001734D
  3845. 12/14 10:38:23.568677 2e18 HA: HAPerRes::ResLunReservationEx::exportState_CMD_V1: export: iqn.1991-05.com.microsoft:home-hypv-h01.home.jorgedelgado.net,400001370000 0 iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore02 1 5F9CF57D0001734D
  3846. 12/14 10:38:23.568694 2e18 HA: HAPerRes::ResLunReservationEx::exportState_CMD_V1: export: iqn.1991-05.com.microsoft:home-hypv-h02.home.jorgedelgado.net,400001370000 3574 iqn.2008-08.com.starwindsoftware:home-hypv-h01-datastore02 0 4A320BAA0002734D
  3847. 12/14 10:38:23.568716 2e18 HA: HAPerRes::ResLunReservationEx::exportState_CMD_V1: export: iqn.1991-05.com.microsoft:home-hypv-h02.home.jorgedelgado.net,400001370000 0 iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore02 0 3FB686150002734D
  3848. 12/14 10:38:23.568732 2e18 HA: HAPerRes::ResLunReservationEx::exportState_CMD_V1: export: iqn.1991-05.com.microsoft:home-hypv-h01.home.jorgedelgado.net,400001370002 0 iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore02 1 5F9CF57D0001734D
  3849. 12/14 10:38:23.568748 2e18 HA: HAPerRes::ResLunReservationEx::exportState_CMD_V1: export: iqn.1991-05.com.microsoft:home-hypv-h02.home.jorgedelgado.net,400001370001 3446 iqn.2008-08.com.starwindsoftware:home-hypv-h01-datastore02 1 4A320BAA0002734D
  3850. 12/14 10:38:23.568770 2e18 HA: HAPerRes::ResLunReservationEx::exportState_CMD_V1: export: iqn.1991-05.com.microsoft:home-hypv-h01.home.jorgedelgado.net,400001370003 3578 iqn.2008-08.com.starwindsoftware:home-hypv-h01-datastore02 1 5F9CF57D0001734D
  3851. 12/14 10:38:23.568788 2e18 HA: HAPerRes::ResLunReservationEx::exportState_CMD_V1: exported reservation state (1191 bytes, 7 sessions)
  3852. 12/14 10:38:23.615563 2e18 func: >>> CHAPartnerNode::SetSyncStatus
  3853. 12/14 10:38:23.615622 2e18 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 1, old status = 1) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore02'.
  3854. 12/14 10:38:23.615650 2e18 func: <<< CHAPartnerNode::SetSyncStatus
  3855. 12/14 10:38:23.615957 2e18 Common: *** sw_common::Sw_Disk_Header::backup: Asynchronous header 'C:\Program Files\StarWind Software\StarWind\headers\datastore02\datastore02_HA.swdsk' operation in progress!
  3856. 12/14 10:38:25.082138 2e18 PR: ResLunSessionEngine::registerSession: LUN 0: existing record for session 0x87a2 from iqn.1991-05.com.microsoft:home-hypv-h02.home.jorgedelgado.net,400001370000 iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore02
  3857. 12/14 10:38:25.212496 2e18 PR: ResLunSessionEngine::registerSession: LUN 0: existing record for session 0x87a4 from iqn.1991-05.com.microsoft:home-hypv-h01.home.jorgedelgado.net,400001370000 iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore02
  3858. 12/14 10:38:25.246884 2e18 PR: ResLunReservation::persistentReservationOut: RESERVE_OUT: REGISTER_IGNORE_EXISTING (p = 0)
  3859. 12/14 10:38:25.246930 2e18 PR: ResLunReservation::registerKey: S[87a4] RESERVE_OUT: REGISTER the same key 5F9CF57D0001734D.
  3860. 12/14 10:38:35.213309 fd8 HA: HAPerRes::processRequest: Request will change the reservation: DataTransferLength = 24, SenseInfoBufferLength = 64,CdbLength = 10, CDB: 5F, 06, 00, 00, 00, 00, 00, 00, 18, 00, 00, 00, 00, 00, 00, 00.
  3861. 12/14 10:38:35.214305 fd8 PR: ResLunReservation::persistentReservationOut: RESERVE_OUT: REGISTER_IGNORE_EXISTING (p = 0)
  3862. 12/14 10:38:35.214347 fd8 PR: ResLunReservation::registerKey: S[d76] RESERVE_OUT: REGISTER the same key 4A320BAA0002734D.
  3863. 12/14 10:39:04.676982 5368 HA: HANode::sendWUSControlRequest: Callback SW_WU_SERV_CONTROL (ENABLE) complete successfully.
  3864.  
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!