Veeam v6.0.0.153 64-bit CBT Issue

Veeam Backup and Replication version 6 CBT  failing

CBT failing on drives other than Hard Disk 1. So far on any VM I tried to backup with Veeam 6;  Environment vCenter 5 all hosts ESXi5;  vCenter 5 was a clean install.    I went through the CTK reset procedure mentioned by aweiss. No dice, same behavior persists.

Also,  if I create a brand new  Version 8 VM with 2 blank hard drives, power it on, install an OS; create a new backup job, add the new VM to the backup job.  Run the backup job twice,  the same warning appears on the brand new VM, and CBT is not used for the reverse incremental backup.   The warning is raised immediately on the 2nd run of the new backup job
"Cannot use CBT: Soap fault. A specified parameter was not correct. . deviceKeyDetail:"



The agent.backup_job.source log  on the proxy speaks of the same
 <  1644> dsk| WARN|It is impossible to create disk changes tracker in accordance with the specified settings. Default tracker will be used.
 <  1644> dsk| >>  |Soap fault. A specified parameter was not correct.
 <  1644> dsk| >>  |deviceKeyDetail: '<InvalidArgumentFault xmlns="urn:internalvim25" xsi:type="InvalidArgument"><invalidProperty>deviceKey</invalidProperty></InvalidArgumentFault>', endpoint: ''
 <  1644> dsk| >>  |--tr:Failed to enumerate changed areas of the disk using CTK. Device key: [2001], size:  size: [2147483648]. VM ref: [vm-951]. Change ID: [52 2d 7b 8f ac 61 51 cc-77 4c 7b ec 91 50 bb 7b/1]

After investigation I see that in fact "2001"  seems to be a valid device key  for the disk device on the target VM.
That is.. the error doesn't make sense because 2001 in fact is the device key associated with the VMDK in the VM's
config spec.
On the very same VMDK that Veeam says CBT couldn't be used on,  I  inspected the VM's CBT status  by invoking
QueryChangedDiskAreas(), using the vSphere PowerCLI,   and I  see what appears a proper response shown....

And I am able to fetch lists of changed blocks through Powershell while the backup is in progress.....
So  Powershell can see the changed blocks, but  Veeam can't?    That doesn't make sense.

Nonetheless...

PowerCLI C:\>$x = Get-VM "test1p" ; $xv = $x | Get-View; foreach($z in $xv.Config.Hardware.Device) {;     if ($z.DeviceInfo.Label.ToLower() -match ".*hard disk.*") {  $z.DeviceInfo.Label, $z.Key | ft;            } }
Hard disk 1  2000
Hard disk 2  2001
PowerCLI C:\> $s = Get-Snapshot -VM $x  ; $sv = $s | Get-View ; $s
Name                 Description                    PowerState
----                 -----------                    ----------
VEEAM BACKUP TEMP... Please do not delete this s... PoweredOff
PowerCLI C:\> $ch0=$xv.QueryChangedDiskAreas($sv.MoRef,2001,0,"52 2d 7b 8f ac 61 51 cc-77 4c 7b ec 91 50 bb 7b/1"); $ch0
PowerCLI C:\> $ch0=$xv
.QueryChangedDiskAreas($sv.MoRef,2001,0,"52 2d 7b 8f ac 61 51 cc-77 4c 7b ec 91
50 bb 7b/1"); $ch0

StartOffset     : 0
Length          : 2147483648
ChangedArea     : {VMware.Vim.DiskChangeExtent, VMware.Vim.DiskChangeExtent, VM
                  ware.Vim.DiskChangeExtent, VMware.Vim.DiskChangeExtent...}
DynamicType     :
DynamicProperty :
PowerCLI C:\> $ch0.ChangedArea
              Start              Length DynamicType         DynamicProperty
              -----              ------ -----------         ---------------
                  0               65536
            1048576            17235968
          135266304               65536
          135528448             5701632
          141557760             2162688
          403701760               65536
          537919488            33619968
          672137216               65536
          940572672               65536
         1074724864              131072



....  later, after backup is done..   .....

PowerCLI C:\> $x = Get-VM "test1p" ; $xv = $x | Get-View;
PowerCLI C:\>$s = New-Snapshot -VM $x  -Name "temp1"
PowerCLI C:\>$xv = $x | Get-View;  $sv = $s | Get-View;  $a=@()
PowerCLI C:\>foreach($z in $sv.Config.Hardware.Device) {;     if ($z.DeviceInfo.Label.ToLower() -match ".*hard disk.*") { $a += $z    } }
PowerCLI C:\>foreach($dsk in $a) { $dsk.Key, $dsk.Backing.Get_ChangeID(); }
2000
52 73 8a c2 4c e9 a1 b8-d0 cf d7 b7 49 e4 23 ab/22
2001
52 2d 7b 8f ac 61 51 cc-77 4c 7b ec 91 50 bb 7b/22

PowerCLI C:\>Remove-Snapshot $s -Confirm:$false
PowerCLI C:\>$s = New-Snapshot -VM $x  -Name "temp2" ; $xv = $x | Get-View ; $sv = $s | Get-View; $a = @();

PowerCLI C:\> $ch1=$xv.QueryChangedDiskAreas($sv.MoRef,2000,0,"52 73 8a c2 4c e9 a1 b8-d0 cf d7 b7 49 e4 23 ab/22"); $ch1
StartOffset     : 0
Length          : 10737418240
ChangedArea     : {VMware.Vim.DiskChangeExtent, VMware.Vim.DiskChangeExtent, VM
                  ware.Vim.DiskChangeExtent, VMware.Vim.DiskChangeExtent...}
DynamicType     :
DynamicProperty :

PowerCLI C:\>$ch2 = $xv.QueryChangedDiskAreas($sv.MoRef, 2001, 0, "52 2d 7b 8f ac 61 51 cc-77 4c 7b ec 91 50 bb 7b/22");  $ch2
StartOffset     : 0
Length          : 2147483648
ChangedArea     : {VMware.Vim.DiskChangeExtent, VMware.Vim.DiskChangeExtent, VM
                  ware.Vim.DiskChangeExtent, VMware.Vim.DiskChangeExtent...}
DynamicType     :
DynamicProperty :

PowerCLI C:\> $ch2.ChangedArea
              Start              Length DynamicType         DynamicProperty
              -----              ------ -----------         ---------------
                  0               65536
            1048576            17235968
          135266304               65536
          135528448             5701632
          141557760             2162688
          403701760               65536
          537919488            33619968
          672137216               65536
          940572672               65536
         1074724864              131072