Troubleshooting Changed Block Tracking (CBT)

This section provides two hints for helping customers with CBT failures.

Verify Change Tracking Enabled

If you reconfigure a virtual machine to set changeTrackingEnabled, but the property remains false, check that you have queried the virtual machine status with VirtualMachine->config() after reconfiguration with VirtualMachine->reconfigure() and not before. Also make sure that virtual machine compatibility is hardware version 7 or higher, and that the virtual machine has undergone a stun-unstun cycle since reconfiguration.

Run the vixDiskCheck Utility

If customers encounter CBT issues when restoring files on a destination VM, they can use vixDiskCheck to ascertain whether there was data corruption after backing up the source VM. The vixDiskCheck utility should be run on the backup proxy VM, either Linux or Windows.

When you unpack the VDDK distribution tarball or ZIP file, the bin/vixDiskCheck utility is available in the resulting folder. Add the pathname of this folder to the LD_LIBRARY_PATH environment on Linux, or to the PATH environment on Windows. Examples below use Bash and PowerShell with install locations /opt/vddk and VMware\VDDK.

$ export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/opt/vddk/lib64"
> $env:Path += 'C:\Program Files\VMware\VDDK\bin'

You can run vixDiskCheck or vixDiskCheck.exe to see its options with explanations. Follow these steps to diagnose:

Delete previous snapshots, if any. Then reset CBT by deactivating and enabling it again.
Power on the VM. Collect the following files and look for the power-on tag.
- CTK files in the vmfs/volumes/datastore/VM folder, for example BaseDisk-ctk.vmdk
- hostd.log in var/run/log and vmware.log in vmfs/volumes/datastore/VM
Take a snapshot (snapshot-1) using the vSphere Client. Collect:
- CTK file in the VM folder under vmfs/volumes/datastore, BaseDisk-ctk.vmdk again.

Run vixDiskCheck to verify consistency of active full backup. Redirect stdout and stderr into vixDiskCheck.log in the current folder, as below. The CBT backup file must be large enough to hold the entire disk.

vixDiskCheck cbtCheck -host IPaddr -user username -password password -vm moref=VmMoref
  -ssmoref SnapshotMoRef -libdir VDDKlibDir -thumb SSLhostThumbprint -cbtdir CBTbackupFile
  -disk Base-disk.vmdk -transportMode nbdssl 2>&1 | tee vixDiskCheck-01.log

Search for CBT DIFF in the log. If any lines are output, then the CBT file is corrupted.
```
grep "CBT DIFF:" vixDiskCheck.log
```
Delete the previous snapshot and take a new snapshot (snapshot-2). Collect the CTK file and logs tagged snapshot-2.

Run vixDiskCheck again to verify consistency of the incremental backup with snapshot-2. Use the same command options as before, except for the snapshot MoRef and the log file path.

vixDiskCheck cbtCheck -host IPaddr -user username -password password -vm moref=VmMoref
  -ssmoref Snapshot2MoRef -libdir VDDKlibDir -thumb SSLhostThumbprint -cbtdir CBTbackupFile
  -disk Base-disk.vmdk -transportMode nbdssl 2>&1 | tee vixDiskCheck-02.log

Repeat the previous step to check consistency of subsequent incremental backups. Make sure to change the snapshot MoRef and the log file name accordingly.

The -cbtdir flag specifies the CBT working directory; vixDiskCheck creates backup.dat and cbt.inf files in that folder. Large file backup.dat is the full backup of the target disk as a sparse local file, which means only allocated areas are written to it, while other areas are holes. Small text file cbt.inf stores the changeId of the snapshot in the latest run, and is read in the next run, so that the utility can queryChangedDiskAreas against the snapshot with this changeId for incremental backup.

In vixDiskCheck logs,

"compare in range" means the range has no changes reported by CBT between the two snapshots. It compares data in the created file with data on the source disk.
"write in range" means the range has changes reported by CBT between the two snapshots. It writes data on the created file from the source disk.