Virtual Disk Development Kit (VDDK) 8.0 Update 3 Release Notes

VDDK 8.0.3 with ESXi 8.0 Update 3 | 25 June 2024 GA | VDDK build 23950268
For VMware Cloud Foundation and vSphere
Check back for additions and updates to these release notes, marked New.

About the Virtual Disk Development Kit

The Virtual Disk Development Kit (VDDK) 8.0.3 is an update to support vSphere 8.0 Update 3 and respond to partner requests for enhancement.

VDDK is used with vSphere Storage APIs for Data Protection (VADP) to develop backup and restore software. For general information about this development kit, how to obtain the software, programming details, and redistribution, see the VDDK landing page on developer.vmware.com.

Changes and New Features

VDDK 8.0.3 offers improved support for NVMe over Fabrics:

New: VDDK supports SAN transport with NVMe over Fabrics.
vSphere 8 contained many enhancements regarding NVMe over Fabrics (NVMe-oF), such as more namespaces, extended reservations, discovery services, and vVol support. Interoperability issues were discovered with SAN transport, so backup applications used NBD/NBDSSL or HotAdd transport to back up NVME-oF datastores. As of 8.0 U3, VDDK also supports SAN transport for NVME-oF backup on a Linux backup proxy.
New: Documentation for backing up VM Service managed VM.
For instructions to backup and restore virtual machines managed by VM Service, a VMware Tanzu facility to simplify DevOps programming on Kubernetes, see chapter “Practical Programming Tasks” in the VDDK Programming Guide. This is a documentation-only feature; no related code changes were made to VDDK.
New: Open source components.
The following open source components were upgraded: curl 8.4.0 to 8.5.0, expat 2.5.0 to 2.6.2, libuv to 1.43.0, openssl 3.0.12 to 3.0.13, openssl FIPS 3.0.8 to 3.0.9, and zlib 1.3 to 1.3.1.

VDDK 8.0.2 offered the following features:

CBT troubleshooting procedure.
Steps for customers to troubleshoot failures in changed block tracking (CBT) were expanded and moved to chapter 10 of the VDDK Programming Guide.
Network resiliency.
Rather than return Error 13 “cannot open file” or “file read failed” after network connection errors, VixDiskLib_Open now returns the underlying network error, such as VIX_E_HOST_SERVER_SHUTDOWN for machine powered off, or VIX_E_HOST_NETWORK_CONN_REFUSED for network dysfunction. This allows backup applications to better handle network errors and react to them by retrying file open and read.
TKGS support for container snapshots.
Container Storage Interface added support for container snapshots in Tanzu Kubernetes Grid Service (TKGS). For details, see section “Container Storage Snapshot” in the VDDK Programming Guide.
Telemetry enhancement.
VDDK “phone home” operates more efficiently. Changes were transparent to backup applications.

For new features in previous VDDK versions, see the VDDK 8.0 U1 Release Notes and before.

Compatibility Notices

New: Microsoft ReFS not supported. All mentions of Resilient File System (ReFS) were removed from the 8.0.3 VDDK Programming Guide. VDDK 7.0 release notes stated that ReFS writes do not work with VixMntapi, but VMware recently discovered that ReFS mount does not work either.

VDDK 8.0 U3 was tested with the following systems for HotAdd proxy backup. Many additional systems are supported as guest operating systems for VADP.

Windows Server 2022
CentOS stream 9
Oracle Linux 8.9
Red Hat Enterprise Linux (RHEL) 9.1, 9.2, 9.3
SUSE Linux Enterprise Server (SLES) 12 SP5, 15 SP5
Photon v4, v5

This table shows recommended VDDK versions for various VMC milestones:

VMC Milestone	Compatible VDDK Versions
1.16 – 1.19	6.7.x, 7.0.x, 8.0.x
1.20, 1.22, 1.24, 1.26	7.0.x, 8.0.x

The VMware policy concerning backward and forward compatibility is for VDDK to support N-2 and N+1 releases. In other words, VDDK 8.0 and all its update releases work with vSphere 6.7, 7.0.x, and (except for new features) the next major release. Note however that vSphere 6.7 reached End of Technical Guidance (EoTG) in November 2023.

Here are possible issues with backward and forward compatibility.

More difficult configuration of NFC logs.
When customers want to change the NFC server log level to help diagnose NBD backup issues, they must do so with JSON edits of Config Store settings. For instructions, see “How to Change NFC Logging Level” in the VDDK 8.0.2 Release Notes.
Best practice for restore with pre-existing snapshot.
When a backed-up VM had snapshots, disaster recovery software can restore a fresh VM with its pre-existing snapshots in place. However for partial file recovery, or to recover a VM back to a point in time, it is difficult to determine how to restore a VM that has an existing snapshot. Should parts of the snapshot be restored to a previous state (if applicable), or only the VM by ignoring the snapshot? Certain backup-restore applications do not handle this situation well; snapshot contention causes a GenericVmConfigFault. One good solution is for customers to delete any pre-existing snapshots before recovery, especially if not needed. This is mandatory for recovery with SAN mode transport, anyway. Restore applications could refuse to recover a VM until any pre-existing snapshot is deleted. If not, a customer work-around is to recover the VM to a different location, avoiding the issue, then retain the recovered VM and delete the VM with pre-existing snapshot.

Recently Resolved Issues

The VDDK 8.0.3 release resolves the following issues.

New: NBD transport on separate vmk network adapter.
When customers attempted backup using NBD transport on an alternate network adapter, for example vmk2 instead of vmk0, NBD network traffic did not bind to the separate vmk when adapters were on the same subnet. It worked only when the network adapters were on different subnets. In this release, the backup traffic always binds to the vmk with “vSphere Backup NFC” service.
New: Quiesced snapshot support for a Windows VM with NVMe disks.
VMTools 12.4.0 and later support quiesced snapshot for Windows VMs with NVMe disks. However due to issues with the old Windows NVMe driver, application-consistent snapshot with NVMe disks is supported only for Windows Server 2022 build 1906 and later.
New: Documentation correction regarding NFC server timeouts.
Settings vixDiskLib.nfcFssrvr.TimeoutMs and vixDiskLib.nfcFssrvrWrite.TimeoutMs are not used in VDDK, so references to them were removed from the 8.0 U3 VDDK Programming Guide.

VDDK 8.0.1 and 8.0.2 resolved the following issues.

Customers complained about nonconfigurable crash dumps.
On Windows, vmacore crash dumps now appear in the VDDK configured tmpDir. On Linux, crash dumps appear as configured in the kernel, like they did before.
Open SSL library upgraded for FIPS support.
The Open SSL library openssl was upgraded to version 3.0 for vSphere 8.0 U1. To enable FIPS with Open SSL 3.0, see section “How to Enable FIPS on VDDK Proxy VM” in the VDDK 8.0.1 Release Notes.
Reminder about CBT Check in VDDK 7.0.
VDDK 7.0 included vixDiskCheck utility with cbtCheck option, and now readPerf and writePerf options. For details, find the VDDK bin64 folder and run vixDiskCheck for help on all its options.

For resolved issued in previous VDDK versions, see the VDDK 8.0 Release Notes and before.

Known Issues and Workarounds

These are unresolved issues known at this time.

New: Non-quiesced snapshot on Linux VM.
After a successful quiesced snapshot of a Linux guest, the snapshot may be reported as not quiesced. With VMware Tools versions older than 10.2.0, or ESXi versions older than 6.7, taking a quiesced snapshot on a Linux VM succeeds but appears as a non-quiesced snapshot, even though the snapshot is actually quiesced. A snapshot is identified as quiesced only if a backup manifest file was provided for the snapshot. This works on a Windows VM with VSS plugin, but is not accurate on a Linux VM using the sync driver to quiesce snapshots. To fix this, customers can upgrade vmtools to 10.2.0 or later, and ESXi to 6.7 or later, so it appears as a File-System quiesced snapshot.
Clean-up function not working with default tmpDirectory.
With a default temporary directory, the folder name can change from backup to backup, if the backup job is a new OS process, so the VixDiskLib_Cleanup function might not clean up everything left behind. To avoid this, programs can set tmpDirectory in the configuration file for VixDiskLib_InitEx.