Virtual Disk Development Kit (VDDK) 8.0 Release Notes

VDDK for vSphere 8.0 | 11 October 2022 | ESXi and vCenter Server, VDDK build 20521017 Initial Availability became GA 8 November 2022 | Last document update 2 November 2023 Check back for additions and updates to these release notes, marked New.

About the Virtual Disk Development Kit

The Virtual Disk Development Kit (VDDK) 8.0 is an update to support vSphere 8.0 and resolve some reported issues.

The VMware policy concerning backward and forward compatibility is for VDDK to support N-2 and N+1 releases. In other words, VDDK 8.0 and all its update releases will support vSphere 6.7, 7.0.x, and (except for new features) the next major release.

VDDK is used with vSphere Storage APIs for Data Protection (VADP) to develop backup and restore software. For general information about this development kit, how to obtain the software, programming details, and redistribution, see the VDDK landing page on developer.vmware.com.

Changes and New Features

VDDK 8.0 offers the following features:

• New: Backing up Docker containers.

  RedHat Enterprise Linux 8 (RHEL8) Universal Base Images (UBI) as a container image has been tested with VDDK backups using NBD/NBDSSL transport. HotAdd transport is still under investigation. VDDK backup of CentOS 8 containers with Base OS Photon V3 or V4 is supported using NBD/NBDSSL mode.

• New: DataSets backup.

  A new section in the programming guide describes how to back up DataSets, a vSphere 8.0 feature for sending and storing data in virtual machines.

• Better order matching of controller numbering.

  KB 67757 and the VDDK Programming Guide for 7.0 U3 describe how to power cycle and correct the backup proxy VM when there is a mismatch between SCSI controllers and host initiators. Disordered SCSI numbering causes HotAdd backup to find the wrong disk. HotAdd logic is changed to automatically match SCSI host initiator numbers with SCSI controller labels. This fix made in VDDK 7.0.3.2 is for Linux, although Windows backup proxies sometimes have a similar issue.

• Improved NBD backup performance.

  On Windows, NBD transport performance was improved in VDDK 7.0.3.2 by changing the default size for NFC send and receive socket buffers to 128KB. On Linux, various socket buffer sizes have minimal performance impact. Customers do not need to change defaults, in case of performance issues, they can change four buffer settings under vixDiskLib.nfcAio.SocketOption in the VDDK configuration file.

• Fewer network connections.

  A fix made in VDDK 7.0.3.2 reduces the number of connections from VDDK to vCenter Server, especially CEIP queries, as requested by backup partners.

• Updated Vix mount functions.

  Version 1.1 of VixMntapi library with new GetVolumeInfoEx function that propagates these fields: bytesPerSector, capacityInBytes, usedSpace, and clusterSizeInBytes (usedSpace and clusterSizeInBytes could be inaccurate if the volume is not mounted).

• FCD change tracking.

  VDDK documentation describes how to use QueryChangedDiskAreas for VM disk. This function can be used for FCD as well. There are two cases for FCD, attached to VM or not. If FCD is attached to a VM, you can use the same API as for VM disk. If FCD is detached, you must call FCD’s QueryChangedDiskAreas API directly to get diffs.

• Containers in NBD mode.

  This release supports NBD and NBDSSL backup of containers. VDDK programs can run in a general container instance.

• New proxy OS support.

  VDDK 8.0 supports five additional operating systems as backup proxy VMs for HotAdd and other transports: Red Hat Enterprise Linux (RHEL) 8.1, 8.2, Photon v3, v4, and Windows 2022, in addition to those supported by VDDK 7.0.3.

For new features in previous VDDK versions, see the VDDK 7.0.3.2 Release Notes and before.

Compatibility Notices

VDDK 8.0 supports the following operating systems for proxy backup:

• Windows Server 2022, 2019, and 2016
• Red Hat Enterprise Linux (RHEL) 7.9, 8.1, 8.2, 8.6, 9.0
• New: RHEL 8.8 was retroactively verified after support in VDDK 8.0.3
• CentOS 7
• SUSE Linux Enterprise Server (SLES) 12 SP5 and 15.1
• Ubuntu 18.04
• Photon v3, v4

This table shows recommended VDDK versions for various VMC milestones:

VMC Milestone	Compatible VDDK Versions
M16 – M19	6.7.x, 7.0.x, 8.0.x
M20	7.0.x, 8.0.x

Here are possible issues with backward and forward compatibility.

• New: Best practice for restore with pre-existing snapshot. When a backed-up VM had snapshots, disaster recovery software can restore a fresh VM with its pre-existing snapshots in place. However for partial file recovery, or to recover a VM back to a point in time, it is difficult to determine how to restore a VM that has an existing snapshot. Should parts of the snapshot be restored to a previous state (if applicable), or only the VM by ignoring the snapshot? Certain backup-restore applications do not handle this situation well; snapshot contention causes a GenericVmConfigFault. One good solution is for customers to delete any pre-existing snapshots before recovery, especially if not needed. This is mandatory for SAN transport recovery, anyway. Restore applications could refuse to recover a VM until any pre-existing snapshot is deleted. If not, a customer work-around is to recover the VM to a different location, avoiding the issue, then retain the recovered VM and delete the VM with pre-existing snapshot.
• Superseded years ago by the CreateSnapshotEx_Task method, CreateSnapshot_Task is deprecated in this release.
• In the configuration file for SAN transport mode, the original keywords that denyList and allowList replaced are no longer accepted for backward compatibility, as they have been since the changeover in vSphere 7.0 U1.
• Missing from documentation (until 7.0.1) was this caveat about multithreading: QueryAllocatedBlocks should be called in the same thread as open and close disk, not from read and write worker threads.
• vVol and vSAN datastores do not support SAN mode transport. SAN transport is explicitly rejected for vSAN and vVols.

Recently Resolved Issues

The VDDK 8.0 release resolves the following issues.

• Snapshot support for cloud native storage (CNS).

  A new section in the programming guide describes how to create volume snapshots in a Kubernetes cluster and retrieve snapshot handles using container storage interface (CSI) based on first class disk (FCD). Cloud native storage (CNS) is a vSphere feature that allows Kubernetes to auto-provision scalable storage on demand. CNS also provides vSphere administrator visibility into container volumes through vCenter Server.

• Could not set NFC server log level.

  In VDDK 8.0 updates, the NFC server logging level will no longer configurable from the VDDK client configuration file. This was originally for debugging but will be discontinued for security reasons.

• Open source libraries upgraded.

  The Open SSL library openssl was upgraded to version 1.0.2zf, the XML streaming library libexpat to version 2.4.9, and the data compression library Zlib to version 1.2.12.

For resolved issued in previous VDDK versions, see the VDDK 7.0.3.2 Release Notes and before.

Known Issues and Workarounds

These are unresolved issues known at this time.

• New: VDDK non-support of NVMe over Fabrics.

  vSphere 8 contains many enhancements regarding NVMe over Fabrics (NVMe-oF) such as more namespaces, extend reservations, discovery services, and vVol support. However it seems unlikely that VDDK handles backup for all these new features. More information will be forthcoming. Meanwhile applications can use NBD/NBDSSL and HotAdd transport to back up NVME-oF datastores.

• Linux backup proxy requires minimum vmx=10.

  Because of a resiliency improvement made for VDDK 7.0.3.2 (corrected HotAdd failures due to mismatch of controller ordering) a Linux backup proxy no longer works with virtual hardware version 9 (vmx=9). Error message is “Could not find the label of controller 0.” The workaround is to upgrade backup proxies to virtual hardware version 10 or later.

• On VMC with NFS, Storage vMotion may conflict with backup.

  During backup a VM's disks are locked against removal. If Storage vMotion is requested against that VM, VM disks remain afterwards, and must be removed manually. The longer backup takes, the higher the likelihood of this occurring. Customers should avoid Storage vMotion during the backup window, but if not, they can find VMDKs not associated with VM or FCD, and remove them.

• Linked clone problems with HotAdd backup on vVol datastores.

  If backup software creates a linked clone target VM from a linked clone VM on a vVol datastore, then performs a backup using HotAdd transport, the proxy VM might become invalid when opening a disk on that target VM. This was a regression from VDDK 7.0.2 to 7.0.3. One alternative is to create the linked clone target VM from a full-clone source VM, rather than from a linked clone VM. Another alternative is to choose a transport mode other than HotAdd.

• Clean-up function not working with default tmpDirectory.

  With a default temporary directory, the folder name can change from backup to backup, if the backup job is a new OS process, so the VixDiskLib_Cleanup function might not clean up everything left behind. To avoid this, programs should set tmpDirectory in the configuration file.