times

VDDK 7.0.3 | 5 Oct 2021 | ESXi build 18644231, VDDK build 18657718
For vSphere 7.0 Update 3 and VMC | Last document update 27 Sept 2021
Check back for additions and updates to these release notes.

About the Virtual Disk Development Kit

The Virtual Disk Development Kit (VDDK) 7.0.3 is an update to support vSphere 7.0 U3 and resolve some reported issues. VDDK 7.0.3 supports ESXi 7.0, vCenter Server 7.0, and VMware Cloud (VMC). Previous 7.0 releases were tested for backward compatibility against vSphere 6.5 and 6.7.

The VMware policy concerning backward and forward compatibility is for VDDK to support N-2 and N+1 releases. In other words, VDDK 7.0 and all its update releases will support vSphere 6.5, 6.7 (except for new features), and the next major release.

VDDK is used with vSphere Storage APIs for Data Protection (VADP) to develop backup and restore software. For general information about this development kit, how to obtain the software, programming details, and redistribution, see the VDDK landing page on code.vmware.com.

Changes and New Features

VDDK 7.0.3 offers the following:

Configurable NFC AIO buffers.
As of VDDK 7.0.3, users can configure asynchronous I/O buffers for NBD(SSL) transport. With high latency storage, backup and restore performance may improve after increasing NFC AIO buffer size. If servers are capable of high concurrency, backup and restore throughput may improve with more NFC AIO buffers. Defaults are buffer size 64K (1) and buffer count 4. NFC server memory consumption increases with larger NFC AIO buffer sizes and counts. Memory for each session can be calculated by multiplying buffer size * buffer count. If session memory exceeds 16MB, disk open operations will fail. In testing, these InitEx configuration file settings performed best, but this depends on hardware setup:
vixDiskLib.nfcAio.Session.BufSizeIn64KB=16 vixDiskLib.nfcAio.Session.BufCount=4
The following are newly supported as operating systems for HotAdd proxy:
- Red Hat Enterprise Linux (RHEL) 8.3
- CentOS 8.3
- SUSE Linux Enterprise Server (SLES) 15.1
Bug fixes as below.

For new features in previous VDDK version, see the VDDK 7.0 Release Notes, VDDK 7.0 U1 Release Notes, and VDDK 7.0 U2 Release Notes.

Compatibility Notices

Missing from documentation (until 7.0.1) was this caveat about multithreading: QueryAllocatedBlocks should be called in the same thread as open and close disk, not from read and write worker threads. See also “Thread Local Storage” below.

vVol and vSAN datastores do not support SAN mode transport. As of VDDK 6.7, SAN transport is explicitly rejected for vSAN and vVols.

As of 6.7.3 and in vSphere 7.0, the VDDK library marks parentHandle internally to prevent closure and ensure cleanup. In earlier releases it was an error to close parentHandle after VixDiskLib_Attach succeeds. See the VDDK 6.7.1 Release Notes for coding advice.

The following table shows recommended VDDK versions for various VMC milestones.

VMC Milestone	Compatible VDDK Versions
M8	6.5.2, 6.5.2 EP1, 6.5.3, 6.5.4, 6.7.x, 7.0.x
M9	6.5.2, 6.5.2 EP1, 6.5.3, 6.5.4, 6.7.x, 7.0.x
M10, M11, M12, M13, M14, M15, M16	6.7.x, 7.0.x

Recently Resolved Issues

The VDDK 7.0.3 release resolves the following issues.

Thread local storage documented.
On Windows, backup applications should call VixDiskLib_InitEx at the very beginning of the program. Some thread local storage was introduced in VDDK 7.0, so now all threads that call VixDiskLib functions must be created and initialized after calling VixDiskLib_InitEx. This is documented in the 7.0.3 VDDK Programming Guide.
Race condition between HotRemove snapshot and host EMM.
When customers run HotAdd backup, which removes a snapshot after backup, and the ESXi host simultaneously enters maintenance mode (EMM), backup software usually tries to clean up by deleting the snapshot. Actually after EMM conflict, all snapshot disks on the VM must be deleted. For instance during ESXi upgrade, the backup proxy VM cannot power off, so that it can then vMotion to the destination host and resume. Thus it is best to delay HotAdd backup when a host signals EMM. VDDK provides or will provide an EMM notification facility so partners can work around this issue.
FCD mount failure on Linux.
With HotAdd backup being performed on a Linux virtual machine, VixMntApi failed to mount First Class Disk (FCD), with error “No volume found to perform verification of reading files.“ The cause was a conflict between HotAdded disk cleanup and FCD marker data. This issue is fixed in VDDK 7.0.3.
Could not unmount error appears.
During a connect, open, read, close, and disconnect sequence, the ssl directory is sometimes not cleaned up, resulting in “Could not unmount” errors. Other situations may cause cleanup failures with similar errors. The cause is that the unmount.dat info file is be missing while its parent folder still exists, leading to cleanup failure. The workaround was to manually delete the parent directory. VDDK 7.0.3 cleans up the parent directory even if the info file is missing.
Meaningful message for datastore full from VixDiskLib write.
Customers requested a more meaningful error when write fails because a datastore is full. The VixDiskLib_write call returned “code 1: Unknown error” although it reports “not enough space” in the VDDK log. As of VDDK 7.0.3, VixDiskLib_write returns a more accurate error.
Preserving previous CBT configuration.
When VirtualMachineDefaultProfileSpec was applied to a VM during ReconfigVM_Task, changed block tracking (CBT) became enabled for all virtual disks, even if CBT had not been previously enabled for certain disks. In vSphere 7.0 U3, the deviceChange list in reconfigure VM task was enhanced to preserve CBT settings, so applying the default VM profile will not change CBT.
Unable to backup VM disks with long names.
VDDK backups fail when virtual machines have long names, as generated by IO Filters. The NFC error message “Cln invalid msg hdr magic” is not helpful, but the VDDK logs say “Invalid string length” with a number. The cause was a 256 byte limitation on the length of NFC key/value pairs. This was increased to 8192 bytes in vSphere 7.0 U3, matching 7.0.3 VixDiskLib.

Known Issues and Workarounds

These are unresolved issues known at this time.

Linked clone problems with HotAdd backup on vVol datastores.
New: If backup software creates a linked clone target VM from a linked clone VM on a vVol datastore, then performs a backup using HotAdd transport, the proxy VM might become invalid when opening a disk on that target VM. This is a regression from VDDK 7.0.2 to 7.0.3. One alternative is to create the linked clone target VM from a full-clone source VM, rather than from a linked clone VM. Another alternative is to choose a transport mode other than HotAdd.
Clean-up function not working with default tmpDirectory.
With a default temporary directory, the folder name can change from backup to backup, if the backup job is a new OS process, so the VixDiskLib_Cleanup function might not clean up everything left behind. As a workaround, programs should set tmpDirectory in the configuration file.
VDDK 6.7 did not support Windows Server 2019 as a backup proxy.
When using Windows Server 2019 as a HotAdd backup proxy with vSphere 6.x, every target VM gets flagged for reboot after backup. The user can ignore this message, as a reboot is not required. and subsequent HotAdd backups will continue to work. This issue was fixed in the VDDK 7.0 release by opening the disk itself instead of the disk adapter.
HotAdd transport mode limitations for vSAN datastores.
If the target VM is on a vSAN datastore and the backup proxy on a non-vSAN datastore, HotAdd transport is not supported. VMware may add support in a future release. A workaround is to vMotion the backup proxy to a vSAN datastore.
Errata in VDDK documentation regarding CEIP phone home.
In the VDDK 6.7 programming guide, section “Phone Home Support” on page 48 was inaccurate. In vSphere 7.0, CEIP is customer controlled. The EnablePhoneHome=1 setting in the VDDK configuration file has no effect. However, backup software should set vendor details as recommended below, otherwise vendor name and version will appear as “Unknown” in VMware analytics. Legal characters: 26 letters, digits, underscore (_), minus (-), period (.), and space. Double quotes are mandatory if setting strings contain spaces. DatabaseDir stores phone home data in a separate folder.
```
vixDiskLib.phoneHome.ProductName = "vendorName or ApplicationName"
vixDiskLib.phoneHome.ProductVersion = "versionNumber"
vixDiskLib.phoneHome.DatabaseDir = "folderName"
```
vSphere 6.7 HTTP File Service not backward compatible until 6.7 U2.
This is related to “Disk open fails in HotAdd mode if name contains special characters” in 6.7.2. Backup partners with on-prem customers running VDDK 6.7.0 or VDDK 6.7.1 can recommend the NoNfcSession=0 setting to work around the problem, and urge an upgrade to VDDK 6.7.2 or 7.0 when feasible. Partners with cloud-based customers should make upgrade to post M5 and VDDK 7.0 mandatory, because NoNfcSession=1 must be set for security reasons. Partners using the Http_Access API in their own programs should code a fix to support multiple vCenter versions with adaptive single and double decoding. This issue will be permanent for partners supporting vSphere 6.7, 6.7 U1, and M5.