times

VDDK 8.0.2 with vSphere 8.0 U2 | 21 September 2023 | VDDK build 22388865
For ESXi and vCenter Server GA | Last document update 01 April 2024
Check back for additions and updates to these release notes, marked New.

About the Virtual Disk Development Kit

The Virtual Disk Development Kit (VDDK) 8.0.2 is an update to support vSphere 8.0 Update 2 and respond to partner requests for enhancement.

The VMware policy concerning backward and forward compatibility is for VDDK to support N-2 and N+1 releases. In other words, VDDK 8.0 and all its update releases will support vSphere 6.7, 7.0.x, and (except for new features) the next major release.

VDDK is used with vSphere Storage APIs for Data Protection (VADP) to develop backup and restore software. For general information about this development kit, how to obtain the software, programming details, and redistribution, see the VDDK landing page on developer.vmware.com.

Changes and New Features

VDDK 8.0.2 offers the following features:

New: CBT troubleshooting procedure.
Steps for customers to troubleshoot failures in changed block tracking (CBT) were expanded and moved to chapter 10 of the VDDK Programming Guide.
New: Network resiliency.
Rather than return Error 13 “cannot open file” or “file read failed” after network connection errors, VixDiskLib_Open now returns the underlying network error, such as VIX_E_HOST_SERVER_SHUTDOWN for machine powered off, or VIX_E_HOST_NETWORK_CONN_REFUSED for network dysfunction. This allows backup applications to better handle network errors and react to them by retrying file open and read.
New: TKGS support for container snapshots.
In this release, Container Storage Interface added support for container snapshots in Tanzu Kubernetes Grid Service (TKGS). For details, see section “Container Storage Snapshot” on pages 85-86 of the VDDK Programming Guide.
New: Telemetry enhancement.
VDDK “phone home” operates more efficiently in this release. Changes are transparent to backup applications.
Quiesced snapshot for Windows VM with NVMe disks.
VMware Tools 12.4.0 is scheduled for release in spring 2024. It will support application quiesced snapshots for a Windows guest (with NVMe HotAdd issue fixed) running Server 2022 build 1906 or later, on virtual hardware version 21 or later. Failing these requirements, fallback is to filesystem quiescing.

VDDK 8.0.1 offered the following features:

Datastore-level access to VMDK.
In this release, it is possible to directly access virtual disks on a datastore. Programs do this by specifying VixDiskLibSpecType as Datastore. At the datastore level, there is no support for snapshots and CBT. See below for steps to employ this feature. Currently only NBD/NBDSSL and HotAdd transport modes are supported. Note that HotAdd transport does not support RDM disks. For datacenters with encrypted disks, NBDSSL transport mode is required, and clusters must activate vSphere HA to guarantee that all hosts have a crypto key ID.
Wildcards in allow and deny lists for SAN.
After configuring allowList and denyList for SAN transport mode, customers and partners requested a wildcard mechanism for ease of use. See section “Initialize Virtual Disk API” in the VDDK Programming Guide for details. VDDK 8.0.1 supports a new wildcard mechanism to shorten list making. You can use meta character expressions * ? [xyz] [a-z] [0-9] with the same meanings as in Bash shell.
Cache file for SAN disk device discovery.
Before doing a SAN mode backup, each instance of VDDK scans disks to determine the device name and VMFS ID of the LUN. In large datacenters with many SAN based disks, this has a significant negative performance impact. Partners requested a cache mechanism to reduce frequency of scans. As implemented, cache file scsiDiskList.json is created in the tmpDirectory specified by the VDDK configuration file. When VDDK fails to open a disk with cached LUN path, because disk was added or deleted since last scan, LUNs are rescanned and the JSON file overwritten. This cache mechanism operates without customer intervention.

For new features in the previous VDDK versions, see the VDDK 8.0 Release Notes and before.

Compatibility Notices

More difficult configuration of NFC logs. When customers want to change the NFC server log level to help diagnose NBD backup issues, they must do so with JSON edits of Config Store settings. See How to Change NFC Logging Level below.

VDDK 8.0 U2 supports the following operating systems for proxy backup (mostly the same as VDDK 8.0):

Windows Server 2022, 2019, and 2016
Red Hat Enterprise Linux (RHEL) 7.9, 8.1, 8.2, 8.6, 9.0
New: Red Hat Enterprise Linux (RHEL) 8.8
CentOS 7
CentOS 8.5
Oracle Linux 8.7
SUSE Linux Enterprise Server (SLES) 12 SP5 and 15.1
Ubuntu 18.04
Photon v3, v4

This table shows recommended VDDK versions for various VMC milestones:

VMC Milestone	Compatible VDDK Versions
1.16 – 1.19	6.7.x, 7.0.x, 8.0.x
1.20, 1.22, 1.24	7.0.x, 8.0.x

Here is a possible issue with backward and forward compatibility.

Best practice for restore with pre-existing snapshot. When a backed-up VM had snapshots, disaster recovery software can restore a fresh VM with its pre-existing snapshots in place. However for partial file recovery, or to recover a VM back to a point in time, it is difficult to determine how to restore a VM that has an existing snapshot. Should parts of the snapshot be restored to a previous state (if applicable), or only the VM by ignoring the snapshot? Certain backup-restore applications do not handle this situation well; snapshot contention causes a GenericVmConfigFault. One good solution is for customers to delete any pre-existing snapshots before recovery, especially if not needed. This is mandatory for SAN transport recovery, anyway. Restore applications could refuse to recover a VM until any pre-existing snapshot is deleted. If not, a customer work-around is to recover the VM to a different location, avoiding the issue, then retain the recovered VM and delete the VM with pre-existing snapshot.

How to Change NFC Logging Level

For security reasons ESXi 8.0 U2 and VDDK 8.0.2 make it difficult for NFC clients to change the NFC server log level. Now it must be done with JSON edits of Config Store. Because NFC log level set in the VDDK configuration file accounts for changes in various vSphere releases, customers are not affected until they run vSphere 8.0 U2.

For troubleshooting NBD(SSL) backups, configuration of NFC logging changed in almost every vSphere release. Here is a summary:

In vSphere 7.0 and before, NFC log level was set in the vpxa.cfg file.
In vSphere 7.0 U1, NFC logs are created by hostd rather than vpxa.
In vSphere 7.0 U2, configuration of NFC log level uses Config Store, set with JSON.
VDDK 8.0.2 and vSphere 8.0 U2 do not allow NFC clients to change the NFC server log level.

Summarizing the various combinations of ESXi and VDDK:

With ESXi 8.0 U1 and earlier paired with VDDK 8.0.1 and earlier, users can still use vixDiskLib.nfc.LogLevel in the VDDK configuration file to set the NFC server log level.

With ESXi 7.0 and earlier paired with VDDK 8.0.2, users must change the vpxa configuration file and restart vpxa.

ssh to the ESXi host
Edit /etc/vmware/vpxa/vpxa.cfg

 <nfc>
  <loglevel>debug</loglevel>
 </nrc>

Run command: /etc/init.d/vpxa restart

With ESXi 7.0 U1 paired with VDDK 8.0.2, users must change the hostd log level and restart hostd.

ssh to the ESXi host
Run command: vim-cmd hostsvc/advopt/update Config.HostAgent.log.level string trivia
Run command: /etc/init.d/hostd restart

With ESXi 7.0 U2 and later paired with VDDK 8.0.2, users must create a temporary file to change settings in Config Store and restart hostd.

ssh to the ESXi host
Run command: configstorecli config current get -c esx -g services -k hostd > /tmp/cs.json
Edit /tmp/cs.json

  "nfcsvc": {      
    "log_level": "TRIVIA",
    "max_memory": 100663296,
    "max_stream_memory": 35651584
  },

Run command: configstorecli config current set -c esx -g services -k hostd -infile /tmp/cs.json
Run command: /etc/init.d/hostd restart

With ESXi 8.0 U2 and any VDDK release, users must follow the JSON steps above.

Recently Resolved Issues

The VDDK 8.0.1 release resolved the following issues.

Customers complained about nonconfigurable crash dumps.
On Windows, vmacore crash dumps now appear in the VDDK configured tmpDir. On Linux, crash dumps appear as configured in the kernel, as they did before.
Open SSL library upgraded for FIPS support.
The Open SSL library openssl is upgraded to version 3.0 for vSphere 8.0 U1. To enable FIPS with Open SSL 3.0, see below.
Reminder about CBT Check in VDDK 7.0.
VDDK 7.0 included vixDiskCheck utility with cbtCheck option, and now readPerf and writePerf options. For details, find the VDDK bin64 folder and run vixDiskCheck for help on all its options.

For resolved issued in previous VDDK versions, see the VDDK 8.0 Release Notes and before.

Known Issues and Workarounds

These are unresolved issues known at this time.

New: Non-quiesced snapshot on Linux VM.
After a successful quiesced snapshot of a Linux guest, the snapshot may be reported as not quiesced. With VMware Tools versions older than 10.2.0, or ESXi versions older than 6.7, taking a quiesced snapshot on a Linux VM succeeds but appears as a non-quiesced snapshot, even though the snapshot is actually quiesced. A snapshot is identified as quiesced only if a backup manifest file was provided for the snapshot. This works on a Windows VM with VSS plugin, but is not accurate on a Linux VM using the sync driver to quiesce snapshots. To fix this, customers can upgrade vmtools to 10.2.0 or later, and ESXi to 6.7 or later, so it appears as a File-System quiesced snapshot.
VDDK non-support of NVMe over Fabrics.
vSphere 8 contains many enhancements regarding NVMe over Fabrics (NVMe-oF) such as more namespaces, extend reservations, discovery services, and vVol support. However it seems unlikely that VDDK handles backup for all these new features. More information will be forthcoming. Meanwhile applications can use NBD/NBDSSL and HotAdd transport to back up NVME-oF datastores.
Clean-up function not working with default tmpDirectory.
With a default temporary directory, the folder name can change from backup to backup, if the backup job is a new OS process, so the VixDiskLib_Cleanup function might not clean up everything left behind. To avoid this, programs should set tmpDirectory in the configuration file.