Release date: 17 April 2018 | ESXi build 8169922, VDDK build 8173251 For vSphere 6.7 GA. Last document update: 14 December 2020 Check frequently for additions and updates to these release notes.

About the Virtual Disk Development Kit

The Virtual Disk Development Kit (VDDK) 6.7 is an update to support vSphere 6.7 and to resolve issues discovered in previous releases. VDDK 6.7 adds support for ESXi 6.7 and vCenter Server 6.7, and was tested for backward compatibility against vSphere 6.0 and 6.5.

VDDK is used with vSphere Storage APIs for Data Protection (VADP) to develop backup and restore software. For general information about this development kit, how to obtain the software, programming details, and redistribution, see the VDDK documentation landing page.

The VMware policy concerning backward and forward compatibility is for VDDK to support N-2 and N+1 releases. In other words, VDDK 6.7 and all its update releases support vSphere 6.0, 6.5 (except for new features) and the next major release.

VDDK 6.7 was tested with the following operating systems to perform proxy backup:

• Windows Server 2016, including version 1709
• Windows Server 2012 and 2012 R2
• Windows Server 2008 R2
• Red Hat Enterprise Linux RHEL 6.7, 6.8, 7.2, and 7.3
• SUSE Linux Enterprise Server SLES 11 SP4 and 12 SP1

Changes and New Features

Back up only allocated blocks. As of vSphere 6.7, developers can call VixDiskLib_QueryAllocatedBlocks to accelerate full backups, especially of large virtual disks, and to assist incremental or differential backups. By knowing which blocks are allocated, an application can avoid reading uninitialized data and thus reduce data transfer during backup. As a consequence, the QueryChangedDiskAreas asterisk option "*" with similar funcationlity is deprecated. Query allocated blocks does not depend on changed block tracking (CBT). You can enable CBT either before or after the call, and behavior is the same. One limitation is that VixDiskLib_QueryAllocatedBlocks returns the whole VMDK for NFS, rather than just the allocated blocks as for VMFS.

Asynchronous mode NBDSSL. Performance of NBDSSL mode data transfer can be improved with this option. To implement asynchronous I/O for NBDSSL mode, use the new functions VixDiskLib_ReadAsync and VixDiskLib_WriteAsync in a callback, followed by VixDiskLib_Wait, instead of the VixDiskLib read and write functions. See vixDiskLibSample.cpp in the SDK for a code example, under -readasyncbench and -writeasyncbench options.

Back up detached IVD (FCD). Improved virtual disk (IVD), formerly first class disk (FCD), is a named virtual disk unassociated with a VM. See the Virtual Disk Programming Guide for procedures to back up and restore detached IVD.

INF information file to install Vstor2 driver. In previous VDDK releases, the vstor2 driver was signed by VMware. In this release, the vstor2 driver is signed by Microsoft. This avoids signature checking problems with virtualization based security (VBS).

New Windows security features. In vSphere 6.7, virtualization based security (VBS) is supported for Windows 10 and Server 2016 VMs, although versions 1607 and 1703 require patches. Check Microsoft documentation for ESXi host hardware compatibility. UEFI with Secure Boot is required to enable VBS inside the Windows guest OS. This is the default virtual hardware version 14 and later, also required. Upgrading BIOS of pre-14 virtual hardware to UEFI requires reinstallation, so VMware recommends selecting UEFI for pre-14 virtual hardware VMs. Fault Tolerance, PCI pass-through, and memory or CPU HotAdd do not yet work with VBS. Disk HotAdd does work, fortunately, See the Virtual Disk Programming Guide for advice on virtual disk backup and restore with VBS and vTPM (virtual trusted platform module).

NFC connection skipped for HotAdd backups. In VMware Cloud environments (SDDC as a Service), only HotAdd transport mode is allowed. This avoids the need for a direct connection between the backup proxy and ESXi hosts, which is seldom permitted by the security topology of VMware Cloud. Previously, an NFC connection was always made during HotAdd, but is optional in this release, by setting vixDiskLib.transport.hotadd.NoNfcSession=1 in the proxy's VDDK configuration file. In cloud enviroments, backup software should check the return code of VixDiskLib_PrepareForAccess, and if VIX_E_HOST_USER_PERMISSIONS (3014), ignore the error and continue (not necessary in on-premise datacenters).

Phone home support. Customer Experience Improvement Program (CEIP) is a way for VMware products to send telemetry data from on-premise and cloud-based products back to VMware. VMware will share analytics to help partners make data-driven decisions for their products. Partners can enable phone-home by adding the line below to the VDDK configuration file, and customers can disable it by removing the line or setting to zero. VMware asks that partners add the two subsequent lines for identification purposes. Using the vSphere Web Client, customers can exit CEIP by clicking Menu > Administration > Customer Experience Improvement Program > Leave. When customers leave CEIP, VDDK telemetry is turned off also, even if enabled in the VDDK configuration file. Double quotes are mandatory if ProductName or ProductVersion have spaces.

vixDiskLib.phoneHome.EnablePhoneHome = 1
vixDiskLib.phoneHome.ProductName = "vendorName or ApplicationName"
vixDiskLib.phoneHome.ProductVersion = "versionNumber"

Compatibility Notices

ESXi hosts and especially vCenter Server should be upgraded in conjunction with VDDK libraries. For customers to get the all latest features and fixes, ESXi and vCenter Server versions should match the VDDK version.

On Windows, VDDK 6.7 depends on the Microsoft Visual C++ vc120 runtime, a change from vc90 previously. However even on Windows Server 2012 R2 and 2008 R2, VMware Tools still depend on vc90.

Signed vstor2 driver. Previous VDDK releases installed a VMware signed vstor2 driver. To avoid problems on Windows systems with strict signature checking, the new vstor2install.bat script installs a Microsoft signed driver package (WHQLed). If vstor2 is already installed on a backup proxy, you should first uninstall it with vstor2uninstall.bat (from the corresponding VDDK version) then run the new vstor2install.bat script. These scripts require administrator privilege. Partners can write custom (un)install scripts in accordance with Microsoft documentation about the “InstallHinfSection function” on MSDN.

Unbuffered HotAdd restore. VMware recommends that programmers set the VDDK flag VIXDISKLIB_FLAG_OPEN_UNBUFFERED when opening virtual disks before performing a restore with HotAdd transport. In vSphere 6.7 releases and later, programs must allocate a data buffer whose memory address is sector size aligned when setting this flag. Programmers may refer to posix_memalign on Linux and _aligned_malloc on Windows. Buffer address alignment to sector size is recommended for older VDDK releases as well.

Recently Resolved Issues

The VDDK 6.7 release resolves the following issues.

• Maximum 25 NFC sessions with ESXi 6.5 hosts.

  In all 6.5 releases of ESXi, including 6.5 U2, the number of NFC sessions was limited to 25, due to a lowered thread count limit. This reduced the concurrency of NBD backups to 25 from 48, as documented for vSphere 6. The issue could be fixed by upgrading to ESXi 6.7 hosts. The NFC server in vSphere 6.7 does not create a new thread for each session, so the thread limit is never hit. The ESXi 6.7 solution may be backported to a future vSphere 6.5.x release.

• Could not connect directly to ESXi on IPv6 networks.

  After connecting to an ESXi host with VixDiskLib_Connect(Ex) in NBD(SSL) mode on an IPv6 network, opening virtual disk with VixDiskLib_Open throws this error: “Host address lookup for server (null) failed: No such host is known.” NBD(SSL) connection to vCenter Server work, as does HotAdd mode. The IPv6 issue is fixed in this release.

• VDDK called an old Visual C++ 9.0 runtime.

  As mentioned above, VDDK 6.7 now calls vc120, although VMware Tools still requires vc90 runtime.

• New more-efficient function call for Query Allocated Blocks.

  In this release the VixDiskLib_QueryAllocatedBlocks function replaces QueryChangedDiskAreas with asterisk change ID ("*") and is more efficient especially for large data sets.

• Intermittent hang observed when opening virtual disks.

  A customer observed intermittent hanging while opening virtual disks. The hang was found to occur inside the VixDiskLibVim module, in VixDiskLib_PrepareForAccess, VixDiskLib_EndAccess, or GetNfcTicket, and may be caused by network problems. In this release, one fix is a timeout for HTTP requests, and another is a timeout for the open operation.

• VDDK crashes after disk open call and session login attempt.

  When trying to retrieve the NFC ticket for a disk after VixDiskLib_OpenEx, VDDK crashed intermittently (due to a null pointer read in gvmomi) when it attempted a session login to vCenter Server. This issue, similar to the one below, is fixed in this release.

• VDDK crashes after disk open call after running for some time.

  When trying to load transport modules after calling VixDiskLib_Open, VDDK crashed every few days due to a null pointer error in gvmomi. This issue, similar to the one above, is fixed in this release.

• SAN mode access to vSAN or VVol datastore hangs ESXi host.

  VVol and vSAN datastores do not support SAN mode transport. If a program calls VixDiskLib_ConnectEx on a VVol or vSAN datastore, specifying SAN mode or nothing (auto-detected SAN mode), the ESXi host stops responding and the VDDK connection fails. The workaround is to explicitly specify another mode (NBDSSL or HotAdd). In this release, SAN transport is explicitly rejected for for VVol and vSAN.

• QueryChangedDiskAreas("*") could return incorrect changed areas.

  With changed block tracking (CBT) enabled on NFS datastores, incremental backups work. However QueryChangedDiskAreas("*") returned all disk areas for initial full backup, so a thin-provisioned virtual disk would get restored as thick. The VDDK 6.5 Release Notes have a workaround for this, but in this release, programs should use VixDiskLib_QueryAllocatedBlocks to avoid the issue.

• Global > License privilege not required for backup. In the vSphere 6.5 Virtual Disk Development Kit Programming Guide it said the Global > License privilege is required for backup, however a license has not been required since vSphere 6.0.

• QueryChangedDiskAreas("*") could be slow and throw FileFault errors.

  The QueryChangedDiskAreas call with asterisk change ID ("*") was slow on large filesystems, and could throw a FileFault error with linked clones. In this release, the VixDiskLib_QueryAllocatedBlocks function should be used instead to determine which sectors are allocated in a VMDK.

For VDDK issues that were resolved before the vSphere 6.7 release, see the VDDK 6.5.x Release Notes.

Known Issues and Workarounds

Several issues were reported by partners after the 6.7 release, others earlier.

• New: I/O threads possibly not initialized on Windows.

  The diskLibPlugin module is dynamically loaded into vixDiskLib when programs call VixDiskLib_InitEx. On Windows, dynamic loading may result in the static thread for local storage not getting initialized. Some operations, such as fetch block map, can fail if I/O threads are created before calling VixDiskLib_InitEx. The workaround is to wait for completion of VixDiskLib_InitEx before creating any I/O threads.

• Clean-up function not working with default tmpDirectory.

  With a default temporary directory, the folder name can change from backup job to job, so the VixDiskLib_Exit function might not clean up everything left behind. As a workaround, programs should set tmpDirectory in the configuration file.

• HotAdd transport mode limitations for vSAN datastores.

  (1) If the target VM is on a vSAN datastore and the backup proxy on a non-vSAN datastore, HotAdd transport is not supported. VMware may add support in a future release. A workaround is to vMotion the backup proxy to a vSAN datastore. (2) If the target VM is on a VMFS or NFS datastore and the backup proxy on a vSAN datastore, HotAdd backup is not supported if SEsparse disks exist. SEsparse disks result when taking snapshots on VMFS-5 or NFS if disk size is >2TB, and always when taking snapshots on VMFS-6 datastores.

• Errata in VDDK documentation regarding CEIP phone home.

  In the VDDK programming guide, section “Phone Home Support” on page 48 is inaccurate. This should be the replacement text:
  The Customer Experience Improvement Program (CEIP) helps VMware understand what products and versions have been installed on a system. This information is collected to assist and streamline problem determination and resolution. Whether this information gets sent to VMware is a customer choice (on-premise) with CEIP opt-in and opt-out, or as part of their Terms of Service. VMware will collect only the product name and version information for improved problem resolution. No other partner-specific information will be collected. VMware asks that vendors add these lines to the VDDK configuration file for identification:

  vixDiskLib.phoneHome.EnablePhoneHome = 1
  vixDiskLib.phoneHome.ProductName = vendorName or ApplicationName
  vixDiskLib.phoneHome.ProductVersion = versionNumber
  

• Forward compatiblity issue with hardware version 17 on vSphere.Next.

  When using HotAdd transport to save a VM of virtual hardware version (VHV) 17 with Changed Block Tracking (CBT) enabled, backup will fail if virtual disk size is > 256GB, because large disks in vSphere.Next require change tracking (CTK) version 2.0, not supported by vSphere 6.7. The VM's log file indicates that ReconfigureVM failed, and the UI shows “Unknown change tracking version” status. To summarize, this happens under these conditions: (1) the backed up VM is VHV 17 or higher, (2) saved disk dize is > 256GB, and (3) its CBT was (re)initialized on a post-6.7.x ESXi host. A workaround is for users to disable CBT for the target VM to avoid CTK, but this is not recommended because then full backups will ensue. The solution is to vMotion the proxy VM to ESXi.Next or greater, which supports CTK version 2.0.

• VMDK managed by Storage Spaces cannot support application quiesced snapshots.

  For virtual disks managed by Windows Storage Spaces, application quiesced snapshots are not supported because of a VMware limitation. A fix has been identified and will appear in a future release. The workaround is for customers to create snapshots with filesystem quiescing instead, by adding these lines to the C:\ProgramData\VMware\VMware Tools\tools.conf file:

  [vmbackup]
  vss.disableAppQuiescing=true 

• After guest OS calls unmap, CBT reports more changed blocks than expected.

  If a guest VM has automatic space reclamation (unmap) enabled, incremental backup sometimes takes extraordinary long. When you check blocks that were backed up, you find the total size of those blocks is close to the total disk size, regardless how many blocks were actually modified. This occurs in all versions of vSphere 6.5, vSphere 6.7, and VMware Cloud (VMC).

  When automatic space reclamation is enabled in the guest, the OS issues unmap requests to underlying storage. However requested blocks include not only unmapped blocks but also unallocated blocks. All those blocks are captured by CBT and considered “changed blocks” then returned to backup software by the vSphere API queryChangedDiskAreas(changeId).

  Backup software can filter out unallocated blocks if it calls VDDK 6.7 or later libraries. It gets allocated area of a virtual disk by calling the VDDK API VixDiskLib_QueryAllocatedBlocks() then taking the intersection with changed blocks reported by CBT. The result is the set of actually changed blocks.

  One workaround is to disable unmap in the guest OS, but that disables space reclamation capability. If your software can use VDDK 6.7 or later libraries, a solution is to take the intersection of VixDiskLib_QueryAllocatedBlocks() and queryChangedDiskAreas(changeId) to calculate the actually changed blocks. See the 6.7 VDDK Programming Guide for details.

• NBD transport in VDDK 6.7 is slow when running against vSphere 6.5.

  When data protection software is compiled with VDDK 6.7 libraries, NBD mode backup and restore is significantly slower than before on vSphere 6.5 or 6.0. This regression was caused by dropping the OPEN_BUFFERED flag when it became the default in VDDK 6.7. The OPEN_BUFFERED flag will be reintroduced in VDDK 6.7.1 for backward compatibility. When performance is important, VMware recommends use of NBD Asynchronous I/O, calling VixDiskLib_WriteAsync and VixDiskLib_Wait.

• Snapshot consolidation may fail after SAN backup.

  If the ddb.adapterType in the VMDK descriptor file is invalid, for example “legacyESX” adapter type, VDDK fails to consolidate snapshots after SAN mode backup. The workaround (or fix) is to change ddb.adapterType to a valid value, such as lsilogic, ide, or buslogic. This issue has existed for a long time, and will be fixed in an upcoming release.

• Quiesce snapshot fails with error message on Windows Server 1709.

  When doing an application-quiesced snapshot on Windows Server 1709, taking and saving the snapshot may fail with this error message: “An error occurred while quiescing the virtual machine. See the virtual machine's event log for details.” This is because during snapshot creation, the automated system recovery (ASR) writer is not in correct state due to a VSS torn component, causing VSS quiescing to fail. The root cause is a Windows Server 1709 design change. The EFI system partition (ESP) is no longer hidden, so mount manager assigns it a volume GUID and the ESP gets included in the VSS snapshot, leading to a torn component. The workaround is to ignore the torn component and proceed with the VSS process. You can do this with the following Registry setting:

  HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Settings
  Key = SnapshotTornComponents (DWORD (32-bit))
  Value = 1 

• HotAdd proxy failure with Windows Server backups.

  If there is SATA controller in the Windows backup proxy, HotAdd mode might not work. The cause is that VDDK does not rescan SATA controllers after HotAdding, so if there exist multiple SATA controllers or ACHI controllers, VDDK might use the wrong controller ID and fail to find the HotAdded disk. Disk open fails, resulting in “HotAdd ManagerLoop caught an exception” and “Error 13 (You do not have access rights to this file)” errors. The workaround in this case is to remove the SATA controller from the Windows backup proxy. A fix has been found and will be provided in a future release. See KB 2151091.