times

Block VAAI (vStorage API for Array Integration) | ESXi builds ≥ 21495797
For vSphere 8.0 Update 1 | Last document update 26 July 2023
When information changes, additions and updates are marked New.

VAAI Implementation of Extended XCOPY

ESXi 8.0 Update 1 and later supports the Extended XCOPY feature to optimize data copies between VMFS datastores that are in different storage arrays. XCOPY operations across storage arrays are supported only for VMFS-6 datastores. ESXi enables this feature, allowing you to offload, migrate and clone workloads. The actual data migration across the arrays must be implemented by the storage arrays.

VAAI (vStorage API for Array Integration) is a VMware technology to help partners integrate storage hardware. XCOPY is one of the Block VAAI primitives for offloading tasks to storage arrays. Storage systems can use XCOPY to perform array-based operations such as migration or cloning of virtual machines, instead of consuming vSphere resources for these tasks. See Introduction to VAAI for details.

XCOPY has been supported for datastores that are part of the same array, that is, both source and destination data stores reside in one storage array. Because of this restriction, XCOPY offload did not work when the source and destination addresses were in two different arrays. As of vSphere 8.0 U1, this works if implemented.

The above limitation is eliminated by Extended XCOPY (E-XCOPY) support. E-XCOPY is useful for Storage vMotion and clone workloads that require copying large amounts of data between two datastores hosted by two different storage arrays. The performance of Storage vMotion and clone workloads is expected to improve significantly when using E-XCOPY compared to traditional software-based data movement.

Support Required on the ESXi Host

On the ESXi host, release 8.0 U1 added support to enable E-XCOPY when issuing I/O across different storage arrays. In case of any failures reported by the array for E-XCOPY operation, ESXi detects and falls back to traditional software-based data copy to complete the operation.

Support Required on the Storage Array

Vendors must implement the following items to support E-XCOPY:

Storage array must already support and be certified for same-array XCOPY.
Because XCOPY requires a T10 plug-in, the storage array must support T10 plug-ins.
Storage array must provide a way to advertise E-XCOPY support by returning an additional descriptor in SCSI_CMD_RECEIVE_COPY_RESULTS (0x84) with SCSI_SERVICE_ACTION_OPERATING_PARAMS (0x03) command output. The array must return a VERIFY (0x7) descriptor, in addition to what it was returning before.
ESXi already issues 0x84 and 0x30 commands as a part of its device inventory. Going forward ESXi should look for the additional VERIFY (0x7) descriptor in this response. If this VERIFY descriptor is present, ESXi recognizes that this array (or this “destination” volume) is capable of supporting E-XCOPY.
As E-XCOPY is implemented, the XCOPY command is sent to the destination device only. The data migration implementation from storage array must take care of migrating data from the destination device to the source device.
In addition, administrators of ESXi hosts will need to configure a new set of claim-rules for this XCOPY across arrays to take effect. This claim-rule in conjunction with array-conveying support via 0x84/0x03 is essential. If this claim-rule is not present, XCOPY across arrays will continue to be software-based host copy. This gives host administrators an override option, if needed, to disable data-copy offload on ESXi hosts in certain situations, even when storage arrays support this feature.
Storage arrays must implement this feature to move data across arrays. Implementation can be array specific.
As a partner, after you implement support, please reach out to VMware for certification of your E-XCOPY feature.

Administrator Configuration of Claim-Rules

To optimize data copy between datastores on different arrays, and allow migrate and clone offloads using E-XCOPY, the administrator must first configure storage claim-rules. Extended XCOPY is supported only with the T10 plug-in, specified after -P option, and with claim-rule class VAAI, specified after the -c option. Auto assign means the system chooses the best known rule ID.

esxcli storage core claimrule add -t vendor -V vendorName -M modelName -P VMW_VAAIP_T10 -c VAAI –autoassign -e

For example, vendorName could be ExampleCom, and modelName could be SuperStore. Once the claim-rule has been established, you can check status of Extended XCOPY with this command:

esxcli storage core device vaai status get -d device

For example, with device naa.68ccf098007f554c61148eb1786e6749, the following appears:

naa.68ccf098007f554c61148eb1786e6749 
   VAAI Plugin Name: VMW_VAAIP_T10 
   ATS Status: supported 
   Clone Status: supported 
   Zero Status: supported 
   Delete Status: supported 
   Ex Clone Status: supported

Administrators should verify that the VAAI_FILTER plug-in device is added, and if not, use the following commands to autoassign, load, and add the T10 plug-in:

esxcli storage core claimrule list -c Filter 
esxcli storage core claimrule add -t vendor -V vendorName -M modelName -P VAAI_FILTER -c Filter --autoassign 
esxcli storage core claimrule load -c Filter 
esxcli storage core claimrule add -t vendor -V vendorName -M modelName -P VMW_VAAIP_T10 -c VAAI --autoassign -e

The claim-rule in the step above uses ESXi's default transfer size and single segment for XCOPY commands. For possibly more efficient values of transfer size and multiple segments, try option -a to use array-reported values for XCOPY commands, and option -s use multiple segments for XCOPY commands. Also available are -m -k options; see below. Do this for both source and destination devices. Source device:

esxcli storage core claimrule add -t vendor -V vendorName -M modelName -P VMW_VAAIP_T10 -c VAAI --autoassign -a -s

Destination device:

esxcli storage core claimrule add -t vendor -V vendorName -M modelName -P VMW_VAAIP_T10 -c VAAI --autoassign -a -s -e

Finally, load the VAAI_T10 plug-in and run the newly added claim rule options.

esxcli storage core claimrule load -c VAAI 
esxcli storage core claimrule run --claimrule-class=Filter

Explanation of Storage Core Options

The -e option is new. It enables Extended XCOPY for the claimrule add commands above.

Option -t specifies the type of matching for a claim-rule, in this case type vendor as specified by the -V option. Option -P specifies plug-in, -c specifies claim-rule class, and -d specifies device.

For claimrule add, the -m option specifies maximum transfer size in MB, and the -k option specifies maximum transfer size in KB. This might perform faster than -a for array-reported values.