vSphere Storage I/O Control allows cluster-wide storage I/O prioritization, which allows better workload consolidation and helps reduce extra costs associated with over provisioning.

Storage I/O Control extends the constructs of shares and limits to handle storage I/O resources. You can control the amount of storage I/O that is allocated to virtual machines during periods of I/O congestion, which ensures that more important virtual machines get preference over less important virtual machines for I/O resource allocation.

When you enable Storage I/O Control on a datastore, ESXi begins to monitor the device latency that hosts observe when communicating with that datastore. When device latency exceeds a threshold, the datastore is considered to be congested and each virtual machine that accesses that datastore is allocated I/O resources in proportion to their shares. You set shares per virtual machine. You can adjust the number for each based on need.

The I/O filter framework (VAIO) allows VMware and its partners to develop filters that intercept I/O for each VMDK and provides the desired functionality at the VMDK granularity. VAIO works along Storage Policy-Based Management (SPBM) which allows you to set the filter preferences through a storage policy that is attached to VMDKs.

Configuring Storage I/O Control is a two-step process:

  1. Enable Storage I/O Control for the datastore.
  2. Set the number of storage I/O shares and upper limit of I/O operations per second (IOPS) allowed for each virtual machine.

By default, all virtual machine shares are set to Normal (1000) with unlimited IOPS.

Note: Storage I/O Control is enabled by default on Storage DRS-enabled datastore clusters.
Note: vSphere 8.0 U1 supports seamless hot-add and hot-remove of a LWD I/O filter during VM reconfiguration.
Note: In this chapter, "Memory" refers to physical RAM.

Virtual Machine Storage Policies

Virtual machine storage policies are essential to virtual machine provisioning. The policies control which type of storage is provided for the virtual machine, how the virtual machine is placed within the storage, and which data services are offered for the virtual machine.

vSphere includes default storage policies. However, you can define and assign new policies.

You use the VM Storage Policies interface to create a storage policy. When you define the policy, you specify various storage requirements for applications that run on virtual machines. You can also use storage policies to request specific data services, such as caching or replication, for virtual disks.

You apply the storage policy when you create, clone, or migrate the virtual machine. After you apply the storage policy, the Storage Policy Based Management (SPBM) mechanism places the virtual machine in a matching datastore and, in certain storage environments, determines how the virtual machine storage objects are provisioned and allocated within the storage resource to guarantee the required level of service. The SPBM also enables requested data services for the virtual machine. vCenter Server monitors policy compliance and sends an alert if the virtual machine is in breach of the assigned storage policy.

See vSphere Storage for more information.

I/O Filters

I/O filters that are associated with virtual disks gain direct access to the virtual machine I/O path regardless of the underlying storage topology.

VMware offers certain categories of I/O filters. In addition, the I/O filters can be created by third-party vendors. Typically, they are distributed as packages that provide an installer to deploy filter components on vCenter Server and ESXi host clusters.

When I/O filters are deployed on the ESXi cluster, vCenter Server automatically configures and registers an I/O filter storage provider, also called a VASA provider, for each host in the cluster. The storage providers communicate with vCenter Server and make data services offered by the I/O filter visible in the VM Storage Policies interface. You can reference these data services when defining common rules for a VM policy. After you associate virtual disks with this policy, the I/O filters are enabled on the virtual disks.

See vSphere Storage for more information.

Storage I/O Control Requirements

Storage I/O Control has several requirements and limitations.

  • Datastores that are Storage I/O Control-enabled must be managed by a single vCenter Server system.
  • Storage I/O Control is supported on Fibre Channel-connected, iSCSI-connected, and NFS-connected storage. Raw Device Mapping (RDM) is not supported.
  • Storage I/O Control does not support datastores with multiple extents.
  • Before using Storage I/O Control on datastores that are backed by arrays with automated storage tiering capabilities, check the VMware Storage/SAN Compatibility Guide to verify whether your automated tiered storage array has been certified to be compatible with Storage I/O Control.

    Automated storage tiering is the ability of an array (or group of arrays) to migrate LUNs/volumes or parts of LUNs/volumes to different types of storage media (SSD, FC, SAS, SATA) based on user-set policies and current I/O patterns. No special certification is required for arrays that do not have these automatic migration/tiering features, including those that provide the ability to manually migrate data between different types of storage media.

Storage I/O Control Resource Shares and Limits

You allocate the number of storage I/O shares and upper limit of I/O operations per second (IOPS) allowed for each virtual machine. When storage I/O congestion is detected for a datastore, the I/O workloads of the virtual machines accessing that datastore are adjusted according to the proportion of virtual machine shares each virtual machine has.

Storage I/O shares are similar to shares used for memory and CPU resource allocation, which are described in Resource Allocation Shares. These shares represent the relative importance of a virtual machine regarding the distribution of storage I/O resources. Under resource contention, virtual machines with higher share values have greater access to the storage array. When you allocate storage I/O resources, you can limit the IOPS allowed for a virtual machine. By default, IOPS are unlimited.

The benefits and drawbacks of setting resource limits are described in Resource Allocation Limit. If the limit you want to set for a virtual machine is in terms of MB per second instead of IOPS, you can convert MB per second into IOPS based on the typical I/O size for that virtual machine. For example, to restrict a back up application with 64 KB IOs to 10 MB per second, set a limit of 160 IOPS.

View Storage I/O Control Shares and Limits

You can view the shares and limits for all virtual machines running on a datastore. Viewing this information allows you to compare the settings of all virtual machines that are accessing the datastore, regardless of the cluster in which they are running.

Procedure

  1. Browse to the datastore in the vSphere Client.
  2. Click the VMs tab.
    The tab displays each virtual machine running on the datastore and the associated shares value, and percentage of datastore shares.

Monitor Storage I/O Control Shares

Use the datastore Performance tab to monitor how Storage I/O Control handles the I/O workloads of the virtual machines accessing a datastore based on their shares.

Datastore performance charts allow you to monitor the following information:
  • Average latency and aggregated IOPS on the datastore
  • Latency among hosts
  • Queue depth among hosts
  • Read/write IOPS among hosts
  • Read/write latency among virtual machine disks
  • Read/write IOPS among virtual machine disks

Procedure

  1. Browse to the datastore in the vSphere Client.
  2. Under the Monitor tab, click Performance.
  3. Select Advanced.

Set Storage I/O Control Resource Shares and Limits

Allocate storage I/O resources to virtual machines based on importance by assigning a relative amount of shares to the virtual machine.

Unless virtual machine workloads are very similar, shares do not necessarily dictate allocation in terms of I/O operations or megabytes per second. Higher shares allow a virtual machine to keep more concurrent I/O operations pending at the storage device or datastore compared to a virtual machine with lower shares. Two virtual machines might experience different throughput based on their workloads.

Prerequisites

See vSphere Storage for information on creating VM storage policies and defining common rules for VM storage policies.

Procedure

  1. Browse to the virtual machine in the vSphere Client.
    1. To find a virtual machine, select a data center, folder, cluster, resource pool, or host.
    2. Click the VMs tab.
  2. Right-click the virtual machine and click Edit Settings.
  3. Click the Virtual Hardware tab and select a virtual hard disk from the list. Expand Hard disk.
  4. Select a VM storage policy from the drop-down menu.
    If you select a storage policy, do not manually configure Shares and Limit - IOPS.
  5. Under Shares, click the drop-down menu and select the relative amount of shares to allocate to the virtual machine (Low, Normal, or High).
    You can select Custom to enter a user-defined shares value.
  6. Under Limit - IOPS, click the drop-down menu and enter the upper limit of storage resources to allocate to the virtual machine.
    IOPS are the number of I/O operations per second. By default, IOPS are unlimited. You select Low (500), Normal (1000), or High (2000), or you can select Custom to enter a user-defined number of shares.
  7. Click OK.

Enable Storage I/O Control

When you enable Storage I/O Control, ESXi monitors datastore latency and throttles the I/O load if the datastore average latency exceeds the threshold.

Procedure

  1. Browse to the datastore in the vSphere Client.
  2. Click the Configure tab.
  3. Click Settings and click General.
  4. Click Edit for Datastore Capabilities.
  5. Select the Enable Storage I/O Control check box.
  6. Click OK.

Results

Under Datastore Capabilities, Storage I/O Control is enabled for the datastore.

Set Storage I/O Control Threshold Value

The congestion threshold value for a datastore is the upper limit of latency that is allowed for a datastore before Storage I/O Control begins to assign importance to the virtual machine workloads according to their shares.

You do not need to adjust the threshold setting in most environments.
Caution: Storage I/O Control might not function correctly if you share the same spindles on two different datastores.
If you change the congestion threshold setting, set the value based on the following considerations.
  • A higher value typically results in higher aggregate throughput and weaker isolation. Throttling will not occur unless the overall average latency is higher than the threshold.
  • If throughput is more critical than latency, do not set the value too low. For example, for Fibre Channel disks, a value below 20ms could lower peak disk throughput. A very high value (above 50ms) might allow very high latency without any significant gain in overall throughput.
  • A lower value will result in lower device latency and stronger virtual machine I/O performance isolation. Stronger isolation means that the shares controls are enforced more often. Lower device latency translates into lower I/O latency for the virtual machines with the highest shares, at the cost of higher I/O latency experienced by the virtual machines with fewer shares.
  • A very low value (lower than 20ms) will result in lower device latency and isolation among I/Os at the potential cost of a decrease in aggregate datastore throughput.
  • Setting the value extremely high or extremely lowly results in poor isolation.

Prerequisites

Verify that Storage I/O Control is enabled.

Procedure

  1. Browse to the datastore in the vSphere Client.
  2. Click the Configure tab and click Settings.
  3. Click General.
  4. Click Edit for Datastore Capabilities.
  5. Select the Enable Storage I/O Control check box.
    Storage I/O Control automatically sets the latency threshold that corresponds to the estimated latency when the datastore is operating at 90% of its peak throughput.
  6. (Optional) Adjust the Congestion Threshold.
    • Select a value from the Percentage of peak throughput drop-down menu.
    The percentage of peak throughput value indicates the estimated latency threshold when the datastore is using that percentage of its estimated peak throughput.
    • Select a value from the Manual drop-down menu.
    The value must be between 5ms and 100ms. Setting improper congestion threshold values can be detrimental to the performance of the virtual machines on the datastore.
  7. (Optional) Click Reset to defaults to restore the congestion threshold setting to the default value (30ms).
  8. Click OK.

Storage DRS Integration with Storage Profiles

Storage Policy Based Management (SPBM) allows you to specify the policy for a virtual machine which is enforced by Storage DRS. A datastore cluster can have set of datastores with different capability profiles. If the virtual machines have storage profiles associated with them, Storage DRS can enforce placement based on underlying datastore capabilities.

As part of Storage DRS integration with storage profiles, the Storage DRS cluster level advanced option EnforceStorageProfiles is introduced. Advanced option EnforceStorageProfiles takes one of these integer values: 0,1 or 2. The default value is 0. When the option is set to 0, it indicates that there is no storage profile or policy enforcement on the Storage DRS cluster. When the option is set to 1, it indicates that there is a storage profile or policy soft enforcement on the Storage DRS cluster. This is analogous with DRS soft rules. Storage DRS will comply with storage profile or policy in the optimum level. Storage DRS will violate the storage profile compliant if it is required to do so. Storage DRS affinity rules will have higher precedence over storage profiles only when storage profile enforcement is set to 1. When the option is set to 2, it indicates that there is a storage profile or policy hard enforcement on the Storage DRS cluster. This is analogous with DRS hard rules. Storage DRS will not violate the storage profile or policy compliant. Storage profiles will have higher precedence over affinity rules. Storage DRS will generate fault: could not fix anti-affinity rule violation

Prerequisites

By default, Storage DRS will not enforce storage policies associated with a virtual machine. Please configure EnforceStorageProfiles option according to your requirements. The options are Default (0), Soft (1) or Hard (2).

Procedure

  1. Log in to the vSphere Client as an Administrator.
  2. In the vSphere Client, click on the Storage DRS cluster, then select Manage > Settings > Storage DRS.
  3. Click Edit > Advanced Options.
  4. Click in the area under the Option heading and type EnforceStorageProfiles
  5. Click in the area under the Value heading to the right of the previously entered advanced option name and type the value of either 0, 1 or 2.
  6. Select Add.
  7. Click OK.