Plan capacity and cache based on the expected data storage consumption. Consider your requirements for availability and endurance.

Planning Capacity in vSAN

You can calculate the capacity of a vSAN datastore to accommodate the virtual machines (VMs) files in the cluster, and to handle failures and maintenance operations.

Raw Capacity

Use this formula to determine the raw capacity of a vSAN datastore. Multiply the total number of disk groups in the cluster by the size of the capacity devices in those disk groups. Subtract the overhead required by the vSAN on-disk format.

Failures to Tolerate

When you plan the capacity of the vSAN datastore, not including the number of virtual machines and the size of their VMDK files, you must consider the Failures to tolerate of the virtual machine storage policies for the cluster.

The Failures to tolerate has an important role when you plan and size storage capacity for vSAN. Based on the availability requirements of a virtual machine, the setting might result in doubled consumption or more, compared with the consumption of a virtual machine and its individual devices.

For example, if the Failures to tolerate is set to 1 failure - RAID-1 (Mirroring), virtual machines can use about 50 percent of the raw capacity. If the FTT is set to 2, the usable capacity is about 33 percent. If the FTT is set to 3, the usable capacity is about 25 percent.

But if the Failures to tolerate is set to 1 failure - RAID-5 (Erasure Coding), virtual machines can use about 75 percent of the raw capacity. If the FTT is set to 2 failures - RAID-6 (Erasure Coding), the usable capacity is about 67 percent. For more information about RAID 5/6, see Administering VMware vSAN.

For information about the attributes in a vSAN storage policy, see Administering VMware vSAN.

Capacity Sizing Guidelines

  • Keep some unused space to prevent vSAN from rebalancing the storage load. vSAN rebalances the components across the cluster whenever the consumption on a single capacity device reaches 80 percent or more. The rebalance operation might impact the performance of applications. To avoid these issues, keep storage consumption to less than 70 percent. vSAN 7.0 Update 1 and later enables you to manage unused capacity using operations reserve and host rebuild reserve.
  • Plan extra capacity to handle any potential failure or replacement of capacity devices, disk groups, and hosts. When a capacity device is not reachable, vSAN recovers the components from another device in the cluster. When a flash cache device fails or is removed, vSAN recovers the components from the entire disk group.
  • Reserve extra capacity to make sure that vSAN recovers components after a host failure or when a host enters maintenance mode. For example, provision hosts with enough capacity so that you have sufficient free capacity left for components to rebuild after a host failure or during maintenance. This extra space is important when you have more than three hosts, so you have sufficient free capacity to rebuild the failed components. If a host fails, the rebuilding takes place on the storage available on another host, so that another failure can be tolerated. However, in a three-host cluster, vSAN does not perform the rebuild operation if the Failures to tolerate is set to 1 because when one host fails, only two hosts remain in the cluster. To tolerate a rebuild after a failure, you must have at least three surviving hosts.

  • Provide enough temporary storage space for changes in the vSAN VM storage policy. When you dynamically change a VM storage policy, vSAN might create a new RAID tree layout of the object. When vSAN instantiates and synchronizes a new layout, the object may consume extra space temporarily. Keep some temporary storage space in the cluster to handle such changes.

  • If you plan to use advanced features, such as software checksum or deduplication and compression, reserve extra capacity to handle the operational overhead.
  • You can use the vSAN Sizer tool https://vsansizer.esp.vmware.com/ to assist with capacity requirements, and to determine how vSAN can your meet performance requirements.

Considerations for Virtual Machine Objects

When you plan the storage capacity in the vSAN datastore, consider the space required in the datastore for the VM home namespace objects, snapshots, and swap files.

  • VM Home Namespace. You can assign a storage policy specifically to the home namespace object for a virtual machine. To prevent unnecessary allocation of capacity and cache storage, vSAN applies only the Failures to tolerate and the Force provisioning settings from the policy on the VM home namespace. Plan storage space to meet the requirements for a storage policy assigned to a VM Home Namespace whose Failures to tolerate is greater than 0.
  • Snapshots. Delta devices inherit the policy of the base VMDK file. Plan extra space according to the expected size and number of snapshots, and to the settings in the vSAN storage policies.

    The space that is required might be different. Its size depends on how often the virtual machine changes data and how long a snapshot is attached to the virtual machine.

  • Swap files. In vSAN 6.7 and later, virtual machine swap files inherit the storage policy of the VM Namespace.

Design Considerations for Flash Caching Devices in vSAN

Plan the configuration of flash devices for vSAN cache and all-flash capacity to provide high performance and required storage space, and to accommodate future growth.

Choosing Between PCIe or SSD Flash Devices

Choose SSD flash devices according to the requirements for performance, capacity, write endurance, and cost of the vSAN storage.

  • Compatibility. The model of the SSD devices must be listed in the vSAN section of the VMware Compatibility Guide.
  • Performance. PCIe devices generally have faster performance than SATA devices.
  • Capacity. The maximum capacity that is available for PCIe devices is generally greater than the maximum capacity that is currently listed for SATA devices for vSAN in the VMware Compatibility Guide.
  • Write endurance. The write endurance of the SSD devices must meet the requirements for capacity or for cache in all-flash configurations, and for cache in hybrid configurations.

    For information about the write endurance requirements for all-flash and hybrid configurations, see the VMware vSAN Design and Sizing Guide. For information about the write endurance class of SSD devices, see the vSAN section of the VMware Compatibility Guide.

  • Cost. PCIe devices generally have higher cost than SSD devices.

Flash Devices as vSAN Cache

Design the configuration of flash cache for vSAN for write endurance, performance, and potential growth based on these considerations.

Table 1. Sizing vSAN Cache
Storage Configuration Considerations
All-flash and hybrid configurations
  • A higher cache-to-capacity ratio eases future capacity growth. Oversizing cache enables you to add more capacity to an existing disk group without the need to increase the size of the cache.
  • Flash caching devices must have high write endurance.
  • Replacing a flash caching device is more complicated than replacing a capacity device because such an operation impacts the entire disk group.
  • If you add more flash devices to increase the size of the cache, you must create more disk groups. The ratio between flash cache devices and disk groups is always 1:1.

    A configuration of multiple disk groups provides the following advantages:

    • Reduced risk of failure. If a single caching device fails, fewer capacity devices are affected.
    • Potentially improved performance if you deploy multiple disk groups that contain smaller flash caching devices.

    However, when you configure multiple disk groups, the memory consumption of the hosts increases.

All-flash configurations

In all-flash configurations, vSAN uses the cache layer for write caching only. The write cache must be able to handle high write activities. This approach extends the life of capacity flash that might be less expensive and might have lower write endurance.

Hybrid configurations

The flash caching device must provide at least 10 percent of the anticipated storage that virtual machines are expected to consume, not including replicas such as mirrors. The Primary level of failures to tolerate attribute from the VM storage policy does not impact the size of the cache.

If the read cache reservation is configured in the active VM storage policy, the hosts in the vSAN cluster must have sufficient cache to satisfy the reservation during a post-failure rebuild or maintenance operation.

If the available read cache is not sufficient to satisfy the reservation, the rebuild or maintenance operation fails. Use read cache reservation only if you must meet a specific, known performance requirement for a particular workload.

The use of snapshots consumes cache resources. If you plan to use several snapshots, consider dedicating more cache than the conventional 10 percent cache-to-consumed-capacity ratio.

Design Considerations for Flash Capacity Devices in vSAN

Plan the configuration of flash capacity devices for vSAN all-flash configurations to provide high performance and required storage space, and to accommodate future growth.

Choosing Between PCIe or SSD Flash Devices

Choose SSD flash devices according to the requirements for performance, capacity, write endurance, and cost of the vSAN storage.

  • Compatibility. The model of the SSD devices must be listed in the vSAN section of the VMware Compatibility Guide.
  • Performance. PCIe devices generally have faster performance than SATA devices.
  • Capacity. The maximum capacity that is available for PCIe devices is generally greater than the maximum capacity that is currently listed for SATA devices for vSAN in the VMware Compatibility Guide.
  • Write endurance. The write endurance of the SSD devices must meet the requirements for capacity or for cache in all-flash configurations, and for cache in hybrid configurations.

    For information about the write endurance requirements for all-flash and hybrid configurations, see the VMware vSAN Design and Sizing Guide. For information about the write endurance class of SSD devices, see the vSAN section of the VMware Compatibility Guide.

  • Cost. PCIe devices generally have higher cost than SSD devices.

Flash Devices as vSAN Capacity

In all-flash configurations, vSAN does not use cache for read operations and does not apply the read-cache reservation setting from the VM storage policy. For cache, you can use a small amount of more expensive flash that has high write endurance. For capacity, you can use flash that is less expensive and has lower write endurance.

Plan a configuration of flash capacity devices by following these guidelines:

  • For better performance of vSAN, use more disk groups of smaller flash capacity devices.
  • For balanced performance and predictable behavior, use the same type and model of flash capacity devices.

Design Considerations for Magnetic Disks in vSAN

Plan the size and number of magnetic disks for capacity in hybrid configurations by following the requirements for storage space and performance.

SAS and NL-SAS Magnetic Devices

Use SAS or NL-SAS magnetic devices by following the requirements for performance, capacity, and cost of the vSAN storage.

  • Compatibility. The model of the magnetic disk must be certified and listed in the vSAN section of the VMware Compatibility Guide.
  • Performance. SAS and NL-SAS devices have faster performance.
  • Capacity. The capacity of SAS or NL-SAS magnetic disks for vSAN is available in the vSAN section of the VMware Compatibility Guide. Consider using a larger number of smaller devices instead of a smaller number of larger devices.
  • Cost. SAS and NL-SAS devices can be expensive.

Magnetic Disks as vSAN Capacity

Plan a magnetic disk configuration by following these guidelines:

  • For better performance of vSAN, use many magnetic disks that have smaller capacity.

    You must have enough magnetic disks that provide adequate aggregated performance for transferring data between cache and capacity. Using more small devices provides better performance than using fewer large devices. Using multiple magnetic disk spindles can speed up the destaging process.

    In environments that contain many virtual machines, the number of magnetic disks is also important for read operations when data is not available in the read cache and vSAN reads it from the magnetic disk. In environments that contain a small number of virtual machines, the disk number impacts read operations if the Number of disk stripes per object in the active VM storage policy is greater than one.

  • For balanced performance and predictable behavior, use the same type and model of magnetic disks in a vSAN datastore.
  • Dedicate a high enough number of magnetic disks to satisfy the value of the Failures to tolerate and the Number of disk stripes per object attributes in the defined storage policies. For information about the VM storage policies for vSAN, see Administering VMware vSAN.

Design Considerations for Storage Controllers in vSAN

Use storage controllers on the hosts of a vSAN cluster that best satisfy your requirements for performance and availability.

  • Use storage controller models, and driver and firmware versions that are listed in the VMware Compatibility Guide. Search for vSAN in the VMware Compatibility Guide.
  • Use multiple storage controllers, if possible, to improve performance and to isolate a potential controller failure to only a subset of disk groups.
  • Use storage controllers that have the highest queue depths in the VMware Compatibility Guide. Using controllers with high queue depth improves performance. For example, when vSAN is rebuilding components after a failure or when a host enters maintenance mode.
  • Use storage controllers in passthrough mode for best performance of vSAN. Storage controllers in RAID 0 mode require higher configuration and maintenance efforts compared to storage controllers in passthrough mode.
  • Deactivate caching on the controller, or set caching to 100 percent Read.