The Physical Storage Design uses vSAN to implement software-defined storage as the primary storage type. Different domains or workload clusters can implement other storage options depending on the requirements and constraints.

vSAN is a hyper-converged storage solution built within the ESXi hypervisor. vSAN (Original Storage Architecture) creates disk groups consisting of hard disk drives and flash devices or all-flash devices in the local ESXi host. It provides a highly resilient shared storage datastore to the vSphere Cluster.

By using vSAN storage policies, you can control capacity, performance, and availability on a per virtual disk basis.

While vSAN is the recommended solution, the physical storage can be based on the extensive list of supported storage providers. This reference architecture focuses on vSAN requirements from a physical storage perspective.

Caution:

When using a storage platform, evaluate the supported use cases across the telco cloud, including file and block storage and cloud-native support considerations such as Read-Write Many persistent volumes.

Table 1. vSAN Infrastructure Requirements

Category

Requirements

Number of ESXi hosts

  • Minimum of 3 ESXi hosts providing storage resources to the vSAN cluster. This can be 3 ESXi hosts or 2 ESXi hosts and 1 vSAN witness.

  • Minimum of 4 ESXi hosts for vSAN automated rebuild.

  • Maximum 200 VMs per host in a vSAN cluster.

vSAN configuration

vSAN can be configured in all-flash or hybrid mode:

  • All-flash mode: All-flash vSAN configuration requires flash devices for both the caching and capacity tiers.

  • Hybrid mode: vSAN hybrid storage configuration requires both magnetic devices and flash caching devices.

Individual ESXi hosts that provide storage resources.

  • Minimum of one flash device. The flash cache tier must be at least 10% of the size of the capacity tier.

  • Minimum of two HDDs for hybrid mode, or an additional flash device for an all-flash configuration.

  • RAID controller that is compatible with vSAN.

  • Minimum 10 Gbps network for vSAN traffic or 25 Gbps when using the vSAN ESA architecture.

  • Host isolation response of vSphere High Availability is set to power off VMs.

vSAN Disk Groups

Disk group sizing is an important factor in the vSAN design. If more ESXi hosts are available in a cluster, more failures are tolerated in the cluster. This capability adds cost because additional hardware is required for the disk groups.

More available disk groups can increase the recoverability of vSAN during a failure. When deciding on the number of disk groups per ESXi host, consider these points:

  • Amount of available space on the vSAN datastore

  • Number of failures that can be tolerated in the cluster

The optimal number of disk groups is a balance between the hardware and space requirements for the vSAN datastore. More disk groups increase space and provide high availability, but it can be expensive.

Note:

The disk groups concept is not applicable for the vSAN Express Storage Architecture.

Storage Controllers

The storage I/O controllers are as important as the selection of disk drives to a vSAN configuration. vSAN supports SAS, SATA, and SCSI adapters in either pass-through or RAID 0 mode. vSAN supports multiple controllers per ESXi host.

  • Multi-Controller Configuration: Multiple controllers can improve performance and mitigate a controller or SSD failure to a smaller number of drives or vSAN disk groups.

  • Single-Controller Configuration: With a single controller, all disks are controlled by one device. A controller failure impacts all storage, including the boot media (if configured).

Controller queue depth is an important aspect of performance. All I/O controllers in the VMware vSAN Hardware Compatibility Guide have a minimum queue depth of 256. If you increase the queue depth to a value higher than 256, ensure that you consider the regular day-to-day operations in your environment. Examples of events that require higher queue depth are as follows:

  • VM deployment operations

  • Re-sync I/O activity because of automatic or manual fault remediation

Physical Storage Recommendations

Design Recommendation

Design Justification

Design Implication

Use all-flash (SSD / NVMe) vSAN in all vSphere clusters.

Provides the best performance with low latency.

When using all-flash vSAN, you can enable de-duplication and compression that saves space on the datastores.

Flash storage costs more than traditional magnetic disks.

For the management cluster, provide a vSAN configuration with at least 6 TB of usable space.

Provides all the required space for this solution while allowing the deployment of additional monitoring and management components in the management cluster.

More space is required on day 1.

For the edge cluster, provide a vSAN configuration with at least 500 GB of usable space.

Provides required storage to run NSX Edge Nodes and Avi Load Balancer Service Engines.

None

For the compute clusters, size the vSAN datastore according to the current workloads plus five years of expected growth.

Ensures that the storage solution is not required to be upgraded as it can cause downtime to workloads.

More space is required on day 1.

When using vSAN, validate the ESXI release before using HCI Mesh or vSAN File Services.

HCI Mesh and vSAN FS is supported with vSAN using ESA architecture starting from vSphere 8.0U2.