This design uses VMware vSAN to implement software-defined storage for the consolidated cluster. By using vSAN, you have a high level of control on the storage subsystem.

vSAN is a hyper-converged storage software that is fully integrated with the hypervisor. vSAN creates a cluster of server hard disk drives and solid state drives, and presents a flash-optimized, highly-resilient, shared storage datastore to ESXi hosts and virtual machines. By using vSAN storage policies, you can control capacity, performance, and availability on a per virtual machine basis.

Requirements and Dependencies

The software-defined storage module has the following requirements and options.

Requirement Category

Requirements

Number of hosts

  • Minimum of 3 ESXi hosts providing storage resources to the vSAN cluster.

vSAN configuration

vSAN is configured as hybrid storage or all-flash storage. 

  • A vSAN hybrid storage configuration requires both magnetic devices and flash caching devices.

  • An all-flash vSAN configuration requires flash devices for both the caching and capacity tiers.  

Requirements for individual hosts that provide storage resources

  • Minimum of one SSD. The SSD flash cache tier should be at least 10% of the size of the HDD capacity tier.

  • Minimum of two HDDs for hybrid, or two additional flash devices for an all-flash configuration

  • RAID controller that is compatible with vSAN. 

  • 10 Gbps network for vSAN traffic.

  • vSphere High Availability host isolation response set to power off virtual machines. With this setting, you prevent split-brain conditions if isolation or network partition occurs. In a split-brain condition, the virtual machine might be powered on by two ESXi hosts by accident.

    See design decision ROBO-VI-VC-007 for more details.

Hybrid Mode and All-Flash Mode

vSAN has two modes of operation: all-flash and hybrid.

Hybrid Mode

In a hybrid storage architecture, vSAN pools server-attached capacity devices (in this case magnetic devices) and caching devices, typically SSDs or PCI-e devices, to create a distributed shared datastore.

All-Flash Mode

All-flash storage uses flash-based devices (SSD or PCI-e) as a write cache while other flash-based devices provide high endurance for capacity and data persistence.

Table 1. Design Decisions on the vSAN Mode

Decision ID

Design Decision

Design Justification

Design Implication

ROBO-PHY-STO-001

Configure vSAN in hybrid mode.

Provides a lower entry point for vSAN. If necessary, you can use an all-flash configuration.

vSAN hybrid mode does not provide the potential performance or additional capabilities such as deduplication of an all-flash configuration.

Sizing Storage

You usually base sizing on the requirements of the IT organization. However, this design provides calculations that are based on a single-region implementation, and is then implemented on a per-region basis. In this way, you can handle storage in a dual-region deployment that has failover capabilities enabled.

This sizing is calculated according to a certain node configuration per region. Although VMware Validated Design has enough memory capacity to handle N-1 host failures, and uses thin-provisioned swap for the vSAN configuration, the potential think-provisioned swap capacity is factored in the calculation.

For sizing tenant virtual machines, this VMware Validated Design uses the following requirements per virtual machine:

  • 2 vCPUs

  • 4 GB of memory

  • Disk capacity of a singe 100 GB VMDK

Table 2. Sizes for Workloads and Hardware for Management and Tenant Components for Remote Office and Branch Office

Category

Quantity

Resource Type

Capacity Consumption

Physical Infrastructure (ESXi)

4

Memory

768 GB

Virtual Infrastructure

9

Disk

422 GB

Swap

48 GB

Operations Management

6

Disk

2,258 GB

Swap

58 GB

Cloud Management

3

Disk

170 GB

Swap

18 GB

Workload

25

Disk

2,500 GB

Swap

100 GB

Total

  • 22 management virtual machines

  • 25 workload virtual machines

  • 4 ESXi hosts

Disk

5,350 GB

Swap

224 GB

Memory

768 GB

Derive the storage space that is required for the capacity tier according to the following calculations For vSAN memory consumption by management ESXi hosts, see VMware Knowledge Base article 2113954.

[Static Base Consumption + (Number of Disk Groups * (Static Disk Group Base Consumption + (Static Flash Device Memory Overhead Per GB * Flash Device Capactiy))) + (Static Capacity Disk Base Consumption * Number of Capacity Disks) ] * Host Quantity = vSAN  Memory Consumption
[5426 MB + (2 Disk Groups * (636 MB + (8MB * 100 GB Flash Storage))) + (70 MB * 2 Magnetic Disks)] * 4 ESXi Hosts

[5426 MB + (2 Disk Groups * (636 MB + 800 MB)) + 140 MB] * 4 ESXi Hosts = [5426 MB + 3012 MB] * 4 ESXi Hosts * 10e-3 GB ≈ 34 GB vSAN Memory Consumption 

Derive the consumption of storage space by the management virtual machines according to the following calculations. See VMware vSAN Design and Sizing Guide.

VM Raw Storage Requirements (without FTT) + VM Swap (without FTT) = Virtual Machine Raw Capacity Requirements

Virtual Machine Raw Capacity Requirements * FTT = Final Virtual Machine Raw Capacity Requirements

5,350 GB Disk + 224 GB Swap = 5574 GB Virtual Machine Raw Capacity Requirements

5,574 GB Virtual Machine Raw Capacity Requirements * 2 (FTT=1, RAID1) = 11,148 GB Final Virtual Machine Raw Capacity Requirements

Derive the requirements for total storage space for the capacity tier according to the following calculations:

vSAN Memory Consumption  + Final Virtual Machine Raw Capacity Requirements = Total Raw Storage Capacity

Total Raw Storage Capacity * 30% Slack Overhead * 1% On-disk Format Overhead * 0.12% Checksum Overhead = Raw Unformatted Storage Capacity
OR
Total Raw Storage Capacity * 30% Slack Overhead * 1% On-disk Format Overhead * 0.12% Checksum Overhead * 20% Estimated Growth = Raw Unformatted Storage Capacity (with 20% Growth Capacity)

Raw Unformatted Storage Capacity / ESXi Quantity = Final Raw Storage Capacity per Host

Based on the calculations for the vSAN memory consumption and the management virtual machine consumption, calculate the final raw storage capacity required for the cluster and per the ESXi hosts.

34 GB + 11,148 GB = 11,182 GB Total Raw Storage Capacity

11,182 GB Total Raw Storage Capacity * 30% Slack Overhead * 1% On-disk Format Overhead * .12% Overhead ≈ 14,699 ≈ 15 TB Raw Unformatted Storage Capacity
15 TB Raw Unformatted Storage Capacity / 4 ESXi hosts ≈ 4 TB Final Raw Storage Capacity per host

Derive the storage space that is required for the caching tier according to the following calculation:

Raw Unformatted Storage Capacity  * 50% * 10% = Total Flash Device Capacity

Total Flash Device Capacity / ESXi Quantity = Final Flash Device Capacity per Host
15 TB Raw Unformatted Storage Capacity * 50% * 10% Cache Required ≈ 800 GB Flash Device Capacity
800 GB Flash Device Storage Capacity / 4 ESXi Hosts ≈ 200 GB of Flash Device Capacity per Host
Table 3. Design Decisions on the vSAN Disk Configuration

Decision ID

Design Decision

Design Justification

Design Implication

ROBO-PHY-STO-002

In the consolidated cluster, for each host:

  • For the caching tier, provide 200 GB or more of SSD storage.

  • For the capacity tier, provide 4 TB or more of traditional HDD storage.

Provides enough capacity for the management VMs and a starting point for workload VMs with a minimum of 10% flash-based caching and 30% of overhead.

When using only a single disk group, you limit the amount of striping (performance) capability and increase the size of the fault domain.

Scale disk space as necessary to accommodate workload VMs. Disk requirements can be higher according to the disk size of the workload.