vSphere supports vMotion of both vPMEMDisk and vPMEM. vPMEMDisk vMotion is conducted as XvMotion, where both local storage and memory contents are transferred to another host. vPMEM vMotion is conducted as a compute vMotion, where vPMEM is transferred as memory along with vRAM. Note that PMEM is a host local storage and seamless live-migration like vMotion is only possible in a vSphere environment (unlike bare-metal OS)

We used 2 identical hosts connected over 40 GbE network. We created 3 vmknics for vMotion over the 40 GbE physical NIC.

Oracle DB vMotion

In this section, we show vMotion performance with a VM running HammerDB against an Oracle database.
 

Highlights:

  • vMotion time is 30% lower in the vPMEM configuration compared to the NVMe SSD configuration

Configuration:

We used the VM detailed in Table 18 to do the vMotion between hosts. The client and server are in different VMs on different hosts connected via 10G link.

Results:

In both the PMEM cases, downtime is less than half of a second, as shown in Table 16.

 

Memory Transfer (GB)

Disk Transfer (GB)

Downtime (secs)

Total Duration (secs)

SAN

128

0

0.293

53.04

NVMe SSD

128

485

0.534

526.64

vPMEMDISK

128

485

0.284

631.89

vPMem

128 + 485

0

0.259

379.67

Table 16: vMotion HammerDB workload configuration

We see that vPMEM vMotion is 30% faster than that of the NVMe SSD configuration, even though the amount of data transferred is the same in both cases. As explained in FIO results, vPMEMDisk vMotion falls under the pure write scenario and, as a result, the vMotion duration is longer.

Figure 23 shows the normalized throughput reported by the below Oracle query while vMotion is occurring.

select value from v$sysstat where name='user commits';

This metric closely tracks the HammerDB throughput. We see that in all the cases the throughput is restored to normal after the vMotion is complete. The SAN performane in Figure 23 is roughly 60% of NVMe SSD. Although the performance is lower with SAN, it comes with the added benefit of vSphere High Availability (HA). All other configurations (local storage) do not have vSphere HA support. Interestingly, when vMotion is taking place, the throughput in the vPMem case is 13% better than the NVMe SSD case with no vMotion.

 

Figure 23: HammerDB vMotion performance

Redis vMotion

We used the Redis VM detailed in Table 17 for vMotion between the hosts. We added SAN configuration to get a baseline with compute vMotion. Table 13 gives the details of the Redis parameters used in this test.

DB Size

25M keys (120 GB in memory)

Data Size

4 KB

Driver

memtier_benchmark

Parameters

pipeline=4; clients=50; threads=4

Tests

50% SETs

Table 17: vMotion Redis workload configuration

Results:

Table 18 shows the amount of memory and disk transferred, along with VM downtime and total duration for vMotion. In the vPMEM case, both 128 GB of vRAM and 128 GB of vPMEM need to be transferred. In the vPMEM-aware case, vRAM is not touched by Redis and only a few gigabytes of vRAM need to be transferred along with the 128 GB of vPMEM. Although the amount of memory (RAM or PMEM) transferred is the same in both SAN and vPMEM-aware cases, the total vMotion duration is higher in the vPMEM-aware case. We believe this is because memory is touched/dirtied at a higher rate in the vPMEM-aware case and this results in more pre-copy-phase iterations of vMotion. Interested readers can learn more about vMotion details in vMotion Architecture, Best Practices, and Performance in vSphere 5.0.

 

Memory Transfer (GB)

Disk Transfer (GB)

Downtime (secs)

Total Duration (secs)

SAN

128

0

0.128

58.91

NVMe SSD

128

128

0.123

161.93

vPMEMDISK

128

128

0.449

194.59

vPMem

128 + 128

0

0.305

108.52

vPMEM-aware

128

0

0.186

85.84

Table 18: vMotion Redis downtime

Note: Spectre/Meltdown mitigations can impact the performance of a workload depending on the I/O rate and/or the frequency of system calls. [1]

 

[1] We used kernel 3.10.0-693.21 (patched) to measure the impact of Spectre/Meltdown mitigations. In FIO, we see a 38% loss in the vPMEM configuration and only a 7% loss in the vPMEM-aware (libpmem) configuration. The lower overhead in libpmem is because it is a user-space library. For OLTP workloads, like HammerDB with Oracle Database, we observe a less than 10% performance degradation with vPMEM.

check-circle-line exclamation-circle-line close-line
Scroll to top icon