Software-Based Memory Virtualization

ESXi virtualizes guest physical memory by adding an extra level of address translation.

• The VMM maintains the combined virtual-to-machine page mappings in the shadow page tables. The shadow page tables are kept up to date with the guest operating system's virtual-to-physical mappings and physical-to-machine mappings maintained by the VMM.
• The VMM intercepts virtual machine instructions that manipulate guest operating system memory management structures so that the actual memory management unit (MMU) on the processor is not updated directly by the virtual machine.
• The shadow page tables are used directly by the processor's paging hardware.
• There is non-trivial computation overhead for maintaining the coherency of the shadow page tables. The overhead is more pronounced when the number of virtual CPUs increases.

This approach to address translation allows normal memory accesses in the virtual machine to execute without adding address translation overhead, after the shadow page tables are set up. Because the translation look-aside buffer (TLB) on the processor caches direct virtual-to-machine mappings read from the shadow page tables, no additional overhead is added by the VMM to access the memory. Note that software MMU has a higher overhead memory requirement than hardware MMU. Hence, in order to support software MMU, the maximum overhead supported for virtual machines in the VMkernel needs to be increased. In some cases, software memory virtualization may have some performance benefit over hardware-assisted approach if the workload induces a huge amount of TLB misses.

Performance Considerations

The use of two sets of page tables has these performance implications.

• No overhead is incurred for regular guest memory accesses.

• Additional time is required to map memory within a virtual machine, which happens when:

  • The virtual machine operating system is setting up or updating virtual address to physical address mappings.
  • The virtual machine operating system is switching from one address space to another (context switch).
• Like CPU virtualization, memory virtualization overhead depends on workload.