by Cameron Cameron, Senior Consultant

Note: It is assumed that the audience for this document already understands how VMware manages memory for all of the VMs running on each ESXi host.

One of the great advantages of VMware is the ability to oversubscribe CPU and memory resources, allowing consolidation of workloads. When virtualizing systems for our clients, which have a large memory footprint (i.e. databases), we often see physical memory being depleted before we approach the limit of the compute capacity of an ESXi host. For Tier-1 workloads, we recommend configuring memory reservations, especially in environments where these business-critical workloads are mixed with lower-tier VMs.

Not every VMware shop chooses to use memory reservations. Sometimes, this can trigger memory-ballooning when certain events, including vMotion, cause the provisioned vRAM to approach, or match, the physical RAM on the ESXi host. When this happens, the ESXi host will likely experience a memory balloon event, and any VM not protected by a memory reservation may be impacted. Relying on a summation of the provisioned vRAM, and comparing it to the physical RAM, may lead to a false sense of security however, as often the memory overhead of each VM has not been taken into account.

Rather, House of Brick recommends that the sum of all virtual machines’ memory (virtual machine memory, VM memory overhead, and VM kernel memory) not exceed the physical memory of the host.

From Overhead Memory on Virtual Machines:

Virtual machines require a certain amount of available overhead memory to power on. You should be aware of the amount of this overhead.

The following table lists the amount of overhead memory a virtual machine requires to power on. After a virtual machine is running, the amount of overhead memory it uses might differ from the amount listed in the table. The sample values were collected with VMX swap enabled and hardware MMU enabled for the virtual machine. (VMX swap is enabled by default.)

Note:
The table provides a sample of overhead memory values and does not attempt to provide information about all possible configurations. You can configure a virtual machine to have up to 64 virtual CPUs, depending on the number of licensed CPUs on the host and the number of CPUs that the guest operating system supports.

Table 1: Sample Overhead Memory on Virtual Machines

What you can see from the table above is that, as the memory allocation to a VM increases, so does the memory overhead. While ESXi can typically reclaim over-allocated memory (via ESXi memory management mechanisms), the overhead cannot be reclaimed, which underscores the need to right-size workloads.

Each organization needs to determine their own policies regarding memory reservations in their VMware environment. But, if you are running Tier-1 workloads with other workloads, are running tight on memory, or are seeing memory-ballooning, be aware that there is memory overhead for each VM that may not have been accounted for during planning.