Shawn Meyers, Principal Architect
Last week I was at VMworld Europe giving a presentation on vNUMA and its impact on SQL server. This post will cover some of the highlights from my presentation and items you should be aware of when dealing with very large virtual machines.
What is vNUMA?
The way VMware presents NUMA architecture to the virtual machine, is very important as Windows, and SQL Server are both fully NUMA aware and they make scheduling decisions for threads based on the NUMA topology. Having the proper NUMA architecture presented to the virtual machine means that the OS and SQL Server will be able to schedule threads in the most efficient manner.
When you read any of the VMware documentation regarding vNUMA, it says it takes effect when a virtual machine is greater than eight vCPUs. However, I don’t know why they just don’t say nine or more, since it causes confusion. I meet people all of the time who think their eight vCPU virtual machines have vNUMA turned on, when this is just not the case, since by default it is nine vCPUs.
Hot-Add Features and Hot-Add Memory and NUMA
There is also some confusion out there regarding the impact of VMware Hot-Add features on vNUMA. When you enable Hot-Plug CPU, it will disable vNUMA in the virtual machine. This means that if you use Hot-Plug CPU, the virtual machine will have no information about the physical NUMA topology.
Additionally, hot-add memory is where I see most of the confusion. When you turn on hot-add memory, it does not impact vNUMA but on ESXi 5.x, hot-add memory only adds the memory to the first NUMA node and your memory is imbalanced until the next power cycle of the virtual machine. However, this issue has been resolved in ESX 6.x, and memory added is balanced across the NUMA nodes.
When VMware Determines vNUMA
One of the other items that surprised me is when vNUMA is determined. VMware determines the vNUMA configuration on first power cycle of the virtual machine and stores it in the .VMX file. It does not recalculate vNUMA unless the number of vCPUs changes, or a flag in the advanced settings tells it to recalculate on next power cycle. If you migrate a virtual machine between hosts that have different NUMA configurations you will encounter a NUMA misalignment, which can cause performance issues.
To force the NUMA to reconfigure on every power cycle, there are two advanced settings under the VM, which will force this to occur. These settings are off by default because there are a few Linux based workloads, which do not work well with NUMA configuration changes, and so these are the safest default settings. For all large virtual machines that have vNUMA enabled, consider changing these advanced settings. Keep in mind that you can only adjust the VM advanced settings when the virtual machine is powered off, and these two settings will not be there, you will need to add them. If you have large virtual machines in a non-homogenous cluster, then the following settings should also be added.
numa.autosize = “TRUE”
numa.autosize.once = “FALSE”
CONCLUSION
In this post I have covered the highlights of my vNUMA and SQL Server session at VMworld. However, if you would like to watch the full presentation, you can do so on the VMUG website where a link to the webinar version of this presentation is archived. There is a lot more about NUMA covered in this recorded session, which isn’t covered in this post. If you have any questions about pNUMA or vNUMA, please leave a comment and I will get back you to answer your specific question.
