Mike Stone (@HoBMStone), CIO & Principal Architect
In our travels helping customers design and tune systems for running Oracle workloads, we often recommend the use of Jumbo Frames at the network layer. Wikipedia has a nice summary of Jumbo Frames with many additional references. But the short of it is, that we can reduce the CPU and bandwidth required to transmit data when we know the data is going to be large in size or when a transmission unit larger than 1500 bytes is synergistic with the traffic.
Network Attached Storage (NAS)
An example of the first use case is NAS protocols such as SMB, NFS, iSCSI and FCoE. When NAS is a key component of the infrastructure, we always recommend the use of a dedicated storage network and Jumbo Frames.
Synergistic Traffic
For us, the primary example of synergistic traffic is the Oracle RAC interconnect, which typically sends data in 8K blocks. By using a 9,000 byte Jumbo Frames, we dramatically improve the efficiency of Oracle Cache Fusion. When used in conjunction with NODELAY (disabling the Nagle Algorithm) the throughput and performance of the interconnect can be improved by as much as 50% by allowing each block to be transmitted in a single network packet and to be sent without delay.
So, now that you’re on board and understand two important use cases for Jumbo Frames, let’s discuss the basics and pitfalls of making it work.
First, NAS storage really became a serious enterprise option with the introduction of gbit (1000baseT). We have found that configuring Jumbo Frames can increase throughput performance by at least 25%. With the advent of 10gbit networking, the price point for NAS makes it a serious consideration for cases that were previously the domain of fiber only. So using Jumbo Frames is also a key to realizing the maximum performance from your NAS.
Requirements and Limitations
Of course, with any technology there are always caveats. Every piece of network equipment involved must support it. That means everything from the NICs through the switches, routers and firewalls. This is generally not a problem as long as you are not using home / small office hardware.
Next, configuration can be somewhat tedious – particularly in a non-heterogeneous network. Each vendor’s implementation and support tools may differ, so configuring it for a LAN/VLAN may involve some research. Finally, it must be configured at the ESX layer in vSphere as well as the guest operating systems.
Once it’s configured, be sure to verify that any pre-existing traffic is still flowing correctly. There is always a risk that the configuration was incomplete or that an incompatible piece of equipment could cause disruption of network traffic over existing VLANs. This is probably the source of the majority of problems we encounter when enabling Jumbo Frames on existing segments. It is for this reason that we recommend implementation be done during off hours or scheduled maintenance windows.
Verification
Finally, we need to verify that it’s working. VMware KB Article 1003712 is a great summary of how to accomplish this. The quick and dirty is to do a ping from end to end to ensure it’s working, but the article also includes some basic troubleshooting steps:
To test Jumbo Frames:
- Test ping to your neighbor’s virtual machine interface with the command:
- From Windows: ping -f -l 8972 a.b.c.d
- From Unix: ping -s 8972 a.b.c.d
The output appears similar to:
PING a.b.c.d (a.b.c.d) 8972(9000) bytes of data. 8980 bytes from a.b.c.d: icmp_seq=1 ttl=128 time=3.36 ms
Hopefully, you now have a better understanding of the where, why and how to use Jumbo Frames to improve performance at the network layer. In our experience, the use of Jumbo Frames can improve throughput and performance, and we encourage you to research whether or not it can improve performance in your environment.
