Jeff Stonacek, Principal Architect

High availability is a common topic addressed by IT professionals worldwide. Real Application Clusters (RAC) is Oracle’s answer to delivering the highest levels of availability for your databases. RAC is a complex clustering technology that ensures your database is always available. The inner workings of RAC are beyond the scope of this article, but keep in mind is that with advanced features comes increased complexity. Specific to RAC, shared storage is required for the database and clusterware components.

This post specifically addresses adding shared storage to multiple virtual machines in a vSphere environment without downtime. Additionally, I will describe in detail a bug that exists in ESXi 5.5 preventing shared storage from being added to running virtual machines.

Shared Storage and VMware

While this post is specific to running Oracle RAC on VMware, the issues described here relate to any system that requires shared storage. When it comes to sharing storage on VMware there are three main options:

• RDM
• Shared SCSI Bus
• VMDK with Multi-writer

Using a shared SCSI bus is not desirable as it disables one of the main features of vSphere, vMotion. RDMs are not desirable as they induce a high level of overhead for VMware and storage administrators. Using VMDKs with the multi-writer flag is the most desirable option when it comes to configuring storage that must be shared amongst multiple virtual machines.

Multi-writer Complexity

The issue with using the multi-writer flag is that it is an advanced virtual machine setting. Setting the flag is simply a matter of navigating to the VM Options tab of the virtual machine settings and adding an advanced setting.

The problem with adding the any advanced setting through the GUI or web interface is that it requires the virtual machine to be powered off. Having to power off the virtual machines in a RAC cluster just to add storage is counter intuitive to the HA concept. The question House of Brick receives most often is, “I don’t have to power off my physical machines to add storage to my RAC cluster, so why do I have to do this with VMware?”

Perl SDK / Power CLI

The answer to the hot-add multi-writer flag problem is to use the VMware Perl SDK or the Power CLI to add the advanced setting with the virtual machine running. House of Brick has developed several Perl SDK and Power CLI scripts to help our customers perform automated maintenance on vSphere including adding VMDKs and setting the multi-writer flag for shared storage.

Hot-Add Bug

It seems that in IT no problem ever has a simple solution. Even though the functionality of using the Perl SDK or Power CLI to add VMDKs and set the multi-writer flag is supported by VMware in vSphere 5.5, there is a bug that prevents this from happening. This bug affects both update 0 (build 1331820) and update 1 (build 1623387) of ESXi version 5.5.

The bug in the process is related to the lock mode that ESXi places on the newly created VMDK file. When the VMDK is initially created, ESXi places a mode 1 lock on the file. Once the multi-writer flag is set, the lock is changed to mode 3, or a shared lock. A mode 3 lock allows for multiple virtual machines to share a VMDK file. However, the bug causes the lock to remain in mode 1 even after the multi-writer flag is set and the mode is not changed to 3 until the VM is rebooted. Below is an example of debug output from a VMDK that was hot-added to a virtual machine along with setting the multi-writer flag on ESXi 5.5 Update 0.

# vmkfstools -D crs010_1-flat.vmdk
Lock [type 10c00001 offset 121083904 v 116, hb offset 3346432
gen 81, mode 1, owner 53e3c1df-3efab912-6666-000c2986f4ca mtime 1257
num 0 gblnum 0 gblgen 0 gblbrk 0]
Addr <4, 279, 59>, gen 109, links 1, type reg, flags 0, uid 0, gid 0, mode 600
len 1073741824, nb 1024 tbz 0, cow 0, newSinceEpoch 1024, zla 3, bs 1048576

The Patch

The good news is that VMware has acknowledged this bug and has issued a patch fixing the problem. The fix is located in ESXi 5.5 Patch 2 (build 1892794), which was released on July 1, 2014.

Here is an example of the debug output from a VMDK that was added to a patched system.

# vmkfstools -D crs010_1-flat.vmdk
Lock [type 10c00001 offset 83988480 v 827, hb offset 3801088
gen 245, mode 3, owner 00000000-00000000-0000-000000000000 mtime 1652
num 1 gblnum 0 gblgen 0 gblbrk 0]
MW Owner[0] HB Offset 3801088 53e3c67f-e57d3651-3730-000c29fbdf25
Addr <4, 189, 10>, gen 820, links 1, type reg, flags 0, uid 0, gid 0, mode 600
len 1073741824, nb 1024 tbz 0, cow 0, newSinceEpoch 1024, zla 3, bs 1048576

Once the system is properly patched, the operation to hot-add the VMDK to multiple virtual machines, along with the multi-writer flag, finishes successfully without requiring you to power off the virtual machines.

In this post I have reviewed the process for adding shared storage to multiple virtual machines in a vSphere environment without downtime. I also covered a bug that exists in ESXi 5.5 preventing shared storage from being added to running virtual machines.