Joe Grant (@dba_jedi), Principal Architect

One of the most asked questions concerning running Oracle Databases in a virtualized environment is:

“How do we present the fastest storage possible?”

It is not always necessary to present the fastest storage available, but rather to present storage that is easy to manage and is as flexible as possible. In addition, there may be storage tools from the storage vendor that may need to be integrated into the solution. Examples are Snap Manger for Oracle (or SQL Server) or other snap technologies. These additional tools may require a specific type of storage such as NFS for all of the data volumes with SMO.

Storage Types

There are several different ways to present storage to a VM, with different pros and cons for each type available. In general, House of Brick recommends using VMDKs whenever possible. Additional tooling at the SAN and/or network layers may influence the final decision for how storage is presented for a specific VM. More than one storage presentation option can be used for any given VM. For example, the OS disk for a VM can be a VMDK and the data volumes can be RDMs or NFS mounts.

VMDK

A Virtual Machine Disk (VMDK) is a flat file that resides in a datastore, which is accessible to the vSphere servers. Through vSphere, the VMDK is presented to the operating system as a physical disk. Using VMDKs for storage has the fewest restrictions on vSphere tooling and any new features added to vSphere will work with VMDKs first, Since VMDKs are the most common storage presentation, VMware support will be most familiar with their use.

IP based storage: NFS, dNFS, iSCSI

It is possible to present IP based storage directly to the guest operating system. Assuming networking is correctly configured at the vSphere layer, there are few to no restrictions on the VM. However, this type of storage does require a reliable IP based network that is capable of high throughput and low latency. Typically 10 GE networks with multiple network interfaces aggregated for increased throughput are used for IP based storage. A common use case for IP based storage is to use dNFS with Oracle databases for the datafiles. The OS and binary disks are still VMDK based. This configuration also allows for compatible SAN tooling to be used, a good example would be Snap Manager for Oracle from NetApp.

RDM

Raw Device Mapping (RDM) is a method of presenting storage from a SAN straight to the guest operating system as if it were physical. There are two compatibility modes for RDMs, virtual and physical. Once storage is presented to the guest operating system it functions as if it were physical. Typical use cases for RDMs are for Microsoft Windows Clustering (the initial reason they were developed) or to enable SAN tooling.

There are some restrictions when using RDMs. The following is copied from the vSphere Storage 5.5 update 2 guide.

RDM Considerations and Limitations
Certain considerations and limitations exist when you use RDMs:

• The RDM is not available for direct-attached block devices or certain RAID devices. The RDM uses a SCSI serial number to identify the mapped device. Because block devices and some direct-attach RAID devices do not export serial numbers, they cannot be used with RDMs.
• If you are using the RDM in physical compatibility mode, you cannot use a snapshot with the disk. Physical compatibility mode allows the virtual machine to manage its own, storage-based, snapshot or mirroring operations.
• Virtual machine snapshots are available for RDMs with virtual compatibility mode.
• You cannot map to a disk partition. RDMs require the mapped device to be a whole LUN.
• If you use vMotion to migrate virtual machines with RDMs, make sure to maintain consistent LUN IDs for RDMs across all participating ESXi hosts.
• Flash Read Cache does not support RDMs in physical compatibility. Virtual compatibility RDMs are supported with Flash Read Cache.

Note: In point 2 the snapshot would be a VM snapshot.

There are additional restrictions when using RDM and these restrictions change based on the version of vSphere. Please refer to the VMware Knowledge Base article 1005241 for further information.

Summary

So in the end the best storage may not necessarily be the fastest storage. Use any database statistics (for Oracle AWR reports), OS statistics and/or metrics from the storage devices to determine true storage performance needs. Then determine the management needs of the environment. Once these pieces of information are gathered and understood, then you can decide which presentation method will be best for your environment. Keep in mind that it is not uncommon to place the OS and data volumes in separate datastores, or to present them from different types of storage.