House of Brick Senior Consultant
Being a true computer nerd, periodically I feel the need to learn a new application or skill. Most recently, I had some discussions with other nerds about the advantages of using Linux’ native udev rules rather than Oracle’s ASMLIB for device persistence on Linux/Unix servers.
First – a quick history of ASMLIB. Originally available on RHEL 4 and 5, ASMLIB became unavailable on RHEL 6 because Oracle stopped maintaining it for RHEL. After much negative publicity, Oracle made the kmod-oracleasm package available for RHEL. This replaced Oracle ASMLIB with about the same functionality, and only required a couple of packages from the Public OEL Yum servers to make it work properly (oracleasm and oracleasm-support). Even so, there are still discussions, often tantamount to religious wars, with regard to the advantages of ASMLIB versus udev.
As a consequence, I felt it would be a good idea for me to learn how to use udev. Udev is designed to accomplish the same task that ASMLIB does, that is persist device names and privileges across system reboots on Linux servers. Linux has the unfortunate habit of renaming disk devices when you add new ones, eg, /dev/sdc might become /dev/sdd when a new device is added, and the new device becomes /dev/sdc. Udev uses the unique SCSI device id (UUID) to associate a specific storage device with a specific mount point on the Linux server, thus making the Linux name moot. Udev can be used on most, if not all, devices on Linux servers although this article only discusses its usage with storage devices. There are numerous examples of udev rules for use with Oracle storage. In general, the format (in OEL/RHEL 6.5) is something like this:
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="14f504e46494c45524856447a376e2d7871364a2d654b4272", NAME+="oracleasm/asm-disk1", OWNER="oracle", GROUP="dba", MODE="0660"
Where KERNEL represents the device interface type (sd is SCSI, hd would be ide, there are others) SUBSYSTEM is the storage type (block) PROGRAM is the command to query the device, RESULT is the return value of the query, and the remainder of the fields show how to use the data returned. In the example above, the device with a name in the pattern SD?1 returning the result “14f504e46494c45524856447a376e2d7871364a2d654b4272” gets named “/dev/oracleasm/asm-disk1”, with Oracle as the owner, dba as the group, and protections are 660. Once the rule set works properly, the file can be duplicated for other nodes when using a cluster. The last two items are the key to the storage udev rules, the long ID number is unique to the storage device, and as a consequence will always be associated with the appropriate name. Thus, when the same set of rules is copied to other nodes on a cluster, each storage device is named correctly and as a result, software such as Oracle can find the storage where it is expected.
Udev rule files are also used to name network devices, cd and dvd drives, etc, and are stored in the directory /udev/rules.d. The rules files are always prefixed with a number indicating the order in which they will be executed followed by a name, followed by a suffix .rules. For example, 99-oracleasm-devices.rules. This is very similar in design to Linux runlevel startup scripts.
As you might have gathered, udev rules are a fundamental piece of making Linux work. So, this really makes me wonder why such a fundamental piece of the Linux OS was changed, in what is a fairly dramatic way, in the next release of Redhat Linux, RHEL 7. Specifically, the location of the scsi_id program has changed, and the parameter used in the command have changed. The NAME option can no longer be used for disk devices, it only works for network devices. So the equivalent of the udev rule above, in RHEL 7 would be this:
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="14f504e46494c45524856447a376e2d7871364a2d654b4272", SYMLINK+="oracleasm/asm-disk1", OWNER="oracle", GROUP="dba", MODE="0660"
Note that it now creates a symbolic link for the device. While I’m not, and do not claim to be, a Linux System Administrator, I would have expected to be able to search for ‘RHEL7 Udev rules’ and get some sort of explanation of this change or the expected rule change required, but to this point I have not found anything. The designers of this may have intended to use a different parameter than SYMLINK in the udev rules, but it appears to be the only option at this point.
To make the creation of udev rules a little quicker, I wrote a very simple script to generate the rules:
[root@oel7614 ~]# cat setudev.sh
#!/bin/sh
COUNTER=0
for id in `lsscsi -i 3 | awk -F " " '{print$7}'`
do
COUNTER=$((COUNTER+1))
echo KERNEL==\"sd?1\", SUBSYSTEM==\"block\", PROGRAM==\"/usr/lib/udev/scsi_id -g -u -d /dev/\$parent\", RESULT==\"${id}\", SYMLINK+=\"oracleasm/asm-disk$COUNTER\", OWNER=\"oracle\", GROUP=\"dba\", MODE=\"0660\"
done
Please note that the script above, while useful for me, makes some assumptions. One, it assumes that the disks of interest are on the third SCSI channel, which is probably not the case on the readers system. And two, it assumes that the UUID shows up in the seventh column of output and these will vary by device type and storage software. I am using openfiler and vmdk’s for my storage.
In order to see all SCSI devices on your server, run the lsscsi. The output should be similar to that shown below, but please note that this is merely an example from my VMware workstation:
[root@oel7614 ~]# lsscsi -i [root@oel7614 ~]# lsscsi -i [1:0:0:0] cd/dvd NECVMWar VMware IDE CDR10 1.00 /dev/sr0 - [2:0:0:0] disk VMware, VMware Virtual S 1.0 /dev/sda 36000c29105108d894764376c6756ebf7 [3:0:0:0] disk OPNFILER VIRTUAL-DISK 0 /dev/sdb 14f504e46494c45524856447a376e2d7871364a2d654b4272 [3:0:0:1] disk OPNFILER VIRTUAL-DISK 0 /dev/sdc 14f504e46494c45524e78676a70372d556d43312d4f706379 [3:0:0:2] disk OPNFILER VIRTUAL-DISK 0 /dev/sdd 14f504e46494c455246305034744e2d786133422d79507066 [3:0:0:3] disk OPNFILER VIRTUAL-DISK 0 /dev/sde 14f504e46494c45524b46664d4c332d5863497a2d58764357 [3:0:0:4] disk OPNFILER VIRTUAL-DISK 0 /dev/sdf 14f504e46494c4552644b424254762d566468522d48355254 [3:0:0:5] disk OPNFILER VIRTUAL-DISK 0 /dev/sdg 14f504e46494c4552657172324e572d4e4676362d576c7944 [3:0:0:6] disk OPNFILER VIRTUAL-DISK 0 /dev/sdh 14f504e46494c45524e4b744332762d795862672d30377676 [3:0:0:7] disk OPNFILER VIRTUAL-DISK 0 /dev/sdi 14f504e46494c455231594d6c4d462d3456776f2d4d523552 [3:0:0:8] disk OPNFILER VIRTUAL-DISK 0 /dev/sdj 14f504e46494c45523452636e62632d4e4177692d51644665 [3:0:0:9] disk OPNFILER VIRTUAL-DISK 0 /dev/sdk 14f504e46494c455269314c68397a2d50646e352d316e4f33 [3:0:0:10] disk OPNFILER VIRTUAL-DISK 0 /dev/sdl 14f504e46494c455233686b36694e2d67514a632d58344e47 [3:0:0:11] disk OPNFILER VIRTUAL-DISK 0 /dev/sdm 14f504e46494c4552686b706e304b2d384e714f2d52586175
This is the output of the setudev script:
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="14f504e46494c45524856447a376e2d7871364a2d654b4272", SYMLINK+="oracleasm/asm-disk1", OWNER="oracle", GROUP="dba", MODE="0660" KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="14f504e46494c45524e78676a70372d556d43312d4f706379", SYMLINK+="oracleasm/asm-disk2", OWNER="oracle", GROUP="dba", MODE="0660" KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="14f504e46494c455246305034744e2d786133422d79507066", SYMLINK+="oracleasm/asm-disk3", OWNER="oracle", GROUP="dba", MODE="0660" KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="14f504e46494c45524b46664d4c332d5863497a2d58764357", SYMLINK+="oracleasm/asm-disk4", OWNER="oracle", GROUP="dba", MODE="0660" KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="14f504e46494c4552644b424254762d566468522d48355254", SYMLINK+="oracleasm/asm-disk5", OWNER="oracle", GROUP="dba", MODE="0660" KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="14f504e46494c4552657172324e572d4e4676362d576c7944", SYMLINK+="oracleasm/asm-disk6", OWNER="oracle", GROUP="dba", MODE="0660" KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="14f504e46494c45524e4b744332762d795862672d30377676", SYMLINK+="oracleasm/asm-disk7", OWNER="oracle", GROUP="dba", MODE="0660" KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="14f504e46494c455231594d6c4d462d3456776f2d4d523552", SYMLINK+="oracleasm/asm-disk8", OWNER="oracle", GROUP="dba", MODE="0660" KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="14f504e46494c45523452636e62632d4e4177692d51644665", SYMLINK+="oracleasm/asm-disk9", OWNER="oracle", GROUP="dba", MODE="0660" KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="14f504e46494c455269314c68397a2d50646e352d316e4f33", SYMLINK+="oracleasm/asm-disk10", OWNER="oracle", GROUP="dba", MODE="0660" KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="14f504e46494c455233686b36694e2d67514a632d58344e47", SYMLINK+="oracleasm/asm-disk11", OWNER="oracle", GROUP="dba", MODE="0660" KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="14f504e46494c4552686b706e304b2d384e714f2d52586175", SYMLINK+="oracleasm/asm-disk12", OWNER="oracle", GROUP="dba", MODE="0660"
The RHEL/OEL 6.5 version of the script and output are:
[root@oel7614 ~]# cat setudev65.sh
#!/bin/sh
COUNTER=0
for id in `lsscsi -i 3 | awk -F " " '{print$7}'`
do
COUNTER=$((COUNTER+1))
echo KERNEL==\"sd?1\", SUBSYSTEM==\"block\", PROGRAM==\"/sbin/scsi_id -g -u -d /dev/\$parent\", RESULT==\"${id}\", NAME+=\"oracleasm/asm-disk$COUNTER\", OWNER=\"oracle\", GROUP=\"dba\", MODE=\"0660\"
done
[root@oel7614 ~]# ./setudev65.sh
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="14f504e46494c45524856447a376e2d7871364a2d654b4272", NAME+="oracleasm/asm-disk1", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="14f504e46494c45524e78676a70372d556d43312d4f706379", NAME+="oracleasm/asm-disk2", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="14f504e46494c455246305034744e2d786133422d79507066", NAME+="oracleasm/asm-disk3", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="14f504e46494c45524b46664d4c332d5863497a2d58764357", NAME+="oracleasm/asm-disk4", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="14f504e46494c4552644b424254762d566468522d48355254", NAME+="oracleasm/asm-disk5", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="14f504e46494c4552657172324e572d4e4676362d576c7944", NAME+="oracleasm/asm-disk6", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="14f504e46494c45524e4b744332762d795862672d30377676", NAME+="oracleasm/asm-disk7", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="14f504e46494c455231594d6c4d462d3456776f2d4d523552", NAME+="oracleasm/asm-disk8", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="14f504e46494c45523452636e62632d4e4177692d51644665", NAME+="oracleasm/asm-disk9", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="14f504e46494c455269314c68397a2d50646e352d316e4f33", NAME+="oracleasm/asm-disk10", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="14f504e46494c455233686b36694e2d67514a632d58344e47", NAME+="oracleasm/asm-disk11", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="14f504e46494c4552686b706e304b2d384e714f2d52586175", NAME+="oracleasm/asm-disk12", OWNER="oracle", GROUP="dba", MODE="0660"
Of course, the above information is not comprehensive. The intent of this blog post is to give a basic tutorial on the usage of udev for storage on Oracle servers, but it will not make you an expert on the subject! If, for example, multi-pathed devices are used, the udev rules are different. There are many other cases where the rules will be different, so this is just a basic introduction.
