OEL - UDEV setup in a Multipath env for RAC/ASM

Environment

OEL 6.4 with
11.2.0.4.0 RAC system

Map /dev/dm-? devices to the asm disk discovery string like /dev/asm

Display ISCSI partitions/Disks available for ASM

# dmsetup ls
--tree -o device

vg_oel64-lv_swap (252:1)

 +-
(8:3)

 +-
(8:2)

vg_oel64-lv_root (252:0)

 +-
(8:2)

grac41_disk1p1
(252:6)

 +-grac41_disk1
(252:2)

+- (8:240)

+- (65:32)

grac41_disk2p1
(252:7)

 +-grac41_disk2
(252:3)

+- (65:0)

+- (65:48)

grac41_disk3p1
(252:5)

 +-grac41_disk3
(252:4)

+- (65:16)

+- (65:64)

What is  Major number 252 ?

# cat /proc/devices |  egrep 'devices|252'

..

Block
devices:

252
device-mapper

-->  Driver for
Device-Mapper

Availabe partitions:

# dmsetup ls
| grep p1

grac41_disk1p1
(252:5)

grac41_disk2p1
(252:7)

grac41_disk3p1
(252:6)

Map /dev/dm-? devices to /dev/asmdisk_OF-disk1 ( for example /dev/dm-5 to asmdisk_OF-disk1)

Create/Add to  12-dm-permissions.rules in /etc/udev/rules.d following
entries:

ENV{DM_NAME}=="grac41_disk1p1",NAME="asmdisk_OF-disk1",
OWNER:="grid", GROUP:="asmadmin",
MODE:="660"

ENV{DM_NAME}=="grac41_disk2p1",NAME="asmdisk_OF-disk2",
OWNER:="grid", GROUP:="asmadmin",
MODE:="660"

ENV{DM_NAME}=="grac41_disk3p1",NAME="asmdisk_OF-disk3",
OWNER:="grid", GROUP:="asmadmin",
MODE:="660"

• Change the protections for underlying sdX disks used by Device Mapper 

Note : Don’t use Name in the 99-oracle-asmdevices.rules file as we get 2 disks for a
single WWID.

• Here we change group, owner and protection of the /dev/sdX devices created by ISCSI. 

Check WWID of ISCSI disks

#  multipath -ll
|      grep dm-

grac41_disk3
(14f504e46494c45525237556f4c352d574b4e6c2d446e736b) dm-4 OPNFILER,VIRTUAL-DISK

grac41_disk2
(14f504e46494c4552786431396d6c2d6d4950492d52517178) dm-2 OPNFILER,VIRTUAL-DISK

grac41_disk1
(14f504e46494c45523264556151442d5261336d2d566e4250) dm-3 OPNFILER,VIRTUAL-DISK

Create/Add to  99-oracle-asmdevices.rules in /etc/udev/rules.d following entries

KERNEL=="sd*", SUBSYSTEM=="block", ENV{DEVTYPE}=="disk",
ENV{ID_SERIAL}=="14f504e46494c45525237556f4c352d574b4e6c2d446e736b",
OWNER="grid", GROUP="asmadmin",
MODE="0660"

KERNEL=="sd*", SUBSYSTEM=="block", ENV{DEVTYPE}=="disk",
ENV{ID_SERIAL}=="14f504e46494c4552786431396d6c2d6d4950492d52517178",
OWNER="grid", GROUP="asmadmin",
MODE="0660"

KERNEL=="sd*", SUBSYSTEM=="block", ENV{DEVTYPE}=="disk",
ENV{ID_SERIAL}=="14f504e46494c45523264556151442d5261336d2d566e4250",
OWNER="grid", GROUP="asmadmin",
MODE="0660"

Reload UDEV ( OEL 6 stile )

# udevadm control --reload-rules

# start_udev

Verify ASM disk protection

# ls -l /dev/asmdisk_OF*

brw-rw----. 1 grid asmadmin 252, 5 Feb 28
10:06 /dev/asmdisk_OF-disk1

brw-rw----. 1 grid asmadmin 252, 7 Feb 28
10:06 /dev/asmdisk_OF-disk2

brw-rw----. 1 grid asmadmin 252, 6 Feb 28
10:06 /dev/asmdisk_OF-disk3

Find all attached sd disks used by our multipath configuration

# multipath -ll | grep sd

|- 21:0:0:0 sdt 65:48 active ready running

`- 22:0:0:0 sdu 65:64 active ready running

|- 17:0:0:0 sdp 8:240 active ready running

`- 18:0:0:0 sdr 65:16 active ready running

|- 20:0:0:0 sdq 65:0  active ready running

`- 19:0:0:0 sds 65:32 active ready running

Verify SD disk protection

# ls -l /dev/sdt /dev/sdu /dev/sdp /dev/sdr /dev/sdq /dev/sds

brw-rw----. 1 grid asmadmin
8, 240 Feb 28 10:55 /dev/sdp

brw-rw----. 1 grid asmadmin 65,   0
Feb 28 10:55 /dev/sdq

brw-rw----. 1 grid asmadmin 65,  16 Feb 28 10:55 /dev/sdr

brw-rw----. 1 grid asmadmin 65,  32 Feb 28 10:55 /dev/sds

brw-rw----. 1 grid asmadmin 65,  48 Feb 28 10:55 /dev/sdt

brw-rw----. 1 grid asmadmin 65,  64 Feb 28 10:55 /dev/sdu

- The major number changes as there are eight major numbers reserved
( or even more ) for SCSI devices

- Those major numbers are 8, 65, 66, 67, 68, 79, 70, and 71

Verify your ASM disks on all nodes before you use asmca to create
an OPENFILER diskgroup