Oracle/Solaris Cluster configuration with Soft partitions Metasets

*Pre-requisites*:

1. Solaris OS, Solaris Cluster Software should have been installed BEFORE proceeding with the below procedure
2. LUNS from the shared storage should have been already mapped to both the nodes and should be visible
3. Multipath software - MPxio or any 3rd party should be enabled.
4. scdidadm -L should confirm multiple-paths to the same LUNs and should show the same on the remote node as well,

STEPS

Define the diskset and the nodes that can master the diskset as follows:

# metaset -s nfsds -a -h phys-node-1 phys-node-2

Only run the metaset command on one node. The metaset command issues rpc calls to the other node to create an entry in the other node's local replicas for this set.
Add drives to the diskset using their fully qualified DID names as follows:

# metaset -s nfsds -a /dev/did/rdsk/d3 /dev/did/rdsk/d4 /dev/did/rdsk/d5 /dev/did/rdsk/d6

Again, rpc calls are made to the other node to ensure that it also has the necessary information in its local state replicas.
When Solaris Volume Manager software creates disksets, it automatically tells the cluster software to create an entry in the cluster database for the diskset. This entry is called a device group, which you can check using the following command:

# scstat -D

Configure mediators on each node, as follows:

# metaset -s nfsds -a -m phys-node-1
# metaset -s nfsds -a -m phys-node-2

Check the status of the mediators as follows:

# medstat -s nfsds

To Create Volumes and File Systems
Create the following volumes.
A mirrored volume composed of two striped submirrors. Each stripe is composed of the two slice 0s of the LUN in each storage array.
Four soft partitions on top of this mirror.

# metainit nfsds/d1 1 2 /dev/did/rdsk/d3s0 /dev/did/rdsk/d4s0

# metainit nfsds/d2 1 2 /dev/did/rdsk/d5s0 /dev/did/rdsk/d6s0

# metinit nfsds/d0 -m nfsds/d1
# metattach nfsds/d0 nfsds/d2
# metainit nfsds/d10 -p nfsds/d0 200m
# metainit nfsds/d11 -p nfsds/d0 200m
# metainit nfsds/d12 -p nfsds/d0 200m
# metainit nfsds/d13 -p nfsds/d0 200m

Create an /etc/lvm/md.tab file on both nodes. Keep a copy of this file offline so you can use it to recreate the configuration, if it becomes necessary to do so.
On the node that is the current primary of the diskset, supply the following commands:

# metastat -s nfsds -p >> /etc/lvm/md.tab

On the other node, supply the following commands:

# metaset -s nfsds -p | tail +2 >> /etc/lvm/md.tab

Create file systems on the logical partitions and mount them globally.
On one node, create the file system as shown here:

# newfs /dev/md/nfsds/rdsk/d10
# newfs /dev/md/nfsds/rdsk/d11
# newfs /dev/md/nfsds/rdsk/d12
# newfs /dev/md/nfsds/rdsk/d13

On all nodes, create mount points as follows:

# mkdir /global/nfsd/fs1 /global/nfsd/fs2 /global/nfsd/fs3 /global/fs4

On all nodes, put following entries in /etc/vfstab as shown here:

/dev/md/nfsds/dsk/d10 /dev/md/nfsds/rdsk/d10 /global/nfsd/fs1 ufs 2 yes global
/dev/md/nfsds/dsk/d11 /dev/md/nfsds/rdsk/d11 /global/nfsd/fs2 ufs 2 yes global
/dev/md/nfsds/dsk/d12 /dev/md/nfsds/rdsk/d12 /global/nfsd/fs3 ufs 2 yes global
/dev/md/nfsds/dsk/d13 /dev/md/nfsds/rdsk/d13 /global/nfsd/fs4 ufs 2 yes global

On one node, mount the global file systems as follows:

# mount /global/nfsd/fs1
# mount /global/nfsd/fs2
# mount /global/nfsd/fs3
# mount /global/nfsd/fs4

Test the configuration.

On one node, run the following commands:

# metaset
# scstat -D

On all nodes, run the following command:

# df -k