Solaris/VxVM–Recover Failed Disk when vxreattach fails

In the event that the paths to a given device suffer a (transient) failure with the SAN (array), the VxVM device will be reported as "failed was" by vxdisk list.

Solaris example

Unless the device is accessible via the OS device handle, VxVM will not be in a position to reattach the disk.

 

# prtvtoc /dev/vx/rdmp/emc0_0281
prtvtoc: /dev/vx/rdmp/emc0_0281: No such device or address

NOTE: At this time the disk access name (emc0_281) is not accessible via the O/S and the corresponding paths have been disabled by DMP:

 

# vxdmpadm getsubpaths dmpnodename=emc0_0281
NAME         STATE[A]   PATH-TYPE[M] CTLR-NAME  ENCLR-TYPE   ENCLR-NAME    ATTRS
================================================================================
c1t5006048C5368E580d334s2  DISABLED(M)    -          c1         EMC          emc0             -
c1t5006048C5368E5A0d325s2  DISABLED(M)    -          c1         EMC          emc0             -

 

When the impacted device is online once again and the O/S can communicate (without issue) with the underlying paths, the disk can normally be recovered back into the impacted diskgroup using the Veritas Volume Manager (VxVM) command "vxreattach".

 

# vxdmpadm getsubpaths dmpnodename=emc0_0281
NAME         STATE[A]   PATH-TYPE[M] CTLR-NAME  ENCLR-TYPE   ENCLR-NAME    ATTRS
================================================================================
c1t5006048C5368E580d334s2  ENABLED(A)    -          c1         EMC          emc0             -
c1t5006048C5368E5A0d325s2  ENABLED(A)    -          c1         EMC          emc0             -

# prtvtoc -s /dev/vx/rdmp/emc0_0281
*                          First     Sector    Last
* Partition  Tag  Flags    Sector     Count    Sector  Mount Directory
       2      5    01          0   4103040   4103039
       7     15    01          0   4103040   4103039

vxreattach

In the event that the "vxreattach" command is unable to associate the disk access (da) name back into the impacted diskgroup, another approach is required.

 

# vxreattach -c <Veritas-Disk-Access-Name>
VxVM vxdisk ERROR V-5-1-558 Disk <Veritas-Disk-Access-Name>: Disk not in the configuration 

 

The purpose of the "vxreattach" command is to reattach disk drives that have once again become accessible.

The vxreattach utility reattaches (recovers) disks back into the impacted diskgroup they were associated with, retaining the same disk media name.
The utility attempts to locate a disk in the same diskgroup with the same Veritas disk ID for the disk to be reattached.
The reattach operation may fail even after locating the disk with the corresponding disk ID, if the original case (or some other cause) for the disk failure still exists.

 
Scenario

Diskgroup "testdg" fails to import initially due to failed disk.

 

# vxdg import testdg
VxVM vxdg ERROR V-5-1-10978 Disk group testdg: import failed:
Disk for disk group not found

# vxdg -f import testdg
VxVM vxdg WARNING V-5-1-560 Disk emc0_0281: Not found, last known location: emc0_0281

# vxdisk -eg testdg list
DEVICE       TYPE           DISK        GROUP        STATUS               OS_NATIVE_NAME   ATTR
emc0_0280    auto:sliced    emc1_0280    testdg      online               c1t5006048C5368E5A0d324s2 std
-            -         emc0_0281    testdg       failed was:emc0_0281

 

In this instance it is not possible to use "vxreattach" to recover the failed disk back into the diskgroup "testdg".

 

# vxreattach -c emc0_0281
VxVM vxdisk ERROR V-5-1-558 Disk emc0_0281: Disk not in the configuration

Recovery procedure


To obtain the Veritas diskgroup (dg) id from the impacted diskgroup, type:
 

# vxdg -q list | grep testdg
testdg       enabled              1311240633.41.dopey

 

To obtain the disk attribute from the diskgroup configuration database, type:

# vxprint -g testdg -dF'%last_da_name %name %diskid'
emc0_0281 emc0_0281 1312382118.46.dopey
emc0_0280 emc1_0280 1311240596.39.dopey

 

To cross match the above diskid "1312382118.46.dopey" for the impacted Veritas disk access name "emc0_281" from the diskgroup configuration database to that of the VxVM kernel disk content, type:

Syntax:

# vxdisk -x DISKID -x DGID -p list | grep <dgid>

In this instance emc0_0281 has the same diskid reported from both outputs. Thus confirming that it is the same disk.
 

# vxdisk -x DISKID -x DGID -p list | grep 1311240633.41.dopey
emc0_0280    1311240596.39.dopey 1311240633.41.dopey
emc0_0281    1312382118.46.dopey 1311240633.41.dopey    <<<<< this is the failed disk

By clearing the diskgroup name the disk can be associated back into the diskgroup.
 

# vxdisk list emc0_0281
Device:    emc0_0281
devicetag: emc0_0281
type:      auto
hostid:    dopey
disk:      name= id=1312382118.46.dopey
group:     name=testdg id=1311240633.41.dopey        <<<<<<<<<< clear "diskgroup name"
info:      format=cdsdisk,privoffset=256,pubslice=2,privslice=2
flags:     online ready private autoconfig autoimport
pubpaths:  block=/dev/vx/dmp/emc0_0281s2 char=/dev/vx/rdmp/emc0_0281s2
guid:      {cf7741f4-bddd-11e0-bccf-0003baa707e3}
udid:      EMC%5FSYMMETRIX%5F000290300822%5F2200281000
site:      -
version:   3.1
iosize:    min=512 (bytes) max=2048 (blocks)
public:    slice=2 offset=65792 len=4037248 disk_offset=0
private:   slice=2 offset=256 len=65536 disk_offset=0  <<<<<<<<<<<
update:    time=1312385647 seqno=0.13
ssb:       actual_seqno=0.0
headers:   0 240
configs:   count=1 len=48144
logs:      count=1 len=7296
Defined regions:
config   priv 000048-000239[000192]: copy=01 offset=000000 enabled
config   priv 000256-048207[047952]: copy=01 offset=000192 enabled
log      priv 048208-055503[007296]: copy=01 offset=000000 enabled
lockrgn  priv 055504-055647[000144]: part=00 offset=000000
Multipathing information:
numpaths:   2
c1t5006048C5368E580d334s2       state=enabled
c1t5006048C5368E5A0d325s2       state=enabled

 

To clear the diskgroup name from the VxVM disk headers, type:

 

# /etc/vx/diag.d/vxprivutil set /dev/vx/rdmp/emc0_0281s2 dgname=""

 

 

NOTE: "s2" following "emc0_0281" happens to be the private region slice for the Veritas disk access (da) name "emc0_0281".

 

# vxdisk list emc0_0281 | grep group
group:     name= id=1311240633.41.dopey

 

Using the "-k" option with vxdg adddisk, it is possible to associate the failed disk back into the impacted diskgroup using the existing disk media (dm) name and corresponding da name:

 

# vxdg -g testdg -k adddisk emc0_0281=emc0_0281

# vxdisk -eg testdg list
DEVICE       TYPE           DISK        GROUP        STATUS               OS_NATIVE_NAME   ATTR
emc0_0280    auto:sliced    emc1_0280    testdg      online               c1t5006048C5368E5A0d324s2 std
emc0_0281    auto:cdsdisk   emc0_0281    testdg      online               c1t5006048C5368E580d334s2 std

Now that the disk has been recovered back into the diskgroup, the volumes can then be recovered.

Solaris/VxVM - Recovering VxVM disk group : Import failed as Duplicate record in configuration

Issue

How to recover VERITAS Volume Manager configuration information while disk group import failed as Duplicate record in configuration

Error

# vxdg -C import oradg
VxVM vxdg ERROR V-5-1-587 Disk group oradg: import failed: No valid disk found containing disk group

When I check in message file I am able to see , its having duplicate record

Jan 18 13:54:53 blr-svr-modbb-01 vxvm:vxconfigd: [ID 592144 daemon.error]       Disk Disk_5, copy 1: Block 1: Duplicate record in configuration

Jan 18 13:54:53 blr-svr-modbb-01 vxvm:vxconfigd: [ID 351026 daemon.error]       Disk Disk_4, copy 1: Block 1: Duplicate record in configuration

Solution

The error indicates that all configuration copies have become corrupt ( due to disk failures, system panic, writing on the disk by an application, the administrator, or bugs in VxVM )

1. Check all the disks included in the disk group oradg

# vxdisk -o alldgs list
DEVICE       TYPE            DISK         GROUP        STATUS
Disk_0       auto:none       -            -            online invalid
Disk_1       auto:none       -            -            online invalid
Disk_2       auto:none       -            -            online invalid
Disk_4       auto:cdsdisk    -           (oradg)       online
Disk_5       auto:cdsdisk    -           (oradg)       online
Disk_6       auto:cdsdisk    dbbackup01   dbbackup     online
c1t0d0s2     auto:sliced     rootdg01     rootdg       online
c1t2d0s2     auto:sliced     rootdg02     rootdg       online

# vxdisk -e alldgs list
DEVICE       TYPE      DISK         GROUP        STATUS       OS_NATIVE_NAME  
Disk_0       auto      -             -            online       c8t202500A0B8476FE0d1s2
Disk_1       auto      -             -            online       c9t201400A0B8476FE0d3s2
Disk_2       auto      -             -            online       c8t202500A0B8476FE0d31s2
Disk_4       auto      -             -            online       c9t201400A0B8476FE0d2s2
Disk_5       auto      -             -            online       c9t201400A0B8476FE0d0s2
Disk_6       auto      dbbackup01    dbbackup     online       c8t202500A0B8476FE0d4s2
c1t0d0s2     auto      rootdg01      rootdg       online       c1t0d0s2   
c1t2d0s2     auto      rootdg02      rootdg       online       c1t2d0s2   

2. Dump the configuration information of the private region to file by using vxprivutil

# /etc/vx/diag.d/vxprivutil dumpconfig /dev/dsk/c9t201400A0B8476FE0d2s2 > /tmp/Disk_4.out# /etc/vx/diag.d/vxprivutil dumpconfig /dev/dsk/c9t201400A0B8476FE0d0s2 > /tmp/Disk_5.out

3. Check the extracted configuration to see if it appears valid. In some cases, there will be duplicate Volume Manager objects that will need to be removed

# cat /tmp/Disk_4.config | vxprint -D - -ht
VxVM vxprint ERROR V-5-1-326 duplicate record, context:         plex data3-01
# cat /tmp/Disk_5.config | vxprint -D - -ht
VxVM vxprint ERROR V-5-1-326 duplicate record, context:         plex data3-01

The configuration information is throwing error stating its having a duplicate record for plex : data3-01

#  cat /tmp/Disk_4.config | sort +1 | awk '{ print $2 }' | uniq -c | grep data3-01
   2 data3-01
#  cat /tmp/Disk_5.config | sort +1 | awk '{ print $2 }' | uniq -c | grep data3-01
   2 data3-01

# cat -n /tmp/Disk_4.config |  grep data3-01
   192  plex data3-01
   576  plex data3-01

# cat -n /tmp/Disk_5.config |  grep data3-01
   192  plex data3-01
   576  plex data3-01

# cat -n /tmp/Disk_4.config |  grep data3
   192  plex data3-01
   521  vol  data3
   576  plex data3-01

# cat -n /tmp/Disk_5.config |  grep data3
   192  plex data3-01
   521  vol  data3
   576  plex data3-01

So the reason for disk group not getting imported is because of the duplicate entry of plex data3-01.

Working Configuration Of oradg

Problematic Configuration Of oradg Comparing the output able to see duplicate entry for plex data3-01

#Config copy 01 #Header nblocks=5120 blksize=128 hdrsize=512 #flags=0x100 (CLEAN) #version: 4/12 #dgname: oradg  dgid: 1341762102.12.blr-svr-modba-01 #config: tid=0.1620 nstpool=0 nrvg=0 nrlink=0 ncache=0 nvol=4 nplex=4 nsd=7 ndm=2 nda=0 #pending: tid=0.1620 nstpool=0 nrvg=0 nrlink=0 ncache=0 nvol=4 nplex=4 nsd=7 ndm=2 nda=0 # #Block    5: flag=0    ref=82   offset=0    frag_size=92  #Block    7: flag=0    ref=63   offset=0    frag_size=104 #Block    9: flag=0    ref=63   offset=104  frag_size=19  #Block   12: flag=0    ref=64   offset=0    frag_size=104 #Block   15: flag=0    ref=64   offset=104  frag_size=27  #Block   20: flag=0    ref=41   offset=0    frag_size=76  #Block   22: flag=0    ref=42   offset=0    frag_size=76  #Block   25: flag=0    ref=43   offset=0    frag_size=76  #Block   28: flag=0    ref=44   offset=0    frag_size=76  #Block   37: flag=0    ref=678  offset=0    frag_size=104 #Block   38: flag=0    ref=678  offset=104  frag_size=6   #Block   40: flag=0    ref=679  offset=0    frag_size=65  #Block   42: flag=0    ref=680  offset=0    frag_size=104 #Block   43: flag=0    ref=320  offset=0    frag_size=76  #Block   44: flag=0    ref=680  offset=104  frag_size=8   #Block   46: flag=0    ref=681  offset=0    frag_size=67  #Block   48: flag=0    ref=682  offset=0    frag_size=104 #Block   49: flag=0    ref=682  offset=104  frag_size=8   #Block   51: flag=0    ref=683  offset=0    frag_size=67  #Block   53: flag=0    ref=328  offset=0    frag_size=76  #Block   54: flag=0    ref=684  offset=0    frag_size=104 #Block   55: flag=0    ref=684  offset=104  frag_size=8   #Block   57: flag=0    ref=685  offset=0    frag_size=67  #Block   64: flag=0    ref=142  offset=0    frag_size=67  #Block   65: flag=0    ref=337  offset=0    frag_size=76  # #Record   41: flags=0    gen_flags=0x4  size=76 #Blocks: 20 sd   oradg01-01   comment="   putil0="   putil1="   putil2="   dm_offset=0   pl_offset=0   len=104857600   update_tid=0.1435   rid=0.1033   guid={ca6eb15e-1dd1-11b2-a1ad-00144f23119f}   plex_rid=0.1031   dm_rid=0.1026   minor=0   detach_tid=0.0   column=0   mkdevice=off   subvolume=off   subcache=off   stale=off   kdetach=off   relocate=off #Record   42: flags=0    gen_flags=0x4  size=76 #Blocks: 22 sd   oradg01-02   comment="   putil0="   putil1="   putil2="   dm_offset=104857600   pl_offset=0   len=209715200   update_tid=0.1435   rid=0.1040   guid={d10d984a-1dd1-11b2-a1ad-00144f23119f}   plex_rid=0.1038   dm_rid=0.1026   minor=0   detach_tid=0.0   column=0   mkdevice=off   subvolume=off   subcache=off   stale=off   kdetach=off   relocate=off #Record   43: flags=0    gen_flags=0x4  size=76 #Blocks: 25 sd   oradg01-03   comment="   putil0="   putil1="   putil2="   dm_offset=314572800   pl_offset=0   len=314572800   update_tid=0.1435   rid=0.1047   guid={d5a1ec3a-1dd1-11b2-a1ad-00144f23119f}   plex_rid=0.1045   dm_rid=0.1026   minor=0   detach_tid=0.0   column=0   mkdevice=off   subvolume=off   subcache=off   stale=off   kdetach=off   relocate=off #Record   44: flags=0    gen_flags=0x4  size=76 #Blocks: 28 sd   oradg01-04   comment="   putil0="   putil1="   putil2="   dm_offset=629145600   pl_offset=0   len=733935360   update_tid=0.1435   rid=0.1054   guid={dd5fab2e-1dd1-11b2-a1ad-00144f23119f}   plex_rid=0.1052   dm_rid=0.1026   minor=0   detach_tid=0.0   column=0   mkdevice=off   subvolume=off   subcache=off   stale=off   kdetach=off   relocate=off #Record   63: flags=0    gen_flags=0x4  size=123 #Blocks: 7 9 dm   oradg00   comment="   putil0="   putil1="   putil2="   diskid=1340346031.13.blr-svr-mocrm-01   last_diskid=1340346031.13.blr-svr-mocrm-01   last_da_name=Disk_4   rid=0.1026   guid={6a645ac6-1dd2-11b2-a1ac-00144f23119f}   allocator_reserved=off   removed=off   spare=off   failing=off   update_tid=0.1571   last_da_dev=118/80   last_disk_offset=2304   ssbid=0.0 #Record   64: flags=0    gen_flags=0x4  size=131 #Blocks: 12 15 dm   oradg02   comment="   putil0="   putil1="   putil2="   diskid=1350993299.30.blr-svr-modba-01   last_diskid=1350993299.30.blr-svr-modba-01   last_da_name=Disk_5   rid=0.1273   guid={84a4fa00-1dd2-11b2-a1ac-00144f23119f}   allocator_reserved=off   removed=off   spare=off   failing=off   update_tid=0.1571   last_da_dev=118/96   last_disk_offset=2304   ssbid=0.65 #Record   82: flags=0    gen_flags=0x4  size=92 #Blocks: 5 dg   oradg   comment="   putil0="   putil1="   putil2="   dgid=1341762102.12.blr-svr-modba-01   rid=0.1025   update_tid=0.1027   nconfig=default   nlog=default   base_minor=59000   version=120   cds=on   alignment=16   last_platform=SOLARIS   maxdev=32767 #Record  320: flags=0    gen_flags=0x4  size=76 #Blocks: 43 sd   oradg02-01   comment="   putil0="   putil1="   putil2="   dm_offset=0   pl_offset=209715200   len=157286400   update_tid=0.1277   rid=0.1276   guid={424410b4-1dd2-11b2-a1ad-00144f23119f}   plex_rid=0.1038   dm_rid=0.1273   minor=0   detach_tid=0.0   column=0   mkdevice=off   subvolume=off   subcache=off   stale=off   kdetach=off   relocate=off #Record  328: flags=0    gen_flags=0x4  size=76 #Blocks: 53 sd   oradg02-02   comment="   putil0="   putil1="   putil2="   dm_offset=157286400   pl_offset=314572800   len=262144000   update_tid=0.1280   rid=0.1279   guid={77ce2210-1dd2-11b2-a1ad-00144f23119f}   plex_rid=0.1045   dm_rid=0.1273   minor=0   detach_tid=0.0   column=0   mkdevice=off   subvolume=off   subcache=off   stale=off   kdetach=off   relocate=off #Record  337: flags=0    gen_flags=0x4  size=76 #Blocks: 65 sd   oradg02-03   comment="   putil0="   putil1="   putil2="   dm_offset=419430400   pl_offset=733935360   len=102828288   update_tid=0.1283   rid=0.1282   guid={b7abe796-1dd1-11b2-a1ae-00144f23119f}   plex_rid=0.1052   dm_rid=0.1273   minor=0   detach_tid=0.0   column=0   mkdevice=off   subvolume=off   subcache=off   stale=off   kdetach=off   relocate=off #Record  678: flags=0    gen_flags=0x4  size=110 #Blocks: 37 38 vol  u01   use_   fs   putil0="   putil1="   putil2="   state="CLEAN   writeback=on   writecopy=off   specify_writecopy=off   pl_num=1   start_opts="   read_pol=SELECT   minor=59000   user=root   group=root   mode=0600   log_   len=104857600   log_len=0   update_tid=0.1620   rid=0.1029   detach_tid=0.0   active=off   forceminor=off   badlog=off   recover_checkpoint=0   sd_num=0   sdnum=0   kdetach=off   storage=off   layered=off   apprecover=off   recover_seqno=0   recov_id=0   primary_datavol=   vvr_tag=0   morph=off   guid={ca6e7cca-1dd1-11b2-a1ad-00144f23119f}   inst_invalid=off   incomplete=off   instant=off   restore=off   snap_after_restore=off   iscachevol=off   oldlog=off   no   norecov=off   logmap_align=0   logmap_len=0   inst_src_guid={00000000-0000-0000-0000-000000000000}   cascaded=off #Record  679: flags=0    gen_flags=0x4  size=65 #Blocks: 40 plex u01-01   comment="   putil0="   putil1="   putil2="   layout=CONCAT   sd_num=1   state="CLEAN   log_sd=   update_tid=0.1620   rid=0.1031   vol_rid=0.1029   detach_tid=0.0   log=off   noerror=off   kdetach=off   stale=off   ncolumn=0   raidlog=off   guid={ca6e9d5e-1dd1-11b2-a1ad-00144f23119f}   mapguid={00000000-0000-0000-0000-000000000000} #Record  680: flags=0    gen_flags=0x4  size=112 #Blocks: 42 44 vol  data1   use_   fs   putil0="   putil1="   putil2="   state="CLEAN   writeback=on   writecopy=off   specify_writecopy=off   pl_num=1   start_opts="   read_pol=SELECT   minor=59001   user=root   group=root   mode=0600   log_   len=367001600   log_len=0   update_tid=0.1620   rid=0.1036   detach_tid=0.0   active=off   forceminor=off   badlog=off   recover_checkpoint=0   sd_num=0   sdnum=0   kdetach=off   storage=off   layered=off   apprecover=off   recover_seqno=0   recov_id=0   primary_datavol=   vvr_tag=0   morph=off   guid={d10d74a0-1dd1-11b2-a1ad-00144f23119f}   inst_invalid=off   incomplete=off   instant=off   restore=off   snap_after_restore=off   iscachevol=off   oldlog=off   no   norecov=off   logmap_align=0   logmap_len=0   inst_src_guid={00000000-0000-0000-0000-000000000000}   cascaded=off #Record  681: flags=0    gen_flags=0x4  size=67 #Blocks: 46 plex data1-01   comment="   putil0="   putil1="   putil2="   layout=CONCAT   sd_num=2   state="CLEAN   log_sd=   update_tid=0.1620   rid=0.1038   vol_rid=0.1036   detach_tid=0.0   log=off   noerror=off   kdetach=off   stale=off   ncolumn=0   raidlog=off   guid={d10d858a-1dd1-11b2-a1ad-00144f23119f}   mapguid={00000000-0000-0000-0000-000000000000} #Record  682: flags=0    gen_flags=0x4  size=112 #Blocks: 48 49 vol  data2   use_   fs   putil0="   putil1="   putil2="   state="CLEAN   writeback=on   writecopy=off   specify_writecopy=off   pl_num=1   start_opts="   read_pol=SELECT   minor=59002   user=root   group=root   mode=0600   log_   len=576716800   log_len=0   update_tid=0.1620   rid=0.1043   detach_tid=0.0   active=off   forceminor=off   badlog=off   recover_checkpoint=0   sd_num=0   sdnum=0   kdetach=off   storage=off   layered=off   apprecover=off   recover_seqno=0   recov_id=0   primary_datavol=   vvr_tag=0   morph=off   guid={d5a1c926-1dd1-11b2-a1ad-00144f23119f}   inst_invalid=off   incomplete=off   instant=off   restore=off   snap_after_restore=off   iscachevol=off   oldlog=off   no   norecov=off   logmap_align=0   logmap_len=0   inst_src_guid={00000000-0000-0000-0000-000000000000}   cascaded=off #Record  683: flags=0    gen_flags=0x4  size=67 #Blocks: 51 plex data2-01   comment="   putil0="   putil1="   putil2="   layout=CONCAT   sd_num=2   state="CLEAN   log_sd=   update_tid=0.1620   rid=0.1045   vol_rid=0.1043   detach_tid=0.0   log=off   noerror=off   kdetach=off   stale=off   ncolumn=0   raidlog=off   guid={d5a1d9ca-1dd1-11b2-a1ad-00144f23119f}   mapguid={00000000-0000-0000-0000-000000000000} #Record  684: flags=0    gen_flags=0x4  size=112 #Blocks: 54 55 vol  data3   use_   fs   putil0="   putil1="   putil2="   state="CLEAN   writeback=on   writecopy=off   specify_writecopy=off   pl_num=1   start_opts="   read_pol=SELECT   minor=59003   user=root   group=root   mode=0600   log_   len=836763648   log_len=0   update_tid=0.1620   rid=0.1050   detach_tid=0.0   active=off   forceminor=off   badlog=off   recover_checkpoint=0   sd_num=0   sdnum=0   kdetach=off   storage=off   layered=off   apprecover=off   recover_seqno=0   recov_id=0   primary_datavol=   vvr_tag=0   morph=off   guid={dd5f881a-1dd1-11b2-a1ad-00144f23119f}   inst_invalid=off   incomplete=off   instant=off   restore=off   snap_after_restore=off   iscachevol=off   oldlog=off   no   norecov=off   logmap_align=0   logmap_len=0   inst_src_guid={00000000-0000-0000-0000-000000000000}   cascaded=off #Record  685: flags=0    gen_flags=0x4  size=67 #Blocks: 57 plex data3-01   comment="   putil0="   putil1="   putil2="   layout=CONCAT   sd_num=2   state="CLEAN   log_sd=   update_tid=0.1620   rid=0.1052   vol_rid=0.1050   detach_tid=0.0   log=off   noerror=off   kdetach=off   stale=off   ncolumn=0   raidlog=off   guid={dd5f98c8-1dd1-11b2-a1ad-00144f23119f}   mapguid={00000000-0000-0000-0000-000000000000}

#Config copy 01 #Header nblocks=5120 blksize=128 hdrsize=512 #flags=0x100 (CLEAN) #version: 4/12 #dgname: oradg  dgid: 1341762102.12.blr-svr-modba-01 #config: tid=0.1620 nstpool=0 nrvg=0 nrlink=0 ncache=0 nvol=4 nplex=4 nsd=7 ndm=2 nda=0 #pending: tid=0.1620 nstpool=0 nrvg=0 nrlink=0 ncache=0 nvol=4 nplex=4 nsd=7 ndm=2 nda=0 # #Block    5: flag=0    ref=82   offset=0    frag_size=92  #Block    7: flag=0    ref=63   offset=0    frag_size=104 #Block    9: flag=0    ref=63   offset=104  frag_size=19  #Block   12: flag=0    ref=64   offset=0    frag_size=104 #Block   15: flag=0    ref=64   offset=104  frag_size=27  #Block   20: flag=0    ref=41   offset=0    frag_size=76  #Block   22: flag=0    ref=42   offset=0    frag_size=76  #Block   25: flag=0    ref=43   offset=0    frag_size=76  #Block   28: flag=0    ref=44   offset=0    frag_size=76  #Block   37: flag=0    ref=678  offset=0    frag_size=104 #Block   38: flag=0    ref=678  offset=104  frag_size=6   #Block   40: flag=0    ref=679  offset=0    frag_size=65  #Block   42: flag=0    ref=680  offset=0    frag_size=104 #Block   43: flag=0    ref=320  offset=0    frag_size=76  #Block   44: flag=0    ref=680  offset=104  frag_size=8   #Block   46: flag=0    ref=681  offset=0    frag_size=67  #Block   48: flag=0    ref=682  offset=0    frag_size=104 #Block   49: flag=0    ref=682  offset=104  frag_size=8   #Block   51: flag=0    ref=683  offset=0    frag_size=67  #Block   53: flag=0    ref=328  offset=0    frag_size=76  #Block   54: flag=0    ref=684  offset=0    frag_size=104 #Block   55: flag=0    ref=684  offset=104  frag_size=8   #Block   57: flag=0    ref=685  offset=0    frag_size=67  #Block   64: flag=0    ref=142  offset=0    frag_size=67  #Block   65: flag=0    ref=337  offset=0    frag_size=76  # #Record   41: flags=0    gen_flags=0x4  size=76 #Blocks: 20 sd   oradg01-01   comment="   putil0="   putil1="   putil2="   dm_offset=0   pl_offset=0   len=104857600   update_tid=0.1435   rid=0.1033   guid={ca6eb15e-1dd1-11b2-a1ad-00144f23119f}   plex_rid=0.1031   dm_rid=0.1026   minor=0   detach_tid=0.0   column=0   mkdevice=off   subvolume=off   subcache=off   stale=off   kdetach=off   relocate=off #Record   42: flags=0    gen_flags=0x4  size=76 #Blocks: 22 sd   oradg01-02   comment="   putil0="   putil1="   putil2="   dm_offset=104857600   pl_offset=0   len=209715200   update_tid=0.1435   rid=0.1040   guid={d10d984a-1dd1-11b2-a1ad-00144f23119f}   plex_rid=0.1038   dm_rid=0.1026   minor=0   detach_tid=0.0   column=0   mkdevice=off   subvolume=off   subcache=off   stale=off   kdetach=off   relocate=off #Record   43: flags=0    gen_flags=0x4  size=76 #Blocks: 25 sd   oradg01-03   comment="   putil0="   putil1="   putil2="   dm_offset=314572800   pl_offset=0   len=314572800   update_tid=0.1435   rid=0.1047   guid={d5a1ec3a-1dd1-11b2-a1ad-00144f23119f}   plex_rid=0.1045   dm_rid=0.1026   minor=0   detach_tid=0.0   column=0   mkdevice=off   subvolume=off   subcache=off   stale=off   kdetach=off   relocate=off #Record   44: flags=0    gen_flags=0x4  size=76 #Blocks: 28 sd   oradg01-04   comment="   putil0="   putil1="   putil2="   dm_offset=629145600   pl_offset=0   len=733935360   update_tid=0.1435   rid=0.1054   guid={dd5fab2e-1dd1-11b2-a1ad-00144f23119f}   plex_rid=0.1052   dm_rid=0.1026   minor=0   detach_tid=0.0   column=0   mkdevice=off   subvolume=off   subcache=off   stale=off   kdetach=off   relocate=off #Record   63: flags=0    gen_flags=0x4  size=123 #Blocks: 7 9 dm   oradg00   comment="   putil0="   putil1="   putil2="   diskid=1340346031.13.blr-svr-mocrm-01   last_diskid=1340346031.13.blr-svr-mocrm-01   last_da_name=Disk_4   rid=0.1026   guid={6a645ac6-1dd2-11b2-a1ac-00144f23119f}   allocator_reserved=off   removed=off   spare=off   failing=off   update_tid=0.1571   last_da_dev=118/80   last_disk_offset=2304   ssbid=0.0 #Record   64: flags=0    gen_flags=0x4  size=131 #Blocks: 12 15 dm   oradg02   comment="   putil0="   putil1="   putil2="   diskid=1350993299.30.blr-svr-modba-01   last_diskid=1350993299.30.blr-svr-modba-01   last_da_name=Disk_5   rid=0.1273   guid={84a4fa00-1dd2-11b2-a1ac-00144f23119f}   allocator_reserved=off   removed=off   spare=off   failing=off   update_tid=0.1571   last_da_dev=118/96   last_disk_offset=2304   ssbid=0.65 #Record   82: flags=0    gen_flags=0x4  size=92 #Blocks: 5 dg   oradg   comment="   putil0="   putil1="   putil2="   dgid=1341762102.12.blr-svr-modba-01   rid=0.1025   update_tid=0.1027   nconfig=default   nlog=default   base_minor=59000   version=120   cds=on   alignment=16   last_platform=SOLARIS   maxdev=32767 #Record  142: flags=0    gen_flags=0x4  size=67 #Blocks: 64 plex data3-01   comment="   putil0="   putil1="   putil2="   layout=CONCAT   sd_num=2   state="CLEAN   log_sd=   update_tid=0.1623   rid=0.1052   vol_rid=0.1050   detach_tid=0.0   log=off   noerror=off   kdetach=off   stale=off   ncolumn=0   raidlog=off   guid={dd5f98c8-1dd1-11b2-a1ad-00144f23119f}   mapguid={00000000-0000-0000-0000-000000000000} #Record  320: flags=0    gen_flags=0x4  size=76 #Blocks: 43 sd   oradg02-01   comment="   putil0="   putil1="   putil2="   dm_offset=0   pl_offset=209715200   len=157286400   update_tid=0.1277   rid=0.1276   guid={424410b4-1dd2-11b2-a1ad-00144f23119f}   plex_rid=0.1038   dm_rid=0.1273   minor=0   detach_tid=0.0   column=0   mkdevice=off   subvolume=off   subcache=off   stale=off   kdetach=off   relocate=off #Record  328: flags=0    gen_flags=0x4  size=76 #Blocks: 53 sd   oradg02-02   comment="   putil0="   putil1="   putil2="   dm_offset=157286400   pl_offset=314572800   len=262144000   update_tid=0.1280   rid=0.1279   guid={77ce2210-1dd2-11b2-a1ad-00144f23119f}   plex_rid=0.1045   dm_rid=0.1273   minor=0   detach_tid=0.0   column=0   mkdevice=off   subvolume=off   subcache=off   stale=off   kdetach=off   relocate=off #Record  337: flags=0    gen_flags=0x4  size=76 #Blocks: 65 sd   oradg02-03   comment="   putil0="   putil1="   putil2="   dm_offset=419430400   pl_offset=733935360   len=102828288   update_tid=0.1283   rid=0.1282   guid={b7abe796-1dd1-11b2-a1ae-00144f23119f}   plex_rid=0.1052   dm_rid=0.1273   minor=0   detach_tid=0.0   column=0   mkdevice=off   subvolume=off   subcache=off   stale=off   kdetach=off   relocate=off #Record  678: flags=0    gen_flags=0x4  size=110 #Blocks: 37 38 vol  u01   use_   fs   putil0="   putil1="   putil2="   state="CLEAN   writeback=on   writecopy=off   specify_writecopy=off   pl_num=1   start_opts="   read_pol=SELECT   minor=59000   user=root   group=root   mode=0600   log_   len=104857600   log_len=0   update_tid=0.1620   rid=0.1029   detach_tid=0.0   active=off   forceminor=off   badlog=off   recover_checkpoint=0   sd_num=0   sdnum=0   kdetach=off   storage=off   layered=off   apprecover=off   recover_seqno=0   recov_id=0   primary_datavol=   vvr_tag=0   morph=off   guid={ca6e7cca-1dd1-11b2-a1ad-00144f23119f}   inst_invalid=off   incomplete=off   instant=off   restore=off   snap_after_restore=off   iscachevol=off   oldlog=off   no   norecov=off   logmap_align=0   logmap_len=0   inst_src_guid={00000000-0000-0000-0000-000000000000}   cascaded=off #Record  679: flags=0    gen_flags=0x4  size=65 #Blocks: 40 plex u01-01   comment="   putil0="   putil1="   putil2="   layout=CONCAT   sd_num=1   state="CLEAN   log_sd=   update_tid=0.1620   rid=0.1031   vol_rid=0.1029   detach_tid=0.0   log=off   noerror=off   kdetach=off   stale=off   ncolumn=0   raidlog=off   guid={ca6e9d5e-1dd1-11b2-a1ad-00144f23119f}   mapguid={00000000-0000-0000-0000-000000000000} #Record  680: flags=0    gen_flags=0x4  size=112 #Blocks: 42 44 vol  data1   use_   fs   putil0="   putil1="   putil2="   state="CLEAN   writeback=on   writecopy=off   specify_writecopy=off   pl_num=1   start_opts="   read_pol=SELECT   minor=59001   user=root   group=root   mode=0600   log_   len=367001600   log_len=0   update_tid=0.1620   rid=0.1036   detach_tid=0.0   active=off   forceminor=off   badlog=off   recover_checkpoint=0   sd_num=0   sdnum=0   kdetach=off   storage=off   layered=off   apprecover=off   recover_seqno=0   recov_id=0   primary_datavol=   vvr_tag=0   morph=off   guid={d10d74a0-1dd1-11b2-a1ad-00144f23119f}   inst_invalid=off   incomplete=off   instant=off   restore=off   snap_after_restore=off   iscachevol=off   oldlog=off   no   norecov=off   logmap_align=0   logmap_len=0   inst_src_guid={00000000-0000-0000-0000-000000000000}   cascaded=off #Record  681: flags=0    gen_flags=0x4  size=67 #Blocks: 46 plex data1-01   comment="   putil0="   putil1="   putil2="   layout=CONCAT   sd_num=2   state="CLEAN   log_sd=   update_tid=0.1620   rid=0.1038   vol_rid=0.1036   detach_tid=0.0   log=off   noerror=off   kdetach=off   stale=off   ncolumn=0   raidlog=off   guid={d10d858a-1dd1-11b2-a1ad-00144f23119f}   mapguid={00000000-0000-0000-0000-000000000000} #Record  682: flags=0    gen_flags=0x4  size=112 #Blocks: 48 49 vol  data2   use_   fs   putil0="   putil1="   putil2="   state="CLEAN   writeback=on   writecopy=off   specify_writecopy=off   pl_num=1   start_opts="   read_pol=SELECT   minor=59002   user=root   group=root   mode=0600   log_   len=576716800   log_len=0   update_tid=0.1620   rid=0.1043   detach_tid=0.0   active=off   forceminor=off   badlog=off   recover_checkpoint=0   sd_num=0   sdnum=0   kdetach=off   storage=off   layered=off   apprecover=off   recover_seqno=0   recov_id=0   primary_datavol=   vvr_tag=0   morph=off   guid={d5a1c926-1dd1-11b2-a1ad-00144f23119f}   inst_invalid=off   incomplete=off   instant=off   restore=off   snap_after_restore=off   iscachevol=off   oldlog=off   no   norecov=off   logmap_align=0   logmap_len=0   inst_src_guid={00000000-0000-0000-0000-000000000000}   cascaded=off #Record  683: flags=0    gen_flags=0x4  size=67 #Blocks: 51 plex data2-01   comment="   putil0="   putil1="   putil2="   layout=CONCAT   sd_num=2   state="CLEAN   log_sd=   update_tid=0.1620   rid=0.1045   vol_rid=0.1043   detach_tid=0.0   log=off   noerror=off   kdetach=off   stale=off   ncolumn=0   raidlog=off   guid={d5a1d9ca-1dd1-11b2-a1ad-00144f23119f}   mapguid={00000000-0000-0000-0000-000000000000} #Record  684: flags=0    gen_flags=0x4  size=112 #Blocks: 54 55 vol  data3   use_   fs   putil0="   putil1="   putil2="   state="CLEAN   writeback=on   writecopy=off   specify_writecopy=off   pl_num=1   start_opts="   read_pol=SELECT   minor=59003   user=root   group=root   mode=0600   log_   len=836763648   log_len=0   update_tid=0.1620   rid=0.1050   detach_tid=0.0   active=off   forceminor=off   badlog=off   recover_checkpoint=0   sd_num=0   sdnum=0   kdetach=off   storage=off   layered=off   apprecover=off   recover_seqno=0   recov_id=0   primary_datavol=   vvr_tag=0   morph=off   guid={dd5f881a-1dd1-11b2-a1ad-00144f23119f}   inst_invalid=off   incomplete=off   instant=off   restore=off   snap_after_restore=off   iscachevol=off   oldlog=off   no   norecov=off   logmap_align=0   logmap_len=0   inst_src_guid={00000000-0000-0000-0000-000000000000}   cascaded=off #Record  685: flags=0    gen_flags=0x4  size=67 #Blocks: 57 plex data3-01   comment="   putil0="   putil1="   putil2="   layout=CONCAT   sd_num=2   state="CLEAN   log_sd=   update_tid=0.1620   rid=0.1052   vol_rid=0.1050   detach_tid=0.0   log=off   noerror=off   kdetach=off   stale=off   ncolumn=0   raidlog=off   guid={dd5f98c8-1dd1-11b2-a1ad-00144f23119f}   mapguid={00000000-0000-0000-0000-000000000000}

4. The veritas volume manager maintains a backup of Disk group & disk information. So from the location /etc/vx/cbr/bk , use the last good known configuration

# cd /etc/vx/cbr/bk
# cd oradg.1341762102.12.blr-svr-modba-01

# cat 1341762102.12.blr-svr-modba-01.cfgrec | vxprint -D - -ht
Disk group: oradg

DG NAME         NCONFIG      NLOG     MINORS   GROUP-ID
ST NAME         STATE        DM_CNT   SPARE_CNT         APPVOL_CNT
DM NAME         DEVICE       TYPE     PRIVLEN  PUBLEN   STATE
RV NAME         RLINK_CNT    KSTATE   STATE    PRIMARY  DATAVOLS  SRL
RL NAME         RVG          KSTATE   STATE    REM_HOST REM_DG    REM_RLNK
CO NAME         CACHEVOL     KSTATE   STATE
VT NAME         NVOLUME      KSTATE   STATE
V  NAME         RVG/VSET/CO  KSTATE   STATE    LENGTH   READPOL   PREFPLEX UTYPE
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE
SC NAME         PLEX         CACHE    DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
DC NAME         PARENTVOL    LOGVOL
SP NAME         SNAPVOL      DCO

dg oradg        default      default  24000    1390069991.36.blr-svr-modba-01

dm oradg00      Disk_4       auto     2048     1365178112 -
dm oradg02      Disk_5       auto     2048     629110528 -

v  data1        -            ENABLED  ACTIVE   367001600 SELECT   -        fsgen
pl data1-01     data1        ENABLED  ACTIVE   367001600 CONCAT   -        RW
sd oradg01-02   data1-01     oradg00  104857600 209715200 0       Disk_4   ENA
sd oradg02-01   data1-01     oradg02  0        157286400 209715200 Disk_5  ENA

v  data2        -            ENABLED  ACTIVE   576716800 SELECT   -        fsgen
pl data2-01     data2        ENABLED  ACTIVE   576716800 CONCAT   -        RW
sd oradg01-03   data2-01     oradg00  314572800 314572800 0       Disk_4   ENA
sd oradg02-02   data2-01     oradg02  157286400 262144000 314572800 Disk_5 ENA

v  data3        -            ENABLED  ACTIVE   836763648 SELECT   -        fsgen
pl data3-01     data3        ENABLED  ACTIVE   836763648 CONCAT   -        RW
sd oradg01-04   data3-01     oradg00  629145600 733935360 0       Disk_4   ENA
sd oradg02-03   data3-01     oradg02  419430400 102828288 733935360 Disk_5 ENA

v  u01          -            ENABLED  ACTIVE   104857600 SELECT   -        fsgen
pl u01-01       u01          ENABLED  ACTIVE   104857600 CONCAT   -        RW
sd oradg01-01   u01-01       oradg00  0        104857600 0        Disk_4   ENA

# cat 1341762102.12.blr-svr-modba-01.cfgrec | vxprint -D - -mvphsr > /tmp/final.out

So now we are actually going to start the recovery procedure

5. Zero out the private region

# vxdisk -f init Disk_4
# vxdisk -f init Disk_5

6. Initialize the diskgroup with the same name and add the other disks into the diskgroup

# vxdg init oradg oradg00=Disk_4 oradg02=Disk_5

7. Use the output from dumpconfig as input to vxprint to make a copy that vxmake can use

# cd /etc/vx/cbr/bk
# cd oradg.1341762102.12.blr-svr-modba-01
# cat 1341762102.12.blr-svr-modba-01.cfgrec | vxprint -D - -mvphsr > /tmp/final.out

8. Use the configuration information saved in /tmp/final.out as input to vxmake

# vxmake -g oradg -d /tmp/final.out
# vxrecover -g oradg -sbE

# vxprint -g oradg -ht
DG NAME         NCONFIG      NLOG     MINORS   GROUP-ID
ST NAME         STATE        DM_CNT   SPARE_CNT         APPVOL_CNT
DM NAME         DEVICE       TYPE     PRIVLEN  PUBLEN   STATE
RV NAME         RLINK_CNT    KSTATE   STATE    PRIMARY  DATAVOLS  SRL
RL NAME         RVG          KSTATE   STATE    REM_HOST REM_DG    REM_RLNK
CO NAME         CACHEVOL     KSTATE   STATE
VT NAME         NVOLUME      KSTATE   STATE
V  NAME         RVG/VSET/CO  KSTATE   STATE    LENGTH   READPOL   PREFPLEX UTYPE
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE
SC NAME         PLEX         CACHE    DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
DC NAME         PARENTVOL    LOGVOL
SP NAME         SNAPVOL      DCO

dg oradg        default      default  24000    1390069991.36.blr-svr-modba-01

dm oradg00      Disk_4       auto     2048     1365178112 -
dm oradg02      Disk_5       auto     2048     629110528 -

v  data1        -            ENABLED  ACTIVE   367001600 SELECT   -        fsgen
pl data1-01     data1        ENABLED  ACTIVE   367001600 CONCAT   -        RW
sd oradg01-02   data1-01     oradg00  104857600 209715200 0       Disk_4   ENA
sd oradg02-01   data1-01     oradg02  0        157286400 209715200 Disk_5  ENA

v  data2        -            ENABLED  ACTIVE   576716800 SELECT   -        fsgen
pl data2-01     data2        ENABLED  ACTIVE   576716800 CONCAT   -        RW
sd oradg01-03   data2-01     oradg00  314572800 314572800 0       Disk_4   ENA
sd oradg02-02   data2-01     oradg02  157286400 262144000 314572800 Disk_5 ENA

v  data3        -            ENABLED  ACTIVE   836763648 SELECT   -        fsgen
pl data3-01     data3        ENABLED  ACTIVE   836763648 CONCAT   -        RW
sd oradg01-04   data3-01     oradg00  629145600 733935360 0       Disk_4   ENA
sd oradg02-03   data3-01     oradg02  419430400 102828288 733935360 Disk_5 ENA

v  u01          -            ENABLED  ACTIVE   104857600 SELECT   -        fsgen
pl u01-01       u01          ENABLED  ACTIVE   104857600 CONCAT   -        RW
sd oradg01-01   u01-01       oradg00  0        104857600 0        Disk_4   ENA
#

The below steps are optional, In case if the volumes & plex are not in Enabled & Active state, set the it to "clean" so that the volumes can be started:# vxvol -g  <diskgroup> init  clean  <vol_name>  <plex_name>

# vxvol -g  oradg  init  clean  u01  u01-01
# vxvol -g  oradg init  clean  data1  data1-01
# vxvol -g  oradg init  clean  data2  data2-01
# vxvol -g  oradg init  clean  data3  data3-01

Start the volumes

# vxvol -g  <diskgroup_name>  -o  delayrecover  start  <vol_name>

# vxvol -g  oradg  -o  delayrecover  start  u01
# vxvol -g  oradg  -o  delayrecover  start  data1
# vxvol -g  oradg  -o  delayrecover  start  data2
# vxvol -g  oradg  -o  delayrecover  start  data3

9. Mount the file systems on the started volumes (fsck may be needs to be run) and then verify

# fsck -F vxfs /dev/vx/rdsk/<diskgroup_name>/<vol_name>

# mount -F vxfs /dev/vx/dsk/<diskgroup_name>/<vol_name> <mount_point>

# df -k <mount_point>

# cat /etc/vfstab
#device device  mount   FS      fsck    mount   mount
#to     mount   to      fsck            point           type    pass    at boot options
#                      
fd      -       /dev/fd fd      -       no      -
/proc   -       /proc   proc    -       no      -
/dev/vx/dsk/bootdg/swapvol      -       -       swap    -       no      nologging
/dev/vx/dsk/bootdg/rootvol      /dev/vx/rdsk/bootdg/rootvol     /       ufs     1       no      nologging
#/dev/vx/dsk/bootdg/globaldevices       /dev/vx/rdsk/bootdg/globaldevices       /globaldevices  ufs     1       yes     -
/devices        -       /devices        devfs   -       no      -
sharefs -       /etc/dfs/sharetab       sharefs -       no      -
ctfs    -       /system/contract        ctfs    -       no      -
objfs   -       /system/object  objfs   -       no      -
swap    -       /tmp    tmpfs   -       yes     -
#NOTE:  volume  rootvol () encapsulated partition c1t0d0s0
#NOTE:  volume  swapvol () encapsulated partition c1t0d0s1
#NOTE:  volume  globaldevices   () encapsulated partition c1t0d0s3
/dev/vx/dsk/bootdg/globaldevices        /dev/vx/rdsk/bootdg/globaldevices       /global/.devices/node@1 ufs     2       no      global
/dev/vx/dsk/oradg/u01   /dev/vx/rdsk/oradg/u01  /u01    vxfs    2       no      -      
/dev/vx/dsk/oradg/data1 /dev/vx/rdsk/oradg/data1        /data1  vxfs    2       no      -
/dev/vx/dsk/oradg/data2 /dev/vx/rdsk/oradg/data2        /data2  vxfs    2       no      -
/dev/vx/dsk/oradg/data3 /dev/vx/rdsk/oradg/data3        /data3  vxfs    2       no      -
/dev/vx/dsk/dbbackup/vol_backup /dev/vx/rdsk/dbbackup/vol_backup        /backup vxfs 2 no -

# mount /data1
# mount /data2
# mount /data3
# mount /u01

# df -h
Filesystem             size   used  avail capacity  Mounted on
/dev/vx/dsk/bootdg/rootvol
                        34G   7.8G    26G    23%    /
/devices                 0K     0K     0K     0%    /devices
ctfs                     0K     0K     0K     0%    /system/contract
proc                     0K     0K     0K     0%    /proc
mnttab                   0K     0K     0K     0%    /etc/mnttab
swap                    91G   2.2M    91G     1%    /etc/svc/volatile
objfs                    0K     0K     0K     0%    /system/object
sharefs                  0K     0K     0K     0%    /etc/dfs/sharetab
/platform/sun4u-us3/lib/libc_psr/libc_psr_hwcap2.so.1
                        34G   7.8G    26G    23%    /platform/sun4u-us3/lib/libc_psr.so.1
/platform/sun4u-us3/lib/sparcv9/libc_psr/libc_psr_hwcap2.so.1
                        34G   7.8G    26G    23%    /platform/sun4u-us3/lib/sparcv9/libc_psr.so.1
fd                       0K     0K     0K     0%    /dev/fd
swap                    91G   328K    91G     1%    /tmp
swap                    91G    64K    91G     1%    /var/run
swap                    91G     0K    91G     0%    /dev/vx/dmp
swap                    91G     0K    91G     0%    /dev/vx/rdmp
/dev/vx/dsk/bootdg/globaldevices                      1002M   3.9M   938M     1%    /global/.devices/node@1
/dev/vx/dsk/rootdg/globaldevice                       961M   3.8M   900M     1%    /global/.devices/node@2
/dev/vx/dsk/oradg/data1                       175G   135G    38G    79%    /data1
/dev/vx/dsk/oradg/data2                       275G   121G   144G    46%    /data2
/dev/vx/dsk/oradg/data3                       399G   211G   177G    55%    /data3
/dev/vx/dsk/oradg/u01                        50G    15G    33G    31%    /u01