Wednesday, May 18, 2016

Migration of Native Solaris Zone (NGZ) to Kernel Zone (KZ)/Logical Domain (LDOM) – Part 1

In my previous post I mentioned, why I am looking to convert a Native Zone into a Kernel Zone/LDOM.
A decision to migration a Native Solaris Zone (NGZ) to a LDOM is not a direct one, but a stepped one.
A NZG needs to be first migrated to a KZ and then this KZ needs to me migrated to a LDOM.
I have spent a lot of time on the web to find if anyone has done this before. I couldn’t find anything. So I took it upon myself to do this task and document the procedure for the benefit of some poor souls who has to undergo the same.
Please note: the below procedure is using Solaris x86/64 on VMware workstation. But it will remain more or less similar on a SPARC system as well.

This post is broken up in 3 parts - due to sheer size on the post itself - plus, to avoid getting "lost in translation"

This posts will be as below:

(1) Part 1: Setting up the infrastructure. I will be using VMware workstation. But it will be be the same for VirtualBox as well. In this post, I will create the physical VM - which will serve as a global zone and create a Native Zone.
(2) Part 2: Will cover steps to convert Native Zone to Kernel zone.
(3) Part 3: will Cover steps to convert Kernel one to LDOM or a seperate VM on VMware or VirtualBox (as in my case)

Post Links as below:

Part 1: http://solariscat.blogspot.com/2016/05/migration-of-native-solaris-zone-ngz-to.html

Part 2: http://solariscat.blogspot.com/2016/05/migration-of-native-solaris-zone-ngz-to_18.html

Part 3: http://solariscat.blogspot.com/2016/05/migration-of-native-solaris-zone-ngz-to_19.html


We will now see how to go about this procedure.
Points to consider:
(1)    DHCP setup is a must – at least for the purpose of the migration.
(2)    The server on which the repository is configured, MUST have a fixed IP.
(3)    The Repository must be published over http and not filesystem/directory.
(4)    Storage space is required for facilitating the migration. This is the space where the images are stored and should be available to the physical servers on which the zones are backed-up as well as migrated to.
(5)    A private link between the global zone and the Kernel Zone is preferred to avoid network latency.

Initial System setup – (to emulate a Box with NGZ)

1.       Create a VM with Solaris 11.2/11.3 along with its repository.

¾     Start VMWare Workstation – As you can see there are no VMs existing.image
¾     Click “file ->
image

image
image
image
image
image
image

¾     Change this to:
image

image

¾     Change this to:
image

image

¾     Change this to:
image

image
image
image
image
image
image

¾     Click Finish
image

¾     Click “Edit Virtual Machine Settings”
image
¾     Click “Add”

image
image
image


¾     Add a new 25 GB LUN. This will host the Repository

image
image
image

¾     Click Add
image
image
image

¾     Add a new 25 GB LUN. This will host the Zones
image

image
image


¾     Click Add
image
image
image




¾     Add a new 50 GB LUN. This will host the Backups

image
image
image

Now you have created 4 Disks in the VM –
1.       for OS – 16 GB
2.       for Repository – 25GB
3.       for Zones – 25 GB
4.       for Backups – 50 GB

Map the ISO image of the OS to the CD/DVD to install the OS.
image

¾     Change this to:
image
¾     Select the ISO image location and file
image
image

¾     Select “Power on”

¾     OS Installation Starts
image
image
image
image
image
image
image

¾     Select the 1st disk 16GB for OS installation
image
image

¾     Change this to:
image
image
¾     Change this to:
image
image
image
¾     Change this to:
image
image
image
image
image
image
image
image
¾     Type root password of your choice – twice
image
image
image
image
image

¾     In the meantime, you need to make some changes in the “Virtual Network Editor”
image
image

Check VMnet1 details. It’s NOT matching with our IP address that we chose for our server.

Our Server IP: 192.9.200.10, whereas,
VMnet1 IP network: 10.10.1.0

Change VMnet1 to 192.9.200.0 Also ensure DHCP is “DISABLED”

¾     Click on change settings
image
¾     Click on VMnet1
image

Change Subnet IP:
From: 10.10.1.0
To: 192.9.200.0

image
image
¾     Click “Apply” & “OK”

¾     OS Installation complete
image

¾     Now Reboot

image
image
image
¾     Now our Physical Server is up & ready.

¾     Lets activate network access to the system & Check if the VM is pingable
image

Login as root and disable Sendmail to avoid those annoying messages coming on the screen. Alternatively, you can all a FQDN to the host file to work around the issue.

¾     To disable Sendmail
# svcadm disable Sendmail
# svcadm disable Sendmail-client

¾     To implement FQDN:
Edit the /etc/hosts file and add an alias as below
image
image
image

¾     Now, activate direct root SSH access. It’s faster to use putty than using the console.
image

¾     Change this to:
image

¾     Restart ssh services
image

¾     Now login as root to 192.9.200.10 using putty
image

image

…And, We are in!!!


¾     Now let’s prepare the repository, zones & backup pools

¾     Create the pools as per the plan
image
image

¾     Optional: Enable compression to save diskspace on your laptop. I have observed that enabling compression actually makes the systems faster…much faster.
image
image

¾     Prepare the Repository:

Download the repository files from the below location and extract it in /Repo filesystem

image

¾     Check existing default repository

root@SolA:~# pkg publisher
PUBLISHER                   TYPE     STATUS P LOCATION
solaris                     origin   online F http://pkg.oracle.com/solaris/release/
root@SolA:~#


¾     Change it to the newly created repository created under /Repo filesystem

root@SolA:~# pkg unset-publisher solaris
Updating package cache                           1/1
root@SolA:~#

root@SolA:~# pkg publisher
PUBLISHER                   TYPE     STATUS P LOCATION
root@SolA:~#

root@SolA:~# pkg set-publisher -g /Repo/Sol11.3_repo/ solaris
root@SolA:~#
root@SolA:~# pkg set-publisher -g /Repo/Sol11.3_SRU/ solaris
root@SolA:~#

root@SolA:~# pkg publisher
PUBLISHER                   TYPE     STATUS P LOCATION
solaris                     origin   online F file:///Repo/Sol11.3_SRU/
solaris                     origin   online F file:///Repo/Sol11.3_repo/
root@SolA:~#

¾     verify repository version and status
¾     Compare it with your existing OS version

root@SolA:~# pkg info entire
          Name: entire
       Summary: Incorporation to lock all system packages to the same build
   Description: This package constrains system package versions to the same
                build.  WARNING: Proper system update and correct package
                selection depend on the presence of this incorporation.
                Removing this package will result in an unsupported system.
      Category: Meta Packages/Incorporations
         State: Installed
     Publisher: solaris
       Version: 0.5.11 (Oracle Solaris 11.3.1.5.0)
 Build Release: 5.11
        Branch: 0.175.3.1.0.5.0
Packaging Date: October  6, 2015 02:00:51 PM
          Size: 5.46 kB
          FMRI: pkg://solaris/entire@0.5.11,5.11-0.175.3.1.0.5.0:20151006T140051Z
root@SolA:~#


root@SolA:~# pkg info -r entire
          Name: entire
       Summary: entire incorporation including Support Repository Update (Oracle Solaris 11.3.7.5.0).
   Description: This package constrains system package versions to the same
                build.  WARNING: Proper system update and correct package
                selection depend on the presence of this incorporation.
                Removing this package will result in an unsupported system.  For
                more information see:
                https://support.oracle.com/rs?type=doc&id=2045311.1
      Category: Meta Packages/Incorporations
         State: Not installed
     Publisher: solaris
       Version: 0.5.11 (Oracle Solaris 11.3.7.5.0)
 Build Release: 5.11
        Branch: 0.175.3.7.0.5.0
Packaging Date: April 13, 2016 01:57:22 PM
          Size: 5.46 kB
          FMRI: pkg://solaris/entire@0.5.11,5.11-0.175.3.7.0.5.0:20160413T135722Z
root@SolA:~#

As you can see in the above outputs, the Repo version is higher than the OS version.

¾     Update the OS to the latest version using the repository

root@SolA:~# pkg update --accept
------------------------------------------------------------
Package: pkg://solaris/consolidation/osnet/osnet-incorporation@0.5.11,5.11-0.175.3.7.0.5.0:20160413T135024Z
License: lic_OTN

You acknowledge that your use of this Oracle Solaris software product
is subject to, and may not exceed the use for which you are authorized,
(i) the license or cloud services terms that you accepted when you
obtained the right to use Oracle Solaris software; or (ii) the license
terms that you agreed to when you placed your Oracle Solaris software
order with Oracle; or (iii) the Oracle Solaris software license terms
included with the hardware that you acquired from Oracle; or, if (i),
(ii) or (iii) are not applicable, then, (iv) the OTN License Agreement
for Oracle Solaris (which you acknowledge you have read and agree to)
available at
http://www.oracle.com/technetwork/licenses/solaris-cluster-express-license-167852.html.
Note: Software downloaded for trial use or downloaded as replacement
media may not be used to update any unsupported software.

           Packages to install:   2
            Packages to update: 217
       Create boot environment: Yes
Create backup boot environment:  No

DOWNLOAD                                PKGS         FILES    XFER (MB)   SPEED
Completed                            219/219   11898/11898  387.1/387.1    0B/s

PHASE                                          ITEMS
Removing old actions                       1437/1437
Installing new actions                     2433/2433
Updating modified actions                11925/11925
Updating package state database                 Done
Updating package cache                       217/217
Updating image state                            Done
Creating fast lookup database                   Done
Updating package cache                           1/1

A clone of solaris exists and has been updated and activated.
On the next boot the Boot Environment solaris-1 will be
mounted on '/'.  Reboot when ready to switch to this updated BE.

Updating package cache                           1/1

---------------------------------------------------------------------------
NOTE: Please review release notes posted at:

http://www.oracle.com/pls/topic/lookup?ctx=solaris11&id=SERNS
---------------------------------------------------------------------------


¾     Reboot the system and compare the versions once again

root@SolA:~# pkg info entire
          Name: entire
       Summary: entire incorporation including Support Repository Update (Oracle Solaris 11.3.7.5.0).
   Description: This package constrains system package versions to the same
                build.  WARNING: Proper system update and correct package
                selection depend on the presence of this incorporation.
                Removing this package will result in an unsupported system.  For
                more information see:
                https://support.oracle.com/rs?type=doc&id=2045311.1
      Category: Meta Packages/Incorporations
         State: Installed
     Publisher: solaris
       Version: 0.5.11 (Oracle Solaris 11.3.7.5.0)
 Build Release: 5.11
        Branch: 0.175.3.7.0.5.0
Packaging Date: April 13, 2016 01:57:22 PM
          Size: 5.46 kB
          FMRI: pkg://solaris/entire@0.5.11,5.11-0.175.3.7.0.5.0:20160413T135722Z
root@SolA:~#

root@SolA:~# pkg info -r entire
          Name: entire
       Summary: entire incorporation including Support Repository Update (Oracle Solaris 11.3.7.5.0).
   Description: This package constrains system package versions to the same
                build.  WARNING: Proper system update and correct package
                selection depend on the presence of this incorporation.
                Removing this package will result in an unsupported system.  For
                more information see:
                https://support.oracle.com/rs?type=doc&id=2045311.1
      Category: Meta Packages/Incorporations
         State: Installed
     Publisher: solaris
       Version: 0.5.11 (Oracle Solaris 11.3.7.5.0)
 Build Release: 5.11
        Branch: 0.175.3.7.0.5.0
Packaging Date: April 13, 2016 01:57:22 PM
          Size: 5.46 kB
          FMRI: pkg://solaris/entire@0.5.11,5.11-0.175.3.7.0.5.0:20160413T135722Z
root@SolA:~#



As you can see above, the OS version installed and the Repo versions are matching.

Please proceed to Part 2: http://solariscat.blogspot.com/2016/05/migration-of-native-solaris-zone-ngz-to_18.html

Actual Steps for converting the Native Zone to a Kernel Zone is covered here

Tuesday, May 17, 2016

Solaris 11.2/11.3 – IBM Guardium & Heartaches!!!

In my current environment, I have many Solaris boxes. The boxes which were installed many years ago are running LDOMs. But some of these boxes are on Solaris zones – both Native as well as kernel.
These zones/LDOMs are running a host of databases + applications. To monitor these databases, we have a software called Guardium (Now owned IBM). IBM Infosphere Guardium is an appliance, which requires an STAP agent to be installed on the boxes that it needs to be monitored.
The features that IBM Guardium provides is probably excellent. But, trust me, it’s a very unstable product. To make matters worse, the pdf document that are packaged inside the Guardium installation tool is soooo (not a typo) outdated, that it nowhere reflects the actual installation procedure.
Guardium STAP agent, upon installation, gets embedded into the Solaris kernel and starts automatically during system boot-up. So far, so good. On the downside, Guardium agent is so unpredictable that it crashes, most of the times without any specific reason.
Because the Guardium STAP agent is embedded into the kernel, it panics the entire box and hangs the system. This has been acknowledged by IBM – snapshot below:

image

image

If you don’t believe me, check the case “modified date” at the bottom right-hand side.
The most incredible part is the implementation methodology. LDOMS/kernel zone implementation is different from Native zones.
In LDOMs/Kernel zone, the Guardium STAP agent is installed in the LDOM/Kernel zone.
But in case of native zone, the Guardium agent is installed in the global zone.
So, imagine the situation where the system is running hundreds of native zones with Guardium installed/configured on the global zone and the Guardium crashes taking the entire global zone as well as all the Native zones straight to hell and back!!!
Crash Dump Analysis of the system reveals:

crash file:     /sr/3-12417908495/uc/vmdump.1_57700696/this/vmcore.1
release:        5.11 (64-bit)
version:        11.2
usr/src:       36224:29488d54d94a:0.175.2.15.0.4.0:S11.2SRU15.4+0
usr/closed:    2817:a6f8db7e698d:0.175.2.15.0.4.0:S11.2SRU15.4+0
machine:        sun4v
node name:      prctldom3
hw_provider:    Oracle Corporation
system type:    ORCL,SPARC64-X (SPARC64-X+)
hostid:         9007170b
dump_conflags:  0x10100 (DUMP_KERNEL|DUMP_ZFS) on /dev/zvol/dsk/rpool/dump(63.1G)
cluster_bflgs:  0x3 (CLUSTER_CONFIGURED|CLUSTER_BOOTED)
pxfs_software_mount_level: v1 (consolidated version)
current node:   1
dump_uuid:      d8abf8af-255f-4fae-a757-b366ee3d621a
time of crash:  Fri Mar 25 20:13:05 UTC 2016
age of system:  2 hours 8 minutes 15 seconds
panic CPU:      49 (40 CPUs, 126G memory, 2 nodes)
panic string:   BAD TRAP: type=31 rp=2a100643080 addr=643380 mmu_fsr=0 occurred in module "genunix" due to an illegal access to a user address

sanity checks: settings...
NOTE: /etc/system: module ge not loaded for "set ge:ge_intr_mode=0x833"
vmem...CPU...
WARNING: 3 CPUs have 6 threads in their dispatch queues
sysent...
WARNING: unknown module ktap_71327 seen 10 times in sysent table
clock...misc...
WARNING: 23 severe kstat errors (run "kstat xck")
NOTE: system has 6 non-default processor sets (2 CPUs in cp_default)
NOTE: system has 6 non-global zones
NOTE: system has 6 non-default CPU pools
NOTE: system has 1 non-default project
disks...
WARNING: 1 disk is busy with 1 command pending (run "dev busy").
NOTE: user_reserve_hint_pct is set to 80% (99.2G)
done
CAT(vmcore.1/11V)>


CAT(vmcore.1/11V)> panic
panic on CPU 49
panic string:   BAD TRAP: type=31 rp=2a100643080 addr=643380 mmu_fsr=0 occurred in module "genunix" due to an illegal access to a user address
==== panic user (LWP_SYS) thread: 0xc401c6b856c0  PID: 11123  on CPU: 49 ====
cmd: /usr/lib/dbus-daemon --system
fmri: svc:/system/dbus:default
t_procp: 0xc401bf654010
  p_as: 0xc4012c1c7800  size: 3874816  RSS: 2596864
     a_hat: 0xc401bf3ee940
     cnum: CPU48:1/16239
     cpusran: 48,49
  p_zone: 0xc401c1229880 (prdbaml1)
t_stk: 0x2a100643290  sp: 0x2050aa21  t_stkbase: 0x2a10063a000
t_pri: 59 (TS)  pctcpu: 0.006347
t_transience: 2  t_wkld_flags: 0
t_lwp: 0xc40129267ea0  t_tid: 1
  machpcb: 0x2a100643290
  lwp_ap:   0x643380
  t_mstate: LMS_SYSTEM  ms_prev: LMS_KFAULT
  ms_state_start: 0.000660480 seconds earlier
  ms_start: 4.460619520 seconds earlier
t_cpupart: 0xc4012260ea80(6)  t_bind_pset: 6  last CPU: 49
idle: 4459386880 hrticks (4.459386880s)
start: Fri Mar 25 20:13:01 2016
age: 4 seconds (4 seconds)
t_state:     TS_ONPROC
t_flag:      0x1800 (T_PANIC|T_LWPREUSE)
t_proc_flag: 0x104 (TP_TWAIT|TP_MSACCT)
t_schedflag: 0x13 (TS_LOAD|TS_DONT_SWAP|TS_SIGNALLED)
t_acflag:    3 (TA_NO_PROCESS_LOCK|TA_BATCH_TICKS)
p_flag:      0x42000400 (SZONETOP|SMSACCT|SMSFORK)

pc:      unix:panicsys+0x40:   call     unix:setjmp

void unix:panicsys+0x40((const char *)0x10101b38, (va_list)0x2a100642e48, (struct regs *)0x2050b3d0, (int)1, 0x9900001605, , , , , , , , 0x10101b38, 0x2a100642e48)
unix:vpanic_common+0x78(0x10101b38, 0x2a100642e48, 0x213e7, 0x21467, 0x2a100643120, 0x1600)
void unix:panic+0x1c((const char *)0x10101b38, (void *)0x31, 0x2a100643080, 0x643380, 0, 0x20834cc8, 0x10101ba8, ...)
int unix:die+0x7c((unsigned)0x31, (struct regs *)0x2a100643080, (caddr_t)0x643380, (uint_t)0)
void unix:trap+0xabc((struct regs *)0x2a100643080, (caddr_t)0x643380, (uint32_t), (uint32_t))
unix:ktl0+0x64()
-- trap data  type: 0x31 (data access MMU miss)  rp: 0x2a100643080  --
addr: 0x643380
pc:  0x1012af24 genunix:auditsys+0x24:   ldx    [%i0], %i2
npc: 0x1012af28 genunix:auditsys+0x28:   subcc    %i2, 0x1d, %g0  ( cmp   %i2, 0x1d )
global:                       %g1               0xba
       %g2                  0  %g3         0x1011fb70
       %g4             0x1740  %g5         0x20843b78
       %g6                  0  %g7     0xc401c6b856c0
out:  %o0           0x100000  %o1                  2
       %o2         0xff356840  %o3         0xff1b2a40
       %o4                  1  %o5     0xc401bf654010
       %sp      0x2a100642921  %o7         0x1011ee28
loc:  %l0                  0  %l1                  1
       %l2         0xff3ee354  %l3                  2
       %l4               0xff  %l5         0xff1b2a40
       %l6                  0  %l7                  1
in:   %i0           0x643380  %i1      0x2a100643288
       %i2    0xa2aa025b027ff  %i3         0x20840438
       %i4                  2  %i5               0x30
       %fp      0x2a1006429d1  %i7         0x1011fbc8
<trap>int genunix:auditsys+0x24((struct auditcalls *)0x643380, (rval_t *)0x2a100643288)
int64_t genunix:syscall_ap+0x58()
unix:_syscall_no_proc_exit32+0x78()
-- switch to user thread's user stack --

CAT(vmcore.1/11V)>


CAT(vmcore.1/11V)> findstk
building sp list...found 4 stacks
==== stack @ 0x2a100643080 (sp: 0x2a100642881) ====
NULL(genunix:lwp_getdatamodel?)()
struct sysent *genunix:lwp_getsysent+4((klwp_t *))
genunix:auditsys((struct auditcalls *)0x643380, (rval_t *)0x2a100643288) - frame recycled
int64_t genunix:syscall_ap+0x58()
unix:_syscall_no_proc_exit32+0x78()
-- switch to user panic_thread's user stack --

==== stack @ 0x2a100642d10 (sp: 0x2a100642511) ====
NULL(unix:panic?)()
int unix:die+0x7c((unsigned)0x31, (struct regs *)0x2a100643080, (caddr_t)0x643380, (uint_t)0)
void unix:trap+0xabc((struct regs *)0x2a100643080, (caddr_t)0x643380, (uint32_t), (uint32_t))
unix:ktl0+0x64()
-- trap data  type: 0x31 (data access MMU miss)  rp: 0x2a100643080  --
pc:  0x1012af24 genunix:auditsys+0x24:   ldx    [%i0], %i2
npc: 0x1012af28 genunix:auditsys+0x28:   subcc    %i2, 0x1d, %g0  ( cmp   %i2, 0x1d )
 

### Very definitely a user space address rather than a kernel one. 


global:                       %g1               0xba
       %g2                  0  %g3         0x1011fb70
       %g4             0x1740  %g5         0x20843b78
       %g6                  0  %g7     0xc401c6b856c0
out:  %o0           0x100000  %o1                  2
       %o2         0xff356840  %o3         0xff1b2a40
       %o4                  1  %o5     0xc401bf654010
       %sp      0x2a100642921  %o7         0x1011ee28
loc:  %l0                  0  %l1                  1
       %l2         0xff3ee354  %l3                  2
       %l4               0xff  %l5         0xff1b2a40
       %l6                  0  %l7                  1
in:   %i0           0x643380  %i1      0x2a100643288
       %i2    0xa2aa025b027ff  %i3         0x20840438
       %i4                  2  %i5               0x30
       %fp      0x2a1006429d1  %i7         0x1011fbc8
<trap>int genunix:auditsys+0x24((struct auditcalls *)0x643380, (rval_t *)0x2a100643288)
int64_t genunix:syscall_ap+0x58()
unix:_syscall_no_proc_exit32+0x78()
-- switch to user panic_thread's user stack --

==== stack @ 0x2a100642b80 (sp: 0x2a100642381) ====
NULL(genunix:anon_private?)()
faultcode_t genunix:segvn_faultpage+0x83c((struct hat *)0x3100642000, (struct seg *)0x2a100642e80, (caddr_t)0xb, (u_offset_t)0x2000, (struct vpage *)0x2000, (page_t **)0x643380, (uint_t), (enum fault_type), (enum seg_rw), (int))
faultcode_t genunix:segvn_fault+0xb28((struct hat *)0x3100642000, (struct seg *), (caddr_t), (size_t)0x643380, (enum fault_type), (enum seg_rw))
-- error reading next frame @ 0x0 --

==== stack @ 0x2a1006425c0 (sp: 0x2a100641dc1) ====
NULL(genunix:segvn_faultpage?)()
faultcode_t genunix:segvn_fault+0xb28((struct hat *)0xc401bf3ee940, (struct seg *)0, (caddr_t)3, (size_t)0xc401bf3ee940, (enum fault_type)0x20118c00, (enum seg_rw)2)
faultcode_t genunix:as_fault+0x3f0((struct hat *), (struct as *)0x4003ee796fc, (caddr_t)0x2013ac10, (size_t)0x40000000001, (enum fault_type), (enum seg_rw))
-- error reading next frame @ 0x0 --

CAT(vmcore.1/11V)>

CAT(vmcore.1/11V)> modinfo | grep ktap
263 LI 0xc400c6e552c0 0x10a3be90 0x6cb58 1 ktap_71327 (guard_tap driver v9.0 64-bit) >>>>>>>>>>> IBM driver
CAT(vmcore.1/11V)>
 

The above stack closely matches the following Bug
Bug 22319453  - 11.2 SRU 12.4 T5-4 panic in get_syscall_args due to invalid lwp_
ap



Conclusion:
---------------------------

Update from the Bug

***************
All third party modules which does their own syscall interposing can hit this panic due to new variable padded in lwp structure in middle.
IBM ktap and veritas modules are hit this issue till now and they have recompile there drivers once upgrading post 11.2 sru 8

Solaris 11.2.8.4.0 and Later Releases may Cause a System Panic for Systems Using Third Party Kernel Drivers (
Doc ID 2111676.1 )
***************

Also we have a KM document related to this issue.
Solaris 11.2.8.4.0 and Later Releases may Cause a System Panic for Systems Using Third Party Kernel Drivers (
Doc ID 2111676.1
) 

Oracle DocID: 2111676.1 reveals:
image

That leaves me with 2 viable options on my table:
1.       Throw the Guardium boxes out of the window
2.       Convert native zone to kernel zone or LDOMs to minimize exposure to instability & panics.

Currently, till we reach an agreement with management regarding (1), I guess I will have to work on (2).
Will post my “Native zones conversion to kernel zones/LDOMs” plan soon.

Update - 19 May 2016:
Created a detailed series of posts to achieve the "native Zone to Kernel Zone to LDOM" conversion.
Links as below:

Let me know how it worked for you,
regards
Sandeep

Tuesday, April 12, 2016

How to reset/recover Integrated Lights Out Manager (ILOM) password

The default user and password of ILOM is “root/changeme”, but if you have changed the password already and due for some reason you forgot the password, here are the step by step procedure to recover the ILOM password.

First of all, try to change the password with ipmitool, but if it’s still doesn’t work, try below step:

Notes:
– You must be physically present at the server to perform this procedure.
– This procedure uses the default user account to enable you to recover a lost password or to re-create the root user account.
– You cannot change or delete the default user account.

1. Connect to ILOM via serial console and log in using the default user account.

SUNSP-xxxxxxxx login: default

Press and release the physical presence button. Press return when this is completed…

 

2. Prove physical presence at your server.
Press and release the physical presence button.

The Physical Presence button on the Sun SPARC Enterprise T5xxx servers and X-Series is the Locator button:

clip_image001[6]

But, for Sparc T3/T4 model, the physical presence are on rear side, except for T3-1b/T4-1b:
SPARC T3-1/T4-1: (rear) pin-hole to the left of the USB ports
SPARC T3-2/T4-2: (rear) pin-hole to the left of NET0
SPARC T3-4/T4-4: (rear) to the right of OK LED, above the USB port
SPARC T3-1b/T4-1b: (front) Locate button/Physical Presence (White LED)

3. Return to your serial console and press Enter.

You will be prompted for a password.

4. Type the password for the default user account: defaultpassword

5. Reset the account password or re-create the root account.

-> set /SP/users/root password
Enter new password: ********
Enter new password again: ********


6. try login with your new root password

 

Tuesday, March 22, 2016

Oracle/OVM–OVM Server UUID Challenges

 

While installation of the Oracle OVM server, it automatically selects a “random” number to setup a “UUID” – a unique ID number for itself. It is THIS UUID that the OVM Manager detects and registers within itself for all further communication between the OVM Manager and the OVM Server.

 

Having said that, did you land yourself into a situation wherein:

a.       The OVM server refuses to boot successfully?

b.      The OVM server does boot, but the VMs doesn’t appear?

c.       The OVM manager is unable to communicate with the OVM server?

 

What had happened just before you land up in the above scenarios?

1.       Motherboard of the OVM Server was replaced?

2.       NIC of the OVM server was replaced?

 

Any of the above two scenarios can land you up in to the above 3 situations J

Fear not, below is the reason why and how to get out of it. J

 

Every OVM Server receives a unique ID (also called UUID) when the agent starts. Without this ID, a server can’t join a cluster and the manager can’t dispatch commands to the agent.

 

But the million Dollar Question: “How does this UUID get set in the first place? From where does the OVM Server decides where to pick up the UUID from?

 

What I discovered is this:

 

The ovs-agent will do the following:

 

1.       Ask SMBIOS for an id and if it’s valid use it.

2.       If not valid or no ID then, Get the MAC address of physical Ethernet devices on the system and concatenate in an UUID

 

This poses a great risk because if for some reason you change the motherboard (other SMBIOS) or 1 of the network devices on the system (other MAC), you’ll have a new UUID.

 

Now, how do we detect a changed UUID?

 

The following errors points to a changed UUID:

 

1.       Unable to join cluster after reboot

2.       “Unable to send notification” messages on the console

3.       The OVM Manager fails to rediscover the server after reboot (tries 5 times and gives up)

4.       The OVM Server in OVM Manager will become in ERROR state with an error that looks like: “The server has changed IP or is unreachable”

 

 

Solution?? – Fake it…. I mean, FakeUUID it!!!

In order to solve this issue, the agent has an ini parameter you can set to ensure the same UUID all the time. You can find it in the file /etc/ovs-agent/agent.ini. Get the server UUID from OVM Manager and set it in this parameter.

 

[Server]
fakeuuid=34:45:4c:4c:5a:00:10:20:80:59:b5:c0:4a:42:35:45

 

I personally set this on every OVM installation that I do to ensure there are no issues with this ID changing.

Sunday, March 20, 2016

Solaris/Linux: Emails & attachments

 

“mailx” is the utility to send emails from Solaris to the outside world. Emails can be sent in many formats:

 

(1)   Contents of file in email as text:

# mailx -s 'Subject-Here' abc@xyz.com < input.file

# mailx -s 'Uptime Report' abc@xyz.com < /tmp/output.txt

 

Where,

-s ‘Subject’: Specify subject on command line.

abc@xyz.com: To email user.

/tmp/output.txt: Send the content of /tmp/output.txt file using mail command.

 

(2)   Attaching a file as an “attachment” to the email

Example: Attach /tmp/list.tar.gz and and send it

# uuencode /tmp/list.tar.gz /tmp/list.tar.gz | mailx -s "Reports" abc@xyz.com

 

(3)   Attaching a file as an “attachment” and contents of a file as body of the email

Example: Email photo.png along with a text message read from body.txt

# (cat body.txt; uuencode photo.png photo.png) | mailx -s "Subject" abc@xyz.com

 

(4)   Attaching multiple files as “attachments”

Example: Email files - file1.doc, file2.doc & file3.doc – to a single email.

# (uuencode file1.doc file1.doc; uuencode file2.doc file2.doc; uuencode file3.doc file3.doc) | mailx -s "Subject" abc@xyz.com

 

Above commands can be scripted to send emails from the system:

1.      As alerts to the system admins, on adhoc – on-event-basis.

2.      As Periodic/scheduled emails – performance reports, healthcheck reports

3.      On need basis