Documentation by: bleonard
I'm not sure how well known it is that Solaris 11 contains a load balancer. The official documentation, starting with the Integrated Load Balancer Overview, does a great job of explaining this feature. In this blog entry my goal is to provide an implementation example.
For starters, I will be using the HALF-NAT operation mode. Basically, HALF-NAT means that the client's IP address is not mapped so that the servers know the real client address. This is usually preferred for server logging (see ILB Operation Modes for more). I will load balance traffic across 2 zones, each running the Apache Tomcat server. The load balancer itself will be configured as a multi-homed zone. The configuration will look as follows:
Step 1: Create the VNICs
The first step is to create VNICs for all of these interfaces:
#dladm create-vnic -l e1000g0 ilb0
#dladm create-vnic -l e1000g0 ilb1
#dladm create-vnic -l e1000g0 server1
#dladm create-vnic -l e1000g0 server2
Step 2: Create the Zones:
If you don't already have a file system for your zones:
#zfs create -o mountpoint=/zones rpool/zones
Then create the ILB zones:
#zonecfg -z ilb-zone
ilb-zone: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:ilb-zone> create
zonecfg:ilb-zone> set zonepath=/zones/ilb-zone
zonecfg:ilb-zone> set ip-type=exclusive
zonecfg:ilb-zone> add net
zonecfg:ilb-zone:net> set physical=ilb0
zonecfg:ilb-zone:net> end
zonecfg:ilb-zone> add net
zonecfg:ilb-zone:net> set physical=ilb1
zonecfg:ilb-zone:net> end
zonecfg:ilb-zone> verify
zonecfg:ilb-zone> exit
And the server zones (repeat this step for server 2 – changing values where appropriate):
#zonecfg -z server1-zone
server1-zone: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:server1-zone> create
zonecfg:server1-zone> set zonepath=/zones/server1-zone
zonecfg:server1-zone> set ip-type=exclusive
zonecfg:server1-zone> add net
zonecfg:server1-zone:net> set physical=server1
zonecfg:server1-zone:net> end
zonecfg:server1-zone> verify
zonecfg:server1-zone> exit
Step 3: Install the ILB Zone
Then install the ilb-zone (wait to install the server zones as we will just clone this zone):
#zoneadm -z ilb-zone install
A ZFS file system has been created for this zone.
Publisher: Using solaris (http://pkg.oracle.com/solaris/release/ ).
Image: Preparing at /zones/ilb-zone/root.
Cache: Using /var/pkg/download.
Sanity Check: Looking for 'entire' incorporation.
Installing: Core System (output follows)
------------------------------------------------------------
Package: pkg://solaris/consolidation/osnet/osnet-incorporation@0.5.11,5.11-0.151.0.1:20101104T230646Z
License: usr/src/pkg/license_files/lic_OTN
Oracle Technology Network Developer License Agreement
...
Packages to install: 1
Create boot environment: No
DOWNLOAD PKGS FILES XFER (MB)
Completed 1/1 1/1 0.0/0.0
PHASE ACTIONS
Install Phase 11/11
PHASE ITEMS
Package State Update Phase 1/1
Image State Update Phase 2/2
Packages to install: 45
Create boot environment: No
Services to restart: 3
DOWNLOAD PKGS FILES XFER (MB)
Completed 45/45 12511/12511 89.1/89.1
PHASE ACTIONS
Install Phase 17953/17953
PHASE ITEMS
Package State Update Phase 45/45
Image State Update Phase 2/2
Installing: Additional Packages (output follows)
Packages to install: 46
Create boot environment: No
Services to restart: 2
DOWNLOAD PKGS FILES XFER (MB)
Completed 46/46 4498/4498 26.5/26.5
PHASE ACTIONS
Install Phase 6139/6139
PHASE ITEMS
Package State Update Phase 46/46
Image State Update Phase 2/2
Note: Man pages can be obtained by installing SUNWman
Postinstall: Copying SMF seed repository ... done.
Postinstall: Applying workarounds.
Done: Installation completed in 499.617 seconds.
Next Steps: Boot the zone, then log into the zone console (zlogin -C)to complete the configuration process.
I will be using the following sysidcfg file to automate the zone's system configuration. Adjust your values accordingly. The root password is "abc123":
#cat /zones/ilb-zone/root/etc/sysidcfg
system_locale=C
terminal=xterms
network_interface=ilb0 {
primary
hostname=ilb-ext
ip_address=10.0.2.21
netmask=255.255.255.0
default_route=10.0.2.2
protocol_ipv6=no}
network_interface=ilb1 {
hostname=ilb-int
ip_address=192.168.1.21
default_route=NONE
netmask=255.255.255.0
protocol_ipv6=no}
security_policy=none
name_service=NONE
nfs4_domain=dynamic
timezone=US/Eastern
root_password=fto/dU8MKwQRI
Boot and log into the zone:
#zoneadm -z ilb-zone boot
#zlogin -C ilb-zone
[Connected to zone 'ilb-zone' console]
100/100
Hostname: ilb-zone
Loading smf(5) service descriptions: 3/3
network_interface=ilb0 {
ilb0 is not a valid network interface line 3 position 19
Creating new rsa public/private host key pair
Creating new dsa public/private host key pair
Configuring network interface addresses: ilb0 ilb1.
ilb-ext console login: root
Password: abc123
Jul 1 10:54:37 ilb-ext login: ROOT LOGIN /dev/console
Oracle Corporation SunOS 5.11 snv_151a November
2010
root@ilb-ext:~#
Since our ilb-zone has 2 network interfaces, we also want to make sure a packet arriving on one network interface and addressed to a host on a different network is forwarded to the appropriate interface.
root@ilb-ext:~# svcadm enable ipv4-forwarding
Step 4: Install the Serve 1 Zone
We'll create the first server zone as a clone of the ilb-zone. We'll then configure the server 1 zone and clone it to server 2.
Shut down ilb-zone so that it can be cloned:
#zoneadm -z ilb-zone halt
Then clone ilb-zone:
#zoneadm -z server1-zone clone ilb-zone
Here's a sysidcfg file to use with server1-zone:
#cat /zones/server1-zone/root/etc/sysidcfg
system_locale=C
terminal=xterms
network_interface=PRIMARY {
hostname=server1-zone
ip_address=192.168.1.50
netmask=255.255.255.0
default_route=none
protocol_ipv6=no}
security_policy=none
name_service=NONE
nfs4_domain=dynamic
timezone=US/Eastern
root_password=fto/dU8MKwQRI
Then boot and log in to server1-zone:
#zoneadm -z server1-zone boot
Password:
#zlogin -C server1-zone
[Connected to zone 'server1-zone' console]
Hostname: server1-zone
Creating new rsa public/private host key pair
Creating new dsa public/private host key pair
Configuring network interface addresses: server1.
server1-zone console login: root
Password: abc123
Jul 1 14:53:20 server1-zone login: ROOT LOGIN /dev/console
Last login: Fri Jul 1 13:54:37 on console
Oracle Corporation SunOS 5.11 snv_151a November 2010
Also boot back up the ilb zone:
#zoneadm -z ilb-zone boot
Password:
Step 5: Configure Internet Access
Test if you can ping the outside world from within the ilb zone:
root@ilb-ext:~# ping www.oracle.com
ping: unknown host www.oracle.com
Open another terminal window. The new terminal window should have you in the global zone. Copy the/etc/resolve.conf and /etc/[nsswitch.conf files from the global zone to ilb-zone and server1-zone:
#cp /etc/resolv.conf /zones/ilb-zone/root/etc/.
Password:
#cp /etc/nsswitch.conf /zones/ilb-zone/root/etc/.
#cp /etc/resolv.conf /zones/server1-zone/root/etc/.
#cp /etc/nsswitch.conf /zones/server1-zone/root/etc/.
Return to the ilb-zone. You should not be able to reach the outside world:
root@ilb-ext:~# ping www.oracle.com
www.oracle.com is alive
However, server1-zone needs some routing set up before it can reach out as it will route its traffic through the ilb-zone:
root@server1-zone:~# route -p add default 192.168.1.21
add net default: gateway 192.168.1.21
add persistent net default: gateway 192.168.1.21
root@server1-zone:~# ping www.oracle.com
www.oracle.com is alive
Step 6: Install Tomcat
Apache Tomcat will be the service we load balance to:
root@server1-zone:~# pkg install tomcat tomcat-examples runtime/java
Packages to install: 3
Create boot environment: No
Services to restart: 2
DOWNLOAD PKGS FILES XFER (MB)
Completed 3/3 1166/1166 38.9/38.9
PHASE ACTIONS
Install Phase 1504/1504
PHASE ITEMS
Package State Update Phase 3/3
Image State Update Phase 2/2
Loading smf(5) service descriptions: 1/1
root@server1-zone:~# svcadm enable http:tomcat6
Step 7: Configure Routing to the Server Zone
From the global zone we need to be able to reach the server. Add the following route (the -p option makes the changes persistent across network restarts):
#route -p add 192.168.1.0 10.0.2.21
Password:
add net 192.168.1.0: gateway 10.0.2.21
add persistent net 192.168.1.0: gateway 10.0.2.21
And now you should be able to reach Tomcat from the global zone (or any client on that subnet):
Step 8: Cloning the Tomcat Server Zone
Now that we have the Tomcat server running in a zone, we can quickly create another instance. First, we need to shut down the server1-zone:
#zoneadm -z server1-zone halt
Then clone it:
#zoneadm -z server2-zone clone server1-zone
Copy the sysidcfg file you created for server 1 to server 2:
#cp /zones/server1-zone/root/etc/sysidcfg /zones/server2-zone/root/etc/sysidcfg
Then change the hostname and ip_address. This time around we'll also set the default router. Once editing is complete, the file should look as follows:
#cat /zones/server2-zone/root/etc/sysidcfg
system_locale=C
terminal=xterms
network_interface=PRIMARY {
hostname=server2-zone
ip_address=192.168.1.60
netmask=255.255.255.0
default_route=192.168.1.21
protocol_ipv6=no}
security_policy=none
name_service=NONE
nfs4_domain=dynamic
timezone=US/Eastern
root_password=fto/dU8MKwQRI
Then boot and log into the server 2 zone:
#zoneadm -z server2-zone boot
#zlogin -C server2-zone
[Connected to zone 'server2-zone' console]
Hostname: server2-zone
Creating new rsa public/private host key pair
Creating new dsa public/private host key pair
Configuring network interface addresses: server2.
server2-zone console login: root
Password: abc123
Jul 6 10:39:54 server2-zone login: ROOT LOGIN /dev/console
Last login: Fri Jul 1 16:19:45 on console
Oracle Corporation SunOS 5.11 snv_151a November 2010
root@server2-zone:~#
Not only does this 2nd zone install much, much quicker, but Tomcat is already up and running:
It really gives you a feel for how easy it can be to scale using Solaris in a cloud type environment.
Don't forget to boot server1-zone:
#zoneadm -z server1-zone boot
Step 9: Configure Load Balancing
OK, that was a lot of setup just to get to the point of this blog. But now that we have 2 servers running at 2 IP address, let's set up a load balancer to scale traffic between them.
In the ILB zone, install the ILB:
root@ilb-ext:~# pkg install ilb
Packages to install: 1
Create boot environment: No
Services to restart: 1
DOWNLOAD PKGS FILES XFER (MB)
Completed 1/1 11/11 0.2/0.2
PHASE ACTIONS
Install Phase 38/38
PHASE ITEMS
Package State Update Phase 1/1
Image State Update Phase 2/2
Loading smf(5) service descriptions: 1/1
Then enable the ILB service:
root@ilb-ext:~# svcadm enable ilb
Then define a server group:
root@ilb-ext:~# ilbadm create-servergroup -s servers=192.168.1.50:8080,192.168.1.60:8080 tomcatgroup
root@ilb-ext:~# ilbadm show-servergroup
SGNAME SERVERID MINPORT MAXPORT IP_ADDRESS
tomcatgroup _tomcatgroup.0 8080 8080 192.168.1.50
tomcatgroup _tomcatgroup.1 8080 8080 192.168.1.60
Then define a load balancing rule. This is the most complicated part of the process. For starters, I'll try to keep the rule as simple as possible. The rule is enabled (-e), will persist (-p), incoming packets (-i) are matched against destination virtual IP address (vip) and port 10.0.2.20:80. The packet is handled (-m) using round robin (rr). The destination for the packets (-o) is server group tomcatgroup. The rule is calledtomcatrule_rr.
root@ilb-ext:~# ilbadm create-rule -e -p -i vip=10.0.2.20,port=80 -m lbalg=rr,type=HALF-NAT -o servergroup=tomcatgroup tomcatrule_rr
You can view the rule as follows:
root@ilb-ext:~# ilbadm show-rule
RULENAME STATUS LBALG TYPE PROTOCOL VIP PORT
tomcatrule_rr E roundrobin HALF-NAT TCP 10.0.2.20 80
root@ilb-ext:~# ilbadm show-rule -f
RULENAME: tomcatrule_rr
STATUS: E
PORT: 80
PROTOCOL: TCP
LBALG: roundrobin
TYPE: HALF-NAT
PROXY-SRC: --
PMASK: /32
HC-NAME: --
HC-PORT: --
CONN-DRAIN: 0
NAT-TIMEOUT: 120
PERSIST-TIMEOUT: 60
SERVERGROUP: tomcatgroup
VIP: 10.0.2.20
SERVERS: _tomcatgroup.0,_tomcatgroup.1
Finally, we need to tell the outside world that packets destined for our VIP,10.0.2.20, should be sent to ilb0. First, find the MAC address of ilb0:
root@ilb-ext:~# dladm show-vnic ilb0
LINK OVER SPEED MACADDRESS MACADDRTYPE VID
ilb0 ? 1000 2:8:20:bf:2a:d9 random 0
Then resolve the address to interface ilb0 using arp:
root@ilb-ext:~# arp -s 10.0.2.20 2:8:20:bf:2a:d9 pub permanent
You should then be able to ping the VIP:
root@ilb-ext:~# ping 10.0.2.20
10.0.2.20 is alive
Step 10: Load Balance!
You can now point your browser to the virtual IP address and get a result back from one of the Tomcat servers:
Very cool! But from which server was I served? I modified the example snoop.jsp to return the server's hostname and IP Address. Save the snoop.jsp to the/var/tomcat6/webapps/examples/jsp/snp directory in each of your zones.
#cp Downloads/snoop.jsp /zones/server1-zone/root/var/tomcat6/webapps/examples/jsp/snp/.
#cp Downloads/snoop.jsp /zones/server2-zone/root/var/tomcat6/webapps/examples/jsp/snp/.
I've appended the Server Side IP Address section to the bottom of the page,http://10.0.2.20/examples/jsp/snp/snoop.jsp
Step 11: Health Checks
To keep things simple on the first go-around, I avoided health checks. However, it's pointless to have a load balancer that continues to feed requests to a dead server.
Health check options include ping probes, TCP probes, UDP probes and a user-defined script. Since I'm concerned about the health of Tomcat, I've created a simple script:
root@ilb-ext:~# cat /var/hc-tomcat
#!/bin/bash
result=`curl -s http://$2:8080`
if [ ${result:0:5} = "<meta" ]; then
echo 0
else
echo -1
fi
The load balancer provides the following variables to use with your script, of which I'm only using $2:
$1 – VIP (literal IPv4 or IPv6 address)
$2 – Server IP (literal IPv4 or IPv6 address)
$3 – Protocol (UDP, TCP as a string)
$4 – Numeric port range (the user-specified value for hc-port)
$5 – maximum time (in seconds) that the test should wait before returning a failure. If the test runs beyond the specified time, it might be stopped, and the test would be considered failed. This value is user-defined and specified in hc-timeout.
Ensure the script has execute permissions (the ilbd deamon, which runs the health check, is not run as root):
root@ilb-ext:~# chmod +x /var/hc-tomcat
Giving the script a quick test:
root@ilb-ext:~# /var/hc-tomcat n/a 192.168.1.50
0
You then create a health check rule as follows:
root@ilb-ext:~# ilbadm create-healthcheck -h hc-test=/var/hc-tomcat,hc-timeout=2,hc-count=1,hc-interval=10 hc-tomcat
The hc-timeout is how many seconds the health check will wait for a response before giving up. The hc-count is how many times the script will attempt to succeed before claiming the server to be dead. The hc-interval is how often the health-check is performed.
Once created you can view the configured health-checks as follows:
root@ilb-ext:~# ilbadm show-hc
HCNAME TIMEOUT COUNT INTERVAL DEF_PING TEST
hc-tomcat 2 1 10 Y /var/hc-script
Now that we have a health check, we need to add it to our load balancing rule. Unfortunately, ilbadm doesn't have a command to modify an existing load balancing rule, so we have to delete it and create it again:
root@ilb-ext:~# ilbadm delete-rule tomcatrule_rr
We'll create the same rule as before, this time including the health check:
root@ilb-ext:~# ilbadm create-rule -e -p -i vip=10.0.2.20,port=80 -m lbalg=rr,type=HALF-NAT -h hc-name=hc-tomcat -o servergroup=tomcatgroup tomcatrule_rr
Once the new rule is created, the health check goes into effect. You can see the status as follows:
root@ilb-ext:~# ilbadm show-hc-result
RULENAME HCNAME SERVERID STATUS FAIL LAST NEXT RTT
tomcatrule_rr hc-tomcat _tomcatgroup.0 alive 0 10:39:51 10:40:02 2509
tomcatrule_rr hc-tomcat _tomcatgroup.1 alive 0 10:39:57 10:40:09 1869
So now, if probe.jsp is showing that you're hitting server 2 and we then disable Tomcat on Server 1:
root@server1-zone:~# svcadm disable tomcat6
When you refresh your browser you will be directed to Server 2. Of course, any state they you may have been maintaining on Server 1 will be lost. You can also see the status as dead using ilbadm show-hc-result:
root@ilb-ext:~# ilbadm show-hc-result
RULENAME HCNAME SERVERID STATUS FAIL LAST NEXT RTT
tomcatrule_rr hc-tomcat _tomcatgroup.0 dead 4 10:43:36 10:43:45 1102
tomcatrule_rr hc-tomcat _tomcatgroup.1 alive 0 10:43:42 10:43:53 5919
See Administering Health Checks in ILB for the official documentation.
No comments:
Post a Comment