Isolating network traffic with IP instances

With the introduction of Nevada build 57, the Solaris IP stack was enhanced to support IP instances. IP instances allow you to create one or more unique TCP/IP stacks on a server, and each stack can be managed independently. What makes these extremely powerful is the ability to assign an IP instance to a zone or Xen instance, and then configure the IP stack attributes (e.g., IP filter policies, DHCP settings, etc.) from inside the zone or Xen guest domain.

To create an IP instance and assign it to a Solaris zone, you will first need to identify a spare physical NIC to dedicate to the zone (when Crossbow comes around, you will be able to allocate virtual NICs to zones, and these virtual NICs can reside on a physical NIC). Once a NIC is identified, you can use the zonecfg “ip-type” directive and the “exclusive” keyword to allocate an IP instance to a zone:

zonecfg:apache> create
zonecfg:apache> set zonepath=/zones/apache
zonecfg:apache> set ip-type=exclusive
zonecfg:apache> add net
zonecfg:apache:net> set physical=e1000g1
zonecfg:apache:net> end
zonecfg:apache> verify
zonecfg:apache> commit
zonecfg:apache> exit

Once a zone that uses an IP instance is created, the NIC can be configured just like any other interface on a Solaris server. Here is an example of how to plumb an interface in a zone, and apply a basic IP filter policy to that zone:

$ zlogin -C apache

$ ifconfig e1000g1 plumb

$ ifconfig e1000g1 inet 192.168.1.2 netmask 255.255.255.0 broadcast +

$ route add default 192.168.1.1

$ cat /etc/ipf/ipf.conf

### Block all inbound and outbound traffic by default
block in log on e1000g1 all head 100
block out log on e1000g1 all head 150

### Allow inbound SSH connections
pass in quick proto tcp from any to any port = 22 keep state group 100

### Allow my box to utilize all UDP, TCP and ICMP services
pass out quick proto tcp all flags S/SA keep state group 150
pass out quick proto udp all keep state group 150
pass out quick proto icmp all keep state group 150

$ svcadm enable ipfilter

$ ipf -f /etc/ipf.conf

As you can see, this is no different than configuring a physical IP interface from the global zone! IP instances are amazingly cool, and sites that need to isolate traffic between zones will definitely be happy (I am sure they will be even happier once crossbow is available)!

Getting core files when a Solaris hosts gets confused

In the past few months, I have had a couple of Solaris hosts go haywire (e.g., zones hanging, network interfaces no longer responding, etc.). When problems similar to these occur, I like to generate a core file from the running kernel to help the Sun support organization isolate the problem. There are two ways that I am aware of to grab a core file from a borked system. The first method utilizes the reboot utilities “-d” option:

$ reboot -d

This will reboot the host, and will generate a core file as part of the reboot. The second method uses the savecore utility to generate a core file from a running system. To use savecore, you will need to first configure a dedicated dump device (since swap is most likely in use on a production host, you can’t use it). Once the dedicated dump device is configured, the savecore utility can be run with the “-L” option:

$ savecore -Lv

It’s all about a bug free Solaris / Nevada, and these are two methods to help us get there.

Configuring hardware event notifications on X2200 servers

We live in a world where hardware breaks, and when it does, most adminsitrators want to get notified that something failed, and the specific component that failed. The Sun galaxy server line contains built-in hardware monitoring, and allows hardware events to be sent to administrators through SNMP, email and SYSLOG. The hardware notification facility in the X2200 servers complements the alerting capabilities built-in to the fault management architecture (FMA), and when the two are combined, hardware problems should be relatively easy to diagnose and fix.

To configure the X2200 to send email when a hardware event occurs, an email server and recipient email address need to be configured through the ILOM interface. This can be accomplished by logging into the ILOM and using the set comand to configure a recipient and an SMTP server:

/SP -> set /SP/AgentInfo/mail/receiver1 EmailAddress=ops@prefetch.net

/SP -> set /SP/AgentInfo/mail SMTPServer=1.2.3.4

The X2200 also provides a facility to send syslog messages when a hardare event occurs. To configure the X2200 to generate a syslog message in response to a hardware event, the set command can be run to enable syslog events, and to configure the destination syslog server:

/SP -> set /SP/AgentInfo/SEL ipaddress=1.2.3.4

/SP -> set /SP/AgentInfo/SEL status=enable

Once a syslog destination and email server are configured, the set command can be used to enable hardware events. The following set commands will enable hardware events for all sensor types (e.g., fans, CPUs, memory, etc.) and enable email and syslog notifications:

/SP -> set /SP/AgentInfo/PEF/EventFilterTable1 SensorType=all

/SP -> set /SP/AgentInfo/PEF/EventFilterTable1 SendAlert=enable

/SP -> set /SP/AgentInfo/PEF/EventFilterTable1 SendMail=enable

/SP -> set /SP/AgentInfo/PEF/EventFilterTable1 Status=enable

This is good stuff, but there is one downside with the X2200 ILOM (well, actually, there are a lot more, but I will discuss those in a future blog entry). There is currently no way to generate test events from the ILOM, or through the web interface. This is a huge issue IMHO, and it’s unfortunate that Sun didn’t include this in the X2200 ILOM software (most of the other galaxy servers support this, not sure why the X2200 had to be different). Hopefully Sun will address this glaring issue in a future code release.