This past week, I had the need to install opensolaris on a host using a USB thumb drive. To create a bootable USB drive, I first needed to snag the distribution constructor tools via mercurial (I ran these commands from an OpenSolaris host):
$ pkg install SUNWmercurial
$ hg clone ssh://anon@hg.opensolaris.org/hg/caiman/slim_source
The caiman slim source Mercurial repository contains a script named usbcopy, which you can use to copy a USB image from the genunix site to your USB drive:
$ usbcopy /nfs/images/osol-0811.usb
Found the following USB devices:
0: /dev/rdsk/c9t0d0p0 7.6 GB Patriot Memory PMAP
Enter the number of your choice: 0
WARNING: All data on your USB storage will be lost.
Are you sure you want to install to
Patriot Memory PMAP, 7600 MB at /dev/rdsk/c9t0d0p0 ? (y/n) y
Copying and verifying image to USB device
Finished 824 MB in 336 seconds (2.4MB/s)
0 block(s) re-written due to verification failure
Installing grub to USB device /dev/rdsk/c9t0d0s0
Completed copy to USB
After the image was copied, I plugged the drive into my machine and it booted to the opensolaris desktop without issue. From there I did an install and everything appears to be working flawlessly! Nice.
I have been a smoothie addict for several years, and recently got interested in saving some cash and making my own (at $5 a smoothie, the commercial smoothie shops were making $60+ from me each month!). To begin my quest for the perfect smoothie, I had to define a few requirements:
My first task was locating a high quality blender. I had numerous discussions with employees at various smoothie making shops, and they told me that their commercial blenders are crazy expensive (my estimates are in the $300 to $500 range). Ouch! I can understand that kind of investment when you are making 100s of smoothies a month, but this seemed excessive when you are making 1 or 2 a week. I read numerous blender reviews, and settled on the Hamilton Beach 52654 blender based on the awesome feedback from smoothie connoisseurs, and the fact that it is supposed to destroy every chunk of ice it comes in contact with. This blender has far exceeded my expectations, and has made some killer smoothies so far!
Once I had a blender, I needed a way to find organic fruit (strawberries, blueberries, blackberries, rasberries, mango, etc.) for a reasonable price. I checked out a couple of local grocery stores, and settled on frozen berries from Whole Foods (I try to use fresh fruit when possible, but frozen fruit seems to be a better fit for my busy lifestyle). After I had fruit and a killer blender, I needed to find some smoothie recipes to try. The Internet is the best place to find smoothie recipes, and I now have 4 - 5 recipes that I use pretty regularly (my favorite is strawberries, blueberries, bananas, 4 ice cubes, skim milk and a touch of sugar). I’ve experimented now for a few months, and I can safely say that my smoothies are BETTER than any of the commercial places. They are also tailored exactly to my tastes, which is a huge benefit in my mind. Each smoothie costs me roughly $1 to make, and they are oooooh so yummy! If you are a smoothie addict like myself, please leave me a comment with your favorite smoothie recipe.
I received a call this week from one of our network guys because messages from several network devices weren’t being logged by our centralized log server. When I started debugging the issue, I noticed that traffic from the hosts (host1 in this example) was making it to our syslog-ng server:
$ tcpdump -i eth0 host host1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
09:22:01.672377 host1.65093 > syslog.syslog: SYSLOG local4.error, length: 108
09:22:06.904446 host1.65093 > syslog.syslog: SYSLOG local4.notice, length: 128
While the traffic was making it to the server, the syslog messages were not being matched against a rule we had defined. Being the curious guy I am, I decided to read the syslog RFC to get a better understanding of the syslog message format. If you aren’t familiar with syslog messages, they take the following form:
The PRI section contains the facility and priority, the HEADER section contains a timestamp and the hostname or IP address of the device, and the MSG section contains the message the host is trying to send to the server.
When I dumped one of the syslog message as a raw string of bytes, I noticed that the hostname field didn’t match the expression we were trying to match against. For reference, here is the expression we were using (FYI: the host() directive matches against the hostname in the syslog message, not the source IP address in the IP datagram):
filter f_web_hosts { (host(192.168.0.25)) };
Since the hostname and source IP address were different, the rule didn’t match. To get this to work correctly, I needed to use a netmask() statement to filter based on the source IP address in the IP datagram:
filter f_web_hosts { (netmask(192.168.0.25/32)) };
Once this rule was in place, everything worked as expected! Rock on!
I came across the Linux tape library project website the other night, and this looks like a sweet project. If we could combine this solution with ZFS, the combination would be truly incredible! I need to install the LVTL software in my lab and give it a spin!
I am currently working on upgrading a number of Oracle RAC nodes from RHEL4 to RHEL5. After I upgraded the first node in the cluster, my DBA contacted me because the RHEL5 node was extremely sluggish. When I looked at top, I saw that a number of kswapd processes were consuming CPU:
$ top
top - 18:04:20 up 6 days, 3:22, 7 users, load average: 14.25, 12.61, 14.41
Tasks: 536 total, 2 running, 533 sleeping, 0 stopped, 1 zombie
Cpu(s): 12.9%us, 19.2%sy, 0.0%ni, 20.9%id, 45.0%wa, 0.1%hi, 1.9%si, 0.0%st
Mem: 16373544k total, 16334112k used, 39432k free, 4916k buffers
Swap: 16777208k total, 2970156k used, 13807052k free, 5492216k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
491 root 10 -5 0 0 0 D 55.6 0.0 67:22.85 kswapd0
492 root 10 -5 0 0 0 S 25.8 0.0 37:01.75 kswapd1
494 root 11 -5 0 0 0 S 24.8 0.0 42:15.31 kswapd3
8730 oracle -2 0 8352m 3.5g 3.5g S 9.9 22.4 139:36.18 oracle
8726 oracle -2 0 8352m 3.5g 3.5g S 9.6 22.5 138:13.54 oracle
32643 oracle 15 0 8339m 97m 92m S 9.6 0.6 0:01.31 oracle
493 root 11 -5 0 0 0 S 9.3 0.0 43:11.31 kswapd2
8714 oracle -2 0 8352m 3.5g 3.5g S 9.3 22.4 137:14.96 oracle
8718 oracle -2 0 8352m 3.5g 3.5g S 8.9 22.3 137:01.91 oracle
19398 oracle 15 0 8340m 547m 545m R 7.9 3.4 0:05.26 oracle
8722 oracle -2 0 8352m 3.5g 3.5g S 7.6 22.5 139:18.33 oracle
The kswapd process is responsible for scanning memory to locate free pages, and scheduling dirty pages to be written to disk. Periodic kswapd invocations are fine, but seeing kswapd continuosly appearing in the top output is a really really bad thing. Since this host should have had plenty of free memory, I was perplexed by the following output (the free output didn’t match up with the values on the other nodes):
$ free
total used free shared buffers cached
Mem: 16373544 16268540 105004 0 1520 5465680
-/+ buffers/cache: 10801340 5572204
Swap: 16777208 2948684 13828524
To start debugging the issue, I first looked at ipcs to see how much shared memory the database allocated. In the output below, we can see that there is a 128MB and a 8GB shared memory segment allocated:
$ ipcs -a
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x62e08f78 0 oracle 640 132120576 16
0xdd188948 32769 oracle 660 8592031744 87
The first segment is dedicated to the Oracle ASM instance, and the second to the actual database. When I checked the number of huge pages allocated to the machine, I saw something a bit odd:
$ grep Huge cat /proc/meminfo
HugePages_Total: 4106
HugePages_Free: 4051
HugePages_Rsvd: 8
Hugepagesize: 2048 kB
While our DBA had set vm.nr_hugepages to a sufficiently large value in /etc/syscl.conf, the database was utilizing a very small portion of them. This meant that the database was being allocated out of non huge page memory (Linux dedicates memory to the huge page area, and it is wasted if nothing utilizes it), and inactive pages were being paged out to disk since the database wasn’t utilizing the huge page area we reserved for it . After a bit of bc’ing (I love doing my calculations with bc), I noticed that the total amount of memory allocated to huge pages was 8610906112 bytes:
$ grep vm.nr_hugepages /etc/sysctl.conf
vm.nr_hugepages=4106
$ bc
4106*(1024*1024*2)
8610906112
If we add the totals from the two shared memory segments above:
$ bc
8592031744+132120576
8724152320
We can see that we don’t have enough huge page memory to support both shared memory segments. Yikes! After adjusting vm.nr_hugepages to account for both databases, the system no longer swapped and database performance increased. This debugging adventure taught me a couple of things:
Hopefully more work will go into the Linux huge page implementation, and allow me to scratch the second and third items off of my list. Viva la problem resolution!