Working around problems with a CentOS 6 kickstart upgrade

Each time I go to deploy a new release of Centos or Fedora I encounter a new “feature” or “change” that breaks configurations that previously worked. I recently encountered several problems upgrading several hosts to FEDORA 16, and this afternooon I encountered another issue while attempting to upgrade several Centos 5 servers to Centos 6.

The issue with the CentOS upgrade revolves around changes to the software installation groups. Groups that worked perfectly fine with CentOS 5 no longer work with CentOS 6. This results in the installer spitting out “This group does not exist” errors when it parses the “%packages” section in your kickstart.cfg:


So what exactly changed? In CentOS 5.7 we had the following groups available to the installer:

$ yum grouplist

Installed Groups:
   Administration Tools
   Dialup Networking Support
   GNOME Software Development
   Legacy Network Server
   Mail Server
   Network Servers
   Server Configuration Tools
   System Tools
   Web Server
   Windows File Server
   Yum Utilities
Available Groups:
   Authoring and Publishing
   Cluster Storage
   DNS Name Server
   Development Libraries
   Development Tools
   Engineering and Scientific
   FTP Server
   FreeNX and NX
   GNOME Desktop Environment
   Games and Entertainment
   Graphical Internet
   Java Development
   KDE (K Desktop Environment)
   KDE Software Development
   Legacy Software Development
   Legacy Software Support
   MySQL Database
   News Server
   OpenFabrics Enterprise Distribution
   PostgreSQL Database
   Printing Support
   Sound and Video
   Text-based Internet
   X Software Development
   X Window System

And in CentOS 6 this list was revised and expanded to the following groups:

$ yum grouplist

Installed Groups:
   E-mail server
   Graphical Administration Tools
   Hardware monitoring utilities
   Legacy UNIX compatibility
   Networking Tools
   Optional packages
   Performance Tools
   Perl Support
Available Groups:
   Additional Development
   Afrikaans Support
   Albanian Support
   Amazigh Support
   Arabic Support
   Armenian Support
   Assamese Support
   Azerbaijani Support
   Backup Client
   Backup Server
   Basque Support
   Belarusian Support
   Bengali Support
   Bhutanese Support
   Brazilian Portuguese Support
   Breton Support
   Bulgarian Support
   CIFS file server
   Catalan Support
   Chhattisgarhi Support
   Chichewa Support
   Chinese Support
   Compatibility libraries
   Console internet tools
   Coptic Support
   Croatian Support
   Czech Support
   Danish Support
   Debugging Tools
   Desktop Debugging and Performance Tools
   Desktop Platform
   Desktop Platform Development
   Development tools
   Dial-up Networking Support
   Directory Client
   Directory Server
   Dutch Support
   English (UK) Support
   Esperanto Support
   Estonian Support
   Ethiopic Support
   FCoE Storage Client
   FTP server
   Faroese Support
   Fijian Support
   Filipino Support
   Finnish Support
   French Support
   Frisian Support
   Friulian Support
   Gaelic Support
   Galician Support
   General Purpose Desktop
   Georgian Support
   German Support
   Graphics Creation Tools
   Greek Support
   Gujarati Support
   Hebrew Support
   High Availability
   High Availability Management
   Hiligaynon Support
   Hindi Support
   Hungarian Support
   Icelandic Support
   Indonesian Support
   Infiniband Support
   Input Methods
   Interlingua Support
   Internet Applications
   Internet Browser
   Inuktitut Support
   Irish Support
   Italian Support
   Japanese Support
   Java Platform
   KDE Desktop
   Kannada Support
   Kashmiri Support
   Kashubian Support
   Kazakh Support
   Khmer Support
   Kinyarwanda Support
   Konkani Support
   Korean Support
   Kurdish Support
   Lao Support
   Large Systems Performance
   Latin Support
   Latvian Support
   Legacy X Window System compatibility
   Lithuanian Support
   Load Balancer
   Low Saxon Support
   Luxembourgish Support
   Macedonian Support
   Mainframe Access
   Maithili Support
   Malagasy Support
   Malay Support
   Malayalam Support
   Maltese Support
   Manx Support
   Maori Support
   Marathi Support
   Messaging Client Support
   Messaging Server Support
   Mongolian Support
   MySQL Database client
   MySQL Database server
   Myanmar (Burmese) Support
   NFS file server
   Nepali Support
   Network Infrastructure Server
   Network Storage Server
   Network file system client
   Northern Sotho Support
   Norwegian Support
   Occitan Support
   Office Suite and Productivity
   Oriya Support
   PHP Support
   Persian Support
   Polish Support
   Portuguese Support
   PostgreSQL Database client
   PostgreSQL Database server
   Print Server
   Printing client
   Punjabi Support
   Remote Desktop Clients
   Resilient Storage
   Romanian Support
   Russian Support
   SNMP Support
   Sanskrit Support
   Sardinian Support
   Scalable Filesystems
   Scientific support
   Security Tools
   Serbian Support
   Server Platform
   Server Platform Development
   Sindhi Support
   Sinhala Support
   Slovak Support
   Slovenian Support
   Smart card support
   Somali Support
   Southern Ndebele Support
   Southern Sotho Support
   Spanish Support
   Storage Availability Tools
   Swahili Support
   Swati Support
   Swedish Support
   System Management
   System administration tools
   Tagalog Support
   Tajik Support
   Tamil Support
   TeX support
   Technical Writing
   Telugu Support
   Tetum Support
   Thai Support
   Tibetan Support
   Tsonga Support
   Tswana Support
   TurboGears application framework
   Turkish Support
   Turkmen Support
   Ukrainian Support
   Upper Sorbian Support
   Urdu Support
   Uzbek Support
   Venda Support
   Vietnamese Support
   Virtualization Client
   Virtualization Platform
   Virtualization Tools
   Walloon Support
   Web Server
   Web Servlet Engine
   Web-Based Enterprise Management
   Welsh Support
   X Window System
   Xhosa Support
   Zulu Support
   iSCSI Storage Client

So as you can see various groups were removed, some were renamed and others were added. I’ve come to expect problems like this with every CentOS and Fedora upgrade, since there appears to be little to no thought given to compatibility between releases. So with that said, my applications just finished compiling so it’s time to take CentOS 6 for a ride. :)

Adding a new CentOS or Fedora CD or DVD image to a kickstart server

I’ve talked in the past about Yum repositories, and how you can create them. If you are using kickstart and want to import the contents of a CD for use by the installer, there are numerous ways you can tackle this issue. One of the easiest methods is to loopback mount the CDs or DVDs and copy them to your kickstart directory hierarchy:

$ mount -o loop CentOS-6.0-x86_64-bin-DVD1.iso /cd1

$ mount -o loop CentOS-6.0-x86_64-bin-DVD2.iso /cd2

$ cd /cd1 && cp -rp repodata /kickstart/centos/6/x86_64/os &

$ cd /cd1 && cp -rp images /kickstart/centos/6/x86_64/os &

$ cd /cd1 && cp -rp Packages /kickstart/centos/6/x86_64/os &

$ cd /cd2 && cp -rp Packages /kickstart/centos/6/x86_64/os &

You will need to make sure to copy the Packages, repodata and images directories. After the copy completes you can point your kickstart.cfg file at the top level directory of the release you want to install:

url –url=

Posting here so I have this procedure for future reference.

Upgrading from vSphere 4.0 to vSphere 4.1 update 2

This weekend I decided to take the plunge and upgrade my vSphere 4.0 servers to vSphere 4.1 update 2. The upgrades went amazingly well, and I was amazed at just how easy it is to upgrade a cluster with VMWare update manager. The upgrade process took about 30-minutes per host, and I used the following two resources to guide me through the upgrade:

1. VMGuru’s vSphere 4.0 upgrade procedure.

2.VMWare KB article 1022140 which includes a video of the upgrade.

It’s amazing how big VMWare is getting. It seems like just yesterday I was running their desktop virtualization product, and people who didn’t have an extensive IT background would mumble “Virtual what?” Oh how times have changed.

Forcing glusterfs to heal bricks that are inconsistent

I mentioned in my previous post creating clustered file systems with glusterfs that gluster self-healing has a few issues. To see what I’m referring to we can access a gluster client and cp a bunch of data to the glusterfs file system located at /gluster:

[root@fedora-client etc]# cd /gluster

[root@fedora-client etc]# mkdir etc

[root@fedora-client etc]# cd etc

[root@fedora-client etc]# cp -rp /etc/* .

Once I started the copy operation I immediately reboot a brick to stop I/O from going to it. When the box came back up there was a discrepancy between the contents of the two bricks:

Gluster brick #1:

[root@fedora-cluster01 etc]# pwd

[root@fedora-cluster01 etc]# du -sh .
35M .

Gluster brick #2:

[root@fedora-cluster02 etc]# pwd

[root@fedora-cluster02 etc]# du -sh .
4.6M .

Bizarre, ey? This isn’t fixed in due time, you need to follow the instructions in the official documentation and stat the contents of the file system to make them consistent again:

[root@fedora-client etc]# find /gluster -noleaf -print0 | xargs –null stat >/dev/null

This return things to normal on the brick that was rebooted:

[root@fedora12-cluster02 etc]# pwd

[root@fedora12-cluster02 etc]# du -sh .
35M .

There are numerous reasons you could encounter this situation. A server could panic and reboot. You may need to reboot the server to apply updates. Or you may need to perform hardware maintenance (disk replacement, rack maintenance, etc.) which requires the server to be offline for a while. Jeff Darcy posted a nice reply to my original gluster post, and it looks like he’s in the midst of working on a permanent fix for this. It will surely be welcomed from folks who want to put gluster into production. :)

My Top Ten technology books of all time

I was having lunch with a friend last week and we caught up on tons of stuff. Both of us are doing a lot of work with VMWare, storage, Linux and interestingly enough we are both trying to learn about Windows Server 2008. As we were chatting my buddy asked me what I thought the best ten technology books of all time were. I immediately said the entire Richard Stevens collection, but then started to think which other books have had the biggest impact on my career. After a few days of thought I think I came up with my top ten:

1. The C Programming Language

2. Advanced Programming In The UNIX Environment

3. UNIX Network Programming

4. TCP/IP Illustrated Volume I

5. UNIX Internals

6. Solaris Internals

7. Panic! UNIX System Crash Dump Anlysis Handbook

8. UNIX File Systems

9. UNIX Power Tools

10. Self Service Linux

This was a hard list to come up with, but each of these books has had a profound impact on my love of UNIX and technology in general. Given the recent passing of the creator of UNIX Dennis Ritchie, I thought it only fitting that K&R would be placed on the top of my list. Pointers drove me nuts until I finally understood them, but the beauty and elegance of C still amazes me to this day. Which books have touched you the most?

Concert review: The Led Zepplin Experience with Jason Bonham

There are some moments in music you will never forget. One that I recall vividly was the first time I heard Led Zepplin IV with my best friend. From the moment I heard “Hey hey mama” in “Black Dog” I was blown away. These guys were hard, they had great drums, an amazing singer, a great bassist and some of the best guitar riffs I had ever heard. After hearing Zepplin IV three of four times I ran out the following week and blew my entire paycheck on Led Zepplin I, Led Zepplin II, Led Zepplin III and of course Led Zepplin IV. That summer became the summer of Zepplin, and I ended up destroying my Houses of the Holy cassette tape because I played it too much. That set me back an additional $15.

My fascination with led Zepplin landed me concert tickets to see Jimmy Page and Robert Plant in 1998, and boy was that an experience. A huge venue, two of my music idols, Robert Plant belting out all of my favorite Led Zepplin tunes and Jimmy Page playing guitar like no ones business. I can still remember the 20+ minute Kashmir solo they did, and how it brought chills to my body. They were true roll & roll pioneers, and I think they did as much for music as any other music legend. They also inspired me to learn how to play guitar, and for that I am crazy thankful!

Two weeks back my buddy called me up and asked if I wanted to go see Jason Bonham’s Led Zepplin Experience. For those that don’t know, Jason Bonham is the son of the late Led Zepplin drummer John Bonham. I can’t turn down any show that has Led Zepplin associated with it, so of course I said yet. This turned out to be the right choice, and the show was absolutely amazing. Let me share some of the awesomeness with you.

There wasn’t an opening band, and it turns out one wasn’t needed. Jason and his fellow musicians made the evening an intimate tour through Led Zepplin history, and along with playing a wide variety of Led Zepplin songs he also shared various pictures and home movies that his family had made. He also worked through the entire Led Zepplin collection, playing songs from Led Zepplin I – Led Zepplin IV, as well as tunes from Houses of the Holy and Physical Graffiti.

I don’t think a true Zepplin fan could ask for a better setlist. They started the night off with the “Immigrant Song,” and followed it up with timeless classics like “Rock & Roll, “Your Time Is Gonna Come,” “Thank You,” “Moby Dick,” “Over The Hills And Far Away,” “Stairway To Heaven,” “Since I’ve Been Loving You,” “When the Levee Breaks,” “Kashmir,” “Whole Lotta Love,” and one of my personal favorites “Dazed & Confused.” Every song was spot on, and when I closed my eyes I could almost picture myself at a Led Zepplin concert. Jason and his bandmates were THAT GOOD, and are some of the most talented musicians I’ve ever seen live (I was in awe with their guitarist, their vocalist sounded just like Robert Plant, and Jason was AMAZING on drums).

I’ve already mentioned that the show was killer, so you know I’m going to give it a 10/10. So what were the HCMs (Holy Crap Moments) from show? Was it The killer drum solo Jason did during “Moby Dick?” Was it the the guitar heroics on “Stairway To Heaven?” Or was it the amazing keyboard and bass work done during “Dazed & Confused?” I really can’t say, since everything sounded so good. You could see the artists love for Led Zepplin, and that poured out into their instruments. This is one of the best shows I’ve been to in a long time, and I’d like to thank Jason and his band for such a magical evening. If you are a Led Zepplin fan you need to catch this one. It’s the best $20 you can spend (If you don’t think so check out some of his videos on Youtube). :)