First thoughts of CentOS 4.0

I was recently introduced to CentOS 4.0, which is a Linux distribution built from the source RPMs that comprise Redhat Advanced Server. I am currently running several production services on CentOS 4.0, and am extremely happy with the features it brings to the table. Here are several reasons you might be interested in using CentOS:

1. CentOS 4.0 is free, and has the look and feel of Redhat Advanced Server. If you are used to managing servers running one of the Redhat derivatives, you will feel right at home.

2. CentOS 4.0 uses yum as the default package manager!

3. CentOS mirrors the upstream RPMS provided by Redhat, and offers several additional repositories ( e.g., extras, addons, contribs, centosplus, etc.) to extend the base CentOS installation.

4. The CentOS team will provide security updates for CentOS 4.0 until February 29, 2012.

5. CentOS is supposed (I haven’t tested this yet) to be binary compatible with Redhat Advanced Server. This can lower costs for companies that don’t need true parity between production, QE, test and development servers.

6. Redhat Advanced Server is widely deployed, and there are dozens of places to get free support for it. Since CentOS is identical to Redhat Advanced Server, you can use the Redhat support forums to find answers to your questions.

7. Redhat Advanced Server is running some large websites, and you can be assured that corporate customers are leveraging their support contracts to get problems fixed. These fixes turn up in CentOS 4.0, so you benefit for free.

8. You can use Redhat Global File System 6.1 and Cluster Suite 4.0 with CentOS 4.0.

If you are looking for a commercially supported Linux distribution, CentOS is probably not for you. But if you want to use a distribution that is super reliable and community supported, it might be the right distribution for you.

My new domain

After reading UNIX file systems, I got to be a bit of a file system junkie. The book does a good job of covering file system design, the block I/O interfaces, caching, the VFS and VNODE abstractions, and the implementation of FFS, VxFS and EXT3. The book piqued my interest in file systems, so I started doing some research on intelligent I/O schedulers and prefetch algorithms. Performing this research has helped me on numerous occassions with understand how to tailor workloads for a given file system and storage subsystem. Since intelligent prefetch is crucial for getting optimum read performance from a file system, and the name prefetch sounded kinda nifty, I decided to register a domain named Viva la fetch o’ pre!

Calling all vendors

I met with a vendor this week to discuss one of their software solutions, and as part of the 2-hour meeting I was hoping to get further details on how their solution worked, how it scaled, what was required to deploy and manage it, how other customers were applying their technology, and most importantly how the solution would benefit the company I work for. We had explicitly asked the vendor ahead of time to prepare a presentation to address these topics, and what we received was far from that! If I had to describe the vendors presentation, I would have to describe it as buzzword bingo meets Alf (the guy presenting kinda looked like an alien, which is the reason I used the reference to Alf). If you are a solutions provider that deals with customers, I have a few helpful hints for you:

1. Before you start a presentation with a potential customer, summarize the topics you are planning to present, ask the customer if this is what they are looking to see, and adjust the content to align with what the customer is looking for. The vendor in question dove straight into a presentation that addressed none of the concerns we had outlined prior to the meeting, so the meeting ended up being a complete waste of time.

2. Stay on task! If I could compare the presenter I described above to a board game, it would be shoots and laddders. He was all over the place, bouncing from complex subjet to complex subject, the topics he was covering didn’t relate with each other, nor did they have ANY applicability to the problems we were looking to solve with their solution. This not only confused several people in the room (me included, since I had no idea where he was going with his talk), but people started associating the crappy presentation with the product they were trying to sell.

3. If a customer brings you onsite to discuss a specific solution, and they mention that they are looking to use it to solve problem XYZ, stick to describing the solution the customer is interested in using. Nothing irritates me more than when a vendor tries to sell solutions to problems we don’t have, or attempts to sell products that have no applicably to the solution we are investigating. If you want to show us some fancy new product that you are developing, that is fine (most techies like to see presentations on new stuff). Ask the customer if you can come back to do a presentation on your other solutions, and use that time to sell new products.

4. If a customer asks an educated question in a meeting, and you don’t know the answer, reply with “I am not sure, but I will find out.” There are actually some super smart and technically astute people in IT, and
the chance that one of them is in the room is relatively high. Smart people see through your smoke and mirrors, so be open and honest. No one on this earth knows everything, and customers don’t expect you to either.

5. When a customer asks a question to which you say “I will look into it and get you an answer,” by gosh GET the customer an answer, or don’t make promises you can’t keep. Also, when a customer asks a question which you need to research, WRITE IT DOWN. Presenting can be stressful, and remembering questions during a stressful period is difficult. In case you didn’t know, when you write stuff down, it looks (looks are often deceiving though!) like you actually want to help.

6. Don’t try to kiss a customers ass by telling them how “brilliant” or “excellent” a given question is. We are the customer, so it is a given that our questions are brilliant. Leave well enough alone. ;)

7. Show examples of how your solution helped others, and be prepared to answer questions about the deployment. If your going to tout that you saved company XYZ 50-million dollars, be ready to explain how. Also, be ready to answer questions like “what did it cost to implement the solution,” and “what did it cost to manage the solution before and after the solution was deployed.” I realize tabulating these numbers is tricky, but you shouldn’t go around paraiding your solution as a cost saver unless you can back it up (most software solutions have lots of hidden costs, and smart customers will inquire about them).

8. If a customer asks if your solution supports an Operating System (e.g., x64 Solaris 10) or hardware platform (e.g., Fujitsu hardware) that you don’t plan to support, don’t try to BS them by telling them that your “engineering team will need to evaluate that request.” Be truthful, since folks will respect you more if your honest with them up front.

9. Pay attention to the audience, and if they look bored and confused, they probably are. STOP your talk and retool your presentation and delivery.

10. THIS ONE IS IMPORTANT! Define your acronyms, and don’t assume everyone in the room knows what acronym XYZ is. If you are using acronyms that are specific to your product, by gosh describe what it means!

Well enough of this tyrade! I hope this benefits someone. :)

USENIX 2006 wrap up

I recently got back from the annual USENIX technical conference, and thought I would BLOG about my adventures while I was in Boston. The following is a day-by-day breakdown of the sessions and events I attended:

Day 1
The first day I arrived, I attended a presentation from Alex Russell titled “Ajax and Advanced Responsive Day Webapp Development.” Alex began his talk with an overview of Javascript, and then proceeded to talk about the pros and cons of the numerous AJAX toolkits. His talk was exciting, and it was interesting seeing how AJAX fits into the web development model

Day 2
On day 2, I attended the keynote “PlanetLab: Evolution vs. Intelligent Design in Planetary-Scale Infrastructure,” which I was hoping was a talk on deploying and managing large distributed systems. The talk was ok, but nothing special. After the talk let out, I listened to several researchers talk about enhancements to Xen. This was an interesting track, and I thought Optimizing Network Virtualization in Xen was a cool paper. I also got to attend a a Perl guru session and a storage performance workshop during day 2, and had fun chatting with folks at the FreeBSD BOF.

Day 3
I started day 3 with John Sellen’s “RRDTool: Logging and Graphing” guru session. I got a lot of John’s talk, and found that the information was directly applicable to my job. Next I attended Steve Bellovin’s “Permissive Action Links, Nuclear Weapons, and the History of Public Key Cryptography” talk. Steve is a great public speaker, and one of the top security reseearchers in the world. His talk was interesting, especially the techniques he used to gather information on nuclear technology. I attended two additional talks after Steves, but they were somewhat dull and not worth talking about.

Day 4
On day 4 I attended David Sklar’s “Learning PHP5.” This was an awesome talk, and brought me up to speed on how PHP5 works. David is an exceptional speaker, and he definitely knows PHP inside and out.

Once David’s talk let out, I hopped on a plane and zipped back home. I have to say that I was somewhat dissappointed with USENIX 2006. The BOFs where non-existent, attendance seemed to be down from past conferences, and the vendors where nowhere to be seen. In the future, I think I will stick with LISA.

Reasons why people are switching from Solaris to Linux

I met up this week with one of my friends that I haven’t seen in a while. We chatted about life, work and eventually started chatting about Linux and Solaris (we are both SysAdmins). My friend mentioned that his company had decided to quit buying Sun hardware in favor of Dell servers running Redhat Linux Advanced Server. I was shocked to hear this since my friend had actively pushed Solaris in the past, and was one of the folks I regularly got together with to discuss new technologies merged into Nevada. His company has numerous concerns surrounding Solaris 10 manageability and Sun’s lack of ACTIVE support for commonly used opensource packages. We chatted about this for hours over cocktails, and both came to the conclusion that Sun needs to do something to address the following problems with Solaris:

1. Solaris doesn’t ship with a working and supported LAMP stack (I should probably say SAMP stack). My friend’s company is frustrated with having to manually download and build Apache, MySQL, and PHP on their Solaris boxes, and chose to move to Redhat Advanced server to get a working and SUPPORTED LAMP solution out of the box. I am not sure why Sun can’t ship a working and supported SAMP stack with Solaris. This seems like a no brainer to me.

2. Several of the developers at my friends company have transitioned to Fedora Core on their desktops, since the desktop looks pleasant, wireless works out of the box for most chipsets, eclipse is an installation option, and there is a full suite of applications available after the installation. The Fedora Core desktop is quite a bit more usable that JDS (if you don’t believe me, install Fedora Core 5 side-by-side with JDS), so developers have jumped all over it (at least those that don’t use Windows). Sun really needs to do something to improve desktop usability, and they should use the GNOME release from versus their own variant. They also need to do something to address package management, either by adopting blastwave or developing a decent remotely-accessible package repository.

3. Redhat Linux ships and provides regular updates for numerous opensource software (e.g., postgres, MySQL, Apache, Samba, Bind, Sendmail, openssh, openssl, etc), where Sun keeps trying to sell customers the Sun Java One stack, “modifies” an opensource package and diverges the product from what is available everywhere else, and fails to provide timely bug fixes and security patches for the opensource packages that are shipped (Apache, MySQL and Samba are perfect examples) with Solaris. Sun really needs to get some folks focused on supporting the opensource solutions people use, versus shipping opensource software and letting the bits rot.

4. Several key ISVs are pushing Linux and Windows over Solaris, and have switched from Solaris to Linux as their tier I development platform. This typically means that developers will squash more platform-specific bugs in their product prior to shipping it, since they are using that platform daily. Sun needs to do more to get developers writing code on Solaris, since this helps Sun customers in the end.

5. Managing applications and patches on Solaris systems is a disaster, and redhat’s up2date utility is not only efficient, but has numerous options to control the patch notification and update process. This can also be used along with Redhat’s satellite server to provide Enterprise wide patch and application management. While Sun kicked off an effort to address the patch and installation process, I wonder if it will be too little too late.

6. Staying on the cutting edge with Nevada is difficult, since there is currently no way to easily and automatically upgrade from one release of Nevada to another. On Fedora Core servers, you can run ‘yum upgrade’ to get the latest bits. Having to download archives and BFU is tedious, and most admins don’t want to spend their few spare cycles BFU’ing to new releases.

7. Zones are unusable at my friend’s site, since there is currently no way to filter traffic between zones, apply QOS measures to memory, I/O and network resources, and patching a box with zones can take days in some cases (I have experienced this first hand. If you want to see, install Solaris 10, create 25 non-sparse zones, and run smpatch update). Addressing these items would allow SysAdmins to actually patch their systems, and would allow folks to sleep at night knowing that the QOS measures will protect rogue applications from taking down their servers.

8. The Solaris opensource movement was great, but in our opinions it is very much closed to the outside world. How many people outside of Sun have actually done ARC reviews, code reviews, or applied a putback to the kernel source tree (there may be cases, but I can’t find them on This is definitely not something that can happen overnight, but people who have to wait in a queue for a sponsor, or worse yet are ignored (I filed a bug 3-months ago and asked to work on it, and have yet to hear back from Sun) when they try to fix something, will cause people to join communities where their voice actually matters.

That said, Solaris 10 is an awesome Operating System, and comes with some incredible technologies (e.g., ZFS, DTrace, FMA, etc). I truly do hope that Sun takes some steps to address these issues, since it will hopefully lead to further adoption of Solaris.