Getting a daily status report from your Netbackup infrastructure

I support a couple of Netbackup environments, and like to keep tabs on what is going on with my media and master servers. There are a slew of reports available in the GUI and CLI interfaces, and these reports cover everything from Netbackup errors to tape reports to what is occurring with vault jobs. The two reports I find most useful are the jobs report and the errors report. These reports can be access through the bpdbjobs and bperror commands, and produce a nice summary of the jobs that ran and any errors that occurred. I use these so frequently that I run a script to e-mail me the output each morning:

Summary of jobs on nbmaster
Queued:                           2
Waiting-to-Retry:                 0
Active:                           4
Successful:                     479
Partially Successful:             4
Failed:                           6
Incomplete:                       0
Suspended:                        0
Total:                          495

    TIME            SERVER/CLIENT                      TEXT
11/29/2009 20:32:34 media1 client1  backup of client client1 exited with status 13 (file read failed)
11/29/2009 21:10:50 media1 client1  socket read failed: errno = 62 - Timer expired

Configuring yum to use an HTTP or FTP proxy

I have been experimenting with squid at home, and recently configured yum to use the squid proxy server I set up. There are two ways you can get yum to use an HTTP or FTP proxy. First, you can make yum use a proxy for a single session by setting the http_proxy and ftp_proxy environment variables:

$ export http_proxy=http://proxy:3128

$ export ftp_proxy=http://proxy:3128

$ yum update

If you want to make the proxy settings permanent, you can add a proxy directive to /etc/yum.conf:

$ grep proxy /etc/yum.conf
proxy=http://proxy:3128

If your proxy requires you to authenticate, you can add the credentials to the configuration file as well.

Adding 3rd party package repositories to CentOS Linux

As a long time CentOS user, I have grown accustomed to firing up yum to install my favorite packages. Periodically a package I’m looking for isn’t available, and I need to go out to a 3rd party repository to snag it. One awesome source for 3rd party repositories is the repositories section of the CentOS website, which contains the yum repository files for several extra package sources. If you are looking for a file that isn’t available in the stock CentOS distribution, you should take a look at that site!

Creating KVM guests with virt-install and qemu-kvm

In my KVM presentation, I discussed how to create KVM guests using the virt-install utility. To create a KVM guest, you can run the virt-install utility with one or more options that control where the guest will be installed, how to install it, and how to structure the guest hardware profile . Here is one such example:

$ virt-install --connect qemu:///system \
   --name kvmnode1 \
   --ram 512 \
   --file /nfs/vms/kvmnode1.disk1 \
   --file /nfs/vms/kvmnode1.disk2 \
   --network=bridge:br0 \
   --accelerate \
     -s 18 \
   --pxe \
     -d \
   --noautoconsole \
   --mac=54:52:00:53:20:15 \
   --nographics \
   --nonsparse

Under the covers virt-install executes qemu-kvm (at least on RHEL derives distributions), which is the process that is responsible for encapsulating the KVM guest in userspace. To start a guest using qemu-kvm, you can execute something similar to the following:

$ /usr/bin/qemu-kvm -M pc \
   -m 1024 \
   -smp 1 \
   -name kvmnode1 \
   -monitor stdio \
   -boot n \
   -drive file=/nfs/vms/kvmnode1,if=ide,index=0 \
   -net nic,macaddr=54:52:00:53:20:00,vlan=0 \
   -net tap,script=no,vlan=0,ifname=tap0 \
   -serial stdio\
   -nographic \
   -incoming tcp:0:4444

While virt-install is definitely easier to use, there are times when you may need to start a guest manually using qemu-kvm (certain options aren’t available through virt-install, so understanding how qemu-kvm works is key!). Viva la KVM!!!

Dealing with yum checksum errors

I support a couple of yum repositories, and use the yum repository build instructions documented in my previous post to create my repositories. When I tried to apply the latest CentOS 5.3 updates to one of my servers last week, I noticed that I was getting a number of “Error performing checksum” errors:

$ yum repolist
Loaded plugins: fastestmirror
Determining fastest mirrors
Updates | 1.2 kB 00:00
primary.xml.gz | 376 kB 00:00
http://updates/repo/centos/5.3/updates/repodata/primary.xml.gz: [Errno -3] Error performing checksum
Trying other mirror.
primary.xml.gz | 376 kB 00:00
http://updates/repo/centos/5.3/updates/repodata/primary.xml.gz: [Errno -3] Error performing checksum
Trying other mirror.
Error: failure: repodata/primary.xml.gz from Updates: [Errno 256] No more mirrors to try.

After reading through the code in yumRepo.py, I noticed that the error listed above is usually generated when the checksum algorithm specified in the repomd.xml file isn’t supported. The createrepo utility uses the sha256 algorithm by default in Fedora 11 (I created my repositories on a Fedora 11 host), so I decided to create my repository using the sha1 algorithm instead:

$ createrepo -v -s sha1 /var/www/html/repo/centos/5.3/updates

Once I created the repository metadata using the sha1 algorithm, everything worked as expected:

$ yum clean all
Loaded plugins: fastestmirror
Cleaning up Everything
Cleaning up list of fastest mirrors

$ yum repolist
Loaded plugins: fastestmirror
Determining fastest mirrors
Updates | 1.0 kB 00:00
primary.xml.gz | 367 kB 00:00
Updates 634/634
repo id repo name status
Updates Updates enabled : 634
repolist: 634

This debugging experience made me realize two things:

1. Having your package manager written in Python makes debugging super easy

2. Python 2.6 uses hashlib to perform checksums, and Python 2.4 uses the SHA module to perform checksums. The version of the SHA module that ships with CentOS 5.3 doesn’t support sha256, which is why we get the checksum error listed above.

I had a h00t debugging this issue, and am glad everything is working correctly now! Nice!

Creating yum repositories on CentOS and Fedora Linux hosts

I have written about the yum package manager in the past, and it’s one of the main reasons I use CentOS and Fedora Linux. Various 3rd party yum repositories are also available, allowing you to gain access to numerous packages that aren’t available in the stock distributions. This is great, but sometimes you want to create your own packages and distribute them to your clients. This is a piece of cake with yum, since you can create your own yum repositories.

To create your own yum repository, you will first need to install the yum-utils and createrepo packages:

$ yum install yum-utils createrepo

The createrepo package contains the createrepo utility, which can be used to create the repository metadata from a set of RPMS. The yum-utils package contains a couple of useful tools for managing repositories, including the verifytree and reposync commands. Once you have the tools installed, you will need to create a directory to store your RPMs and metadataa.

$ mkdir -p /var/www/html/repo/centos/5.3/updates

After the directory is created, you can use your favorite tool to copy the RPMs to this directory:

$ rsync -a rsync://mirrors.usc.edu/centos/5.3/updates/x86_64/RPMS/ \
/var/www/html/repo/centos/5.3/updates/RPMS

Next up, we need to create the repository metadata to go along with the RPMs. The createrepo utility was written to do just this, and takes the path to the base RPM directory as an argument:

$ createrepo -v -s md5 /var/www/html/repo/centos/5.3/updates

This will take several minutes to run, and will populate a number of files in the repodata directory:

$ cd /var/www/html/repo/centos/5.3/updates/repodata

$ ls -l

total 3208
-rw-r--r--. 1 root root 2509241 2009-11-24 11:43 filelists.xml.gz
-rw-r--r--. 1 root root  381198 2009-11-24 11:43 other.xml.gz
-rw-r--r--. 1 root root  371826 2009-11-24 11:43 primary.xml.gz
-rw-r--r--. 1 root root     984 2009-11-24 11:43 repomd.xml

The repomd.xml file contains the package sources, as well as pointers to one or more .gz files. The .gz files contains the list of packages that are available, as well as metadata to describe the package. If the repository was created successfully, you can verify the contents with the verifytree utility:

$ verifytree /var/www/html/repo/centos/5.3/updates

Loaded plugins: refresh-packagekit
Checking repodata:
  verifying repomd.xml with yum
  verifying filelists checksum
  verifying other checksum
  verifying primary checksum
Checking groups (comps.xml):
  verifying comps.xml with yum
  comps file missing or unparseable

Assuming all went as planned, you should now have a usable repository! To tell your clients to use it, you can create a new repository file in /etc/yum.repos.d. Here is an example:

[mattys-updates]
name=Mattys Update Server
baseurl=http://master/repo/centos/5.3/updates
enabled=1
gpgcheck=1

Now the next time you run ‘yum search’ on your clients, the packages in the newly created repository should be available. I really dig yum, and love the fact that they make the entire package management process relatively simple and straight forward!