Blog O’ Matty

ipmitool + LOM = rad amounts of hardware data collection.

There are so many different hardware sensors on machines now. If you have a machine with a iLOM (like Sun’s line of x86 hardware) you can probe all of this information over the network using ipmitool and direct it at the iLOM.

Usage for ipmitool over the network:

$ ipmitool -I lan -H (ip address of your lom) -U (username on lom) (command)

Typically Sun iLOMs use either "root" or "admin" as the username.   Adjust to your hardware / environment.  Note, ipmitool does send traffic across the wire in clear text.  It isn't encrypting a password, so keep that in mind.

Reporting on if there are any active fault LEDs on the box. (sunoem is of course, just for Sun gear)

$ ipmitool -I lan -H lomipddy -U root sunoem sbled get
Password:
LOCATE           | OFF
OK               | ON
SERVICE          | ON
FAN_FAULT        | OFF
TEMP_FAULT       | OFF
PS_FAULT         | ON
MB/P0/D0/SVC     | OFF
MB/P0/D1/SVC     | OFF
MB/P0/D2/SVC     | OFF
MB/P0/D3/SVC     | OFF
....
..

Reporting what components make up the machine (FRU = field replaceable units)

$ ipmitool -I lan -H ilomipaddy -U root fru
Password:
FRU Device Description : Builtin FRU Device (ID 0)
 Product Manufacturer  : ASPEED
 Product Name          : BMC CONTROLLER

FRU Device Description : /SYS (ID 27)
 Chassis Type          : Rack Mount Chassis
 Chassis Part Number   : 541-1982-06
 Board Product         : ASSY,MOTHERBOARD,DORADO/TUCANA
 Board Part Number     : 511-1394-02
 Board Extra           : 50
 Board Extra           : DT_MB
 Product Manufacturer  : SUN MICROSYSTEMS
 Product Name          : SUN FIRE X4140
 Product Part Number   : 4534567-4

FRU Device Description : MB/P0/D4 (ID 12)
 Product Manufacturer  : Micron Technology
 Product Name          : 4096MB DDR-II 666
 Product Part Number   : 36HTF51272PY-667E1
 Product Version       : 0100
 Product Serial        : D93D8C07

FRU Device Description : MB/P1 (ID 7)
 Product Manufacturer  : AMD
 Product Name          : Six-Core AMD Opteron(tm) Processor 2435
 Product Part Number   : 1008
 Product Version       : 00
...
......

iLOM event logs of what happened to the machine in the past

$ ipmitool -I lan -H ilomipaddy -U root sel elist
Password:
   1 | 12/24/2009 | 02:49:24 | Power Supply PS1/PWROK | State Deasserted
   2 | 06/05/2010 | 07:49:00 | System ACPI Power State ACPI | S0/G0: working | Asserted
   3 | 06/05/2010 | 07:49:03 | Power Supply PS1/PWROK | State Deasserted
   4 | 06/05/2010 | 07:49:04 | Power Supply PS1/VINOK | State Deasserted
   5 | 06/05/2010 | 07:55:14 | Power Supply PS1/PWROK | State Asserted
   6 | 06/05/2010 | 07:55:16 | Power Supply PS1/VINOK | State Asserted
   7 | 06/05/2010 | 08:09:04 | Power Supply PS0/PWROK | State Deasserted
   8 | 06/05/2010 | 08:09:06 | Power Supply PS0/VINOK | State Deasserted
   9 | 06/05/2010 | 08:09:10 | Power Supply PS0/PWROK | State Asserted
   a | 06/05/2010 | 12:23:46 | System ACPI Power State ACPI | S0/G0: working | Asserted
   b | 06/05/2010 | 12:23:51 | Power Supply PS0/VINOK | State Deasserted
....
...

And of course, current voltage / temperature / RPMs of fans, etc.

$ ipmitool -I lan -H ipaddyoflom -U root sdr elist
Password:
ACPI             | EAh | lnc |  7.0 | 0 unspecified
INTSW            | EBh | ok  | 23.0 |
MB/P0/PRSNT      | 01h | ok  |  3.0 | Device Present
MB/P1/PRSNT      | 02h | ok  |  3.1 | Device Present
MB/P0/T_CORE     | 09h | ok  |  3.0 | 7 degrees C
MB/P1/T_CORE     | 0Ah | ok  |  3.1 | 18 degrees C
MB/P0/V_VDDCORE  | 05h | lnr |  3.0 | 0 Volts
MB/P1/V_VDDCORE  | 06h | ok  |  3.1 | 1.03 Volts
MB/P0/V_+0V9     | 0Dh | ok  |  3.0 | 0.90 Volts
MB/P1/V_+0V9     | 0Eh | ok  |  3.1 | 0.90 Volts
MB/P0/V_+1V8     | 11h | ok  |  3.0 | 1.80 Volts
MB/P1/V_+1V8     | 12h | ok  |  3.1 | 1.79 Volts
MB/P0/V_VDDNB    | 15h | lnr |  3.0 | 0 Volts
MB/P1/V_VDDNB    | 16h | ok  |  3.1 | 1.31 Volts
MB/P0/PROCHOT    | 19h | lnc |  3.0 | 0 unspecified
MB/P1/PROCHOT    | 1Ah | lnc |  3.1 | 0 unspecified
MB/T_AMB         | 32h | ok  |  7.0 | 29 degrees C
MB/V_+12V        | 1Eh | ok  |  7.0 | 12.10 Volts
MB/V_+1V2HT      | 26h | ok  |  7.0 | 1.22 Volts
MB/V_+1V5        | 20h | ok  | 10.0 | 1.50 Volts
...
...

Note, since we are hitting a iLOM here to query this information using ipmitool, we're not directly interacting with a machine. In all the examples above, the actual server was powered off. Some cool stuff!

Also, all of these ipmitool commands can also be ran locally from the O/S. Instead of using -I Lan to communicate over the network, you'll want to use -I BMC (if you leave it out, it defaults to this) so the O/S knows to communicate with its own LOM.

Sweet! Only thing left is to gather this data and graph it into Zenoss or throw alerts in Nagios based off of poor values. =)

Note, that ipmitool gathering data is only as useful as the hardware it probes that support it. Various SPARC / x64 machines may return other values. Dell / HP gear may also report on things differently, so check out your vendor's LOM's ability to use ipmitool. You can download the source for ipmitool here.