Benchmarking Solaris and Linux systems with sysbench


With numerous companies and individuals moving to virtualized environments (e.g., VMWare, Solaris containers, Xen, etc), it is important to understand the costs associated with doing so. How do you measure the cost savings and overhead associated with a virtualization layer? Calculating cost savings is easy (especially if you have had a few accounting or finance courses), since it is relatively easy to figure out the costs associated with ongoing operations, data center space, and power and cooling. But measuring the impact from the virtualization layer itself is somewhat difficult, especially since virtualization companies don’t publish meaningful data to show the overhead associated with a given virtualization technology.

Since I think virtualization is the wave of the future, I decided to poke around google to find a set of tools that could help me measure the impact of a given virtualization technology. After a bit of searching, I came across a nifty piece of software called sysbench. sysbench can be used to measure system and application throughput, and includes tests to measure CPU throughput, memory bandwidth, mutexes, file I/O, scheduler performance, and OLTP transaction throughput. sysbench is a snap to setup*, and the documentation contains numerous examples to show the options available to each test. The first thing most folks are interest in is the speed of a given microprocessor. Sysbench comes with a “cpu” test to measure the CPU speed of a given server, and it can be accessed by passing the keyword “cpu” to sysbench’s “–test” option:

$ /opt/sysbench/bin/sysbench --test=cpu run

sysbench v0.4.6: multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1

Doing CPU performance benchmark

Threads started!
Done.

Maximum prime number checked in CPU test: 10000


Test execution summary:
total time: 39.4249s
total number of events: 10000
total time taken by event execution: 39.4209
per-request statistics:
min: 0.0039s
avg: 0.0039s
max: 0.0041s
approx. 95 percentile: 0.0039s

Threads fairness:
events (avg/stddev): 10000.0000/0.00
execution time (avg/stddev): 39.4209/0.00

sysbench also has a super useful memory benchmark built-in, which is accessed by passing the “memory” keyword to the “–test” option:

$ /opt/sysbench/bin/sysbench --test=memory run


sysbench v0.4.6: multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1

Doing memory operations speed test
Memory block size: 1K

Memory transfer size: 102400M

Memory operations type: write
Memory scope type: global
Threads started!
Done.

Operations performed: 104857600 (697855.14 ops/sec)

102400.00 MB transferred (681.50 MB/sec)


Test execution summary:
total time: 150.2570s
total number of events: 104857600
total time taken by event execution: 114.9915
per-request statistics:
min: 0.0000s
avg: 0.0000s
max: 0.0033s
approx. 95 percentile: 0.0000s

Threads fairness:
events (avg/stddev): 104857600.0000/0.00
execution time (avg/stddev): 114.9915/0.00

sysbench also has a built-in test to benchmark the performance of the Operating System scheduler, and it is accessed by passing the “threads” keyword to the “–test” option:

$ /opt/sysbench/bin/sysbench --test=threads run

sysbench v0.4.6: multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1

Doing thread subsystem performance test
Thread yields per test: 1000 Locks used: 8
Threads started!
Done.


Test execution summary:
total time: 7.6939s
total number of events: 10000
total time taken by event execution: 7.6898
per-request statistics:
min: 0.0007s
avg: 0.0008s
max: 0.0034s
approx. 95 percentile: 0.0008s

Threads fairness:
events (avg/stddev): 10000.0000/0.00
execution time (avg/stddev): 7.6898/0.00

These tests are interesting, but the best thing about sysbenhc is it’s ability to simulate read intensive and read/write intensive OLTP workloads against any of the major databases. To issue a read-intensive workload against a MySQL database, the “oltp” test can be used (you need to run the ‘prepare’ command to setup the database prior to running the test):

$ /opt/sysbench/bin/sysbench --test=oltp --oltp-test-mode=simple --num-threads=16 --max-requests= 1000000 --oltp-table-size=10000000 --mysql-socket=/tmp/mysql.sock --mysql-user=loadtest --mysql-password=loadtest --mysql-db=loadtest run


Running the load test ...

sysbench v0.4.6: multi-threaded system evaluation benchmark

No DB drivers specified, using mysql
Running the test with following options:
Number of threads: 16

Doing OLTP test.
Running simple OLTP test
Using Special distribution (12 iterations, 1 pct of values are returned in 75 pct cases)
Using "BEGIN" for starting transactions
Using auto_inc on the id column
Maximum number of requests for OLTP test is limited to 1000000
Threads started!
Done.

OLTP test statistics:
queries performed:
read: 1000000
write: 0
other: 0
total: 1000000
transactions: 1000000 (36216.40 per sec.)
deadlocks: 0 (0.00 per sec.)
read/write requests: 1000000 (36216.40 per sec.)
other operations: 0 (0.00 per sec.)

Test execution summary:
total time: 27.6118s
total number of events: 1000000
total time taken by event execution: 402.0923
per-request statistics:
min: 0.0000s
avg: 0.0004s
max: 1.1231s
approx. 95 percentile: 0.0001s

Threads fairness:
events (avg/stddev): 62500.0000/4939.18
execution time (avg/stddev): 25.1308/0.77

There are tons of options to control each of the tests listed above, and I have found it super useful comparing the results from an Operating System running on raw hardware with that of an Operating System running in a virtualized environment.

The fileio test bombs out due to the use of random() in sb_rand(). If you are interested in using the fileio test, you can email me for a patch to address the issue (I sent the patch to the sysbench maintainers).

This article was posted by Matty on 2006-05-11 01:13:00 -0400 -0400