Rolling shells

In my RAID performance article I’ve given a lot of performance numbers, but it is indeed a good question on how to exactly measure disk performance. To do this you need 2 tools. One that does the actual benchmark and one that measure the different system parameters so you can know the impact of the benchmark.

For the benchmark tool itself I usually use 3 tools. The first one is good old dd. The problem with dd is that it can only do 1 type of benchmark (sequential read and write), but it can to this to and from many sources and that is its strength.
So how do you use dd for a disk benchmark. Well usually like this :

dd if=/dev/zero of=/dev/sdb bs=64k count=100k
dd of=/dev/null if=/dev/sdb bs=64k count=100k

First a word of warning. Using dd like this will mean that any data on /dev/sdb (including filesystems) will be overwritten. The dd command does a write test and the second dd command does a read test. The write test reads from /dev/zero and writes to /dev/sdb, it uses a blocksize of 64k and it reads 102400 blocks. The read test reads from /dev/sdb and writes to /dev/null, it uses the same amount and size of blocks like the read test.
So in this case we are testing /dev/sdb, but this can be any blockdevice. Like /dev/md1 (a software RAID device) or /dev/VolGroup00/Logvol01 (a LVM logical volume.
The amount of blocks you need to read or write to have a valid is very simple. Make sure you process at least twice the amount of memory you have in your system. Why ? Because the Linux kernel does caching you need to make sure you request more data then the size of the cache to make sure you are actually testing the disk and not the Linux kernel caching.
And then the blocksize remains. Well, this depends on the type of usage you will do on the disks or RAID arrays. There are no real fixed rules for this. But generally speaking when you have a lot of small files take a small blocksize (4 to 16k), for standard fileserving and databases take a medium blocksize (64 to 128k) and for large files or applications that do a lot of sequential I/O use a large blocksize (256k and larger).

One final note on dd, since CentOS 5 dd will, when it has finished a test, tell you the speed if obtained. But on older versions of CentOS you will need to calculate this yourself. Another option there is to useddrescue, which is a tool similar to dd but also provides the bandwidth used in its output. You can find ddrescue for CentOS in Dag’s RPMForge repository.

As I said, dd only does sequential I/O but since it does not need a filesystem it is very useful to get a baseline/indication/magnitude/… for the performance of a disk. The next step is to put a filesystem on the device being tested. Then the second tool I use to test performance is iozone. You can also find prebuild rpms for CentOS in Dag’s RPMForge repository.

I usually use iozone like this

iozone -e -i0 -i1 -i2 -+n -r 16k -s8g -t1

Iozone has a look of options (that is part why I like it). If you do iozone -h you will get the complete list. I suggest you go over them when you start testing so you know all your options. But the example above shows enough to get started. The -e option adds the time of the flush operation to the results and the -+n disables the retest feature. The different -i options indicate the tests done by iozone, -i0 is the sequential write test (always needed), -i1 is sequential read and -i0 is random read and write. Another interresting test is the mixed read/write (-i8). The 3 remaining options are : -r 16k is the record (or block) size used, -s8g is the size of the data tested (8 GB in this case, remember the 2x RAM rule here) and -t1 is the amount of threads performing the test (in this case 1).

Depepding in the application you are planning to use you try to use iozone in a way that it mimics as close as possible the IO patterns of your real application. This means playing with the -i (the kind of test), the -r (block size) and -t (number of threads) options.

Other nice features of iozone are the autotest. Here iozone can iterate over different tests using different parameters for file size and record size. Also iozone can output its results in a Excell file, so you later make nice graphs of the tests.

The final test that I perform is a bit less scientific, but still a good indication of performance of a disk or a RAID array. The test is a copy of a directory that contains about 1GB of data. This data is stored in different subdirectories (I think it is about 5 levels deep) and all these subdirs contains either a couple of big files and for the rest a lot of small files.
This means that a copy of this directory involves a lot of metadata activity, and a mix of sequential I/O (the big files) and something that resembles random or mixed I/O (the small files). And the result is usually a big difference compared to the sequential speeds obtained with dd or iozone.

Now that we have all these benchmarks we still need a tool to monitor the system itself and see what is going on. The tool I always use for this is dstat. It is written and maintained by Dag of the RPMForge repo (and of course it also contains CentOS rpms of dstat). The nice thing about dstat is that is has different monitoring modules (and you can also write your own), its output formatting is really good and it can output the results in CSV files so you can also process the results in a spreadsheet. This is a example on how to use dstat :

dstat -c -m -y -n -N eth2,eth3 -d -D sda -i -I 98 3
—-total-cpu-usage—- ——memory-usage—– —system– –net/eth2- –dsk/sda– inter
usr sys idl wai hiq siq| used buff cach free| int csw | recv send| read writ| 98
0 0 100 0 0 0| 190M 195M 924M 2644M|1029 92 | 0 0 | 793B 4996B| 14
0 0 99 1 0 0| 190M 195M 924M 2644M|1008 41 | 21B 0 | 0 23k| 1

All the diferent output are different output modules and they get display in the order you have put them on the commandline. With the (for example) -n -N structure you can specifiy for which devices you would like to see output. In this case the -n refers to network-card statistics and with -N I specified I would like to see the output for eth2 and eth3. A similar systems is used for the disk statistics (-d and -D) and number of interrupts (-i and -I).

To get the list of extra modules available next to the standard onces do this :

dstat -M list
/usr/share/dstat:
app, battery, cpufreq, dbus, freespace, gpfs, gpfsop, nfs3, nfs3op, nfsd3, nfsd3op, postfix, rpc, rpcd, sendmail, thermal, utmp, vmkhba, vmkint, vzcpu, vzubc, wifi,

For more information about all the different dstat options do dstat -h or refer to the dstat man-page.

There is the second post by Daniel that thins goes right to the end. Great news i think.

This is an update to my previous post about getting a Windows license refund. In the meanwhile, some nice things have happened. First of all, my girlfriend was not the last to receive a refund for Windows Vista. Another Dutch Dell customerwas refunded 220 Euro for Windows Vista Premium and Microsoft Works. In summary: he used a slightly modified e-mail based on the e-mail that we sent to Dell. I take my hat off for Dell Netherlands, they seem to grok their customers. Additionally, the Dutch “General Conditions” state the possibility to ask for a refund of the software (I am not sure if this paragraph existed as-is before). A quick and sloppy translation of the relevant part (please refer to the original Dutch text for an accurate formulation):

If you reject the conditions for the use of the software, and if you are a consumer, Dell accepts software returns within 7 working days after the delivery of the software, and Dell will refund the price you paid for the software.

As a final remark: it may be even better to ask your vendor over the phone if you can order a machine without Windows, this may be less work for both sides.

I found this reading very interesting, although i abandoned Windows very very long ago. I hope you get your money back from Microsoft for the sake of Open source. 

Update #1: since some flaming already ensued I’d like to state first and foremost that the Dell representatives were very helpful and polite in handling this. They were open to this customer’s wishes, and she was very satisfied with her purchase and the subsequent refund.
Update #2: After some requests, I have put the e-mails that were sent online.

Recently my girlfriend bought a new computer. She was looking for a model that supported GNU/Linux, and opted for a Dell Inspiron 530, one of the models that can be purchased with Ubuntu in the United States. Unfortunately, in The Netherlands no consumer models are available with Ubuntu or any other GNU/Linux distribution yet. So, with no other options available, she ordered the machine, which was very affordable and had good specs.

Since she had planned installing GNU/Linux all along, and she is not particularly fond of the though of paying the Microsoft tax for software she will wipe out right away, we took care to read the EULA that is shown the first time the machine. The license said that if the EULA is declined, the customer should contact the manufacturer (or installer) about their refund policy. By the way, the EULA box seems to have been engineered to let people accept the EULA as quickly as possible: the box in which the EULA is shown is very small, making it an uncomfortable read. Additionally, there is only a button to accept the EULA, so we appropriately used the power button as a reject button .

After forcefully rejecting the EULA, we cleaned the partition table and installed GNU/Linux (which, as expected, works great on the Inspiron 530). Once everything was configured, she wrote an e-mail to Dell’s customer support. Since this is an English blog, I translated her e-mail:

Dear sir/madam,

A few days ago, I ordered a Dell computer. It was delivered yesterday, to my full satisfaction. The computer was pre-installed with Microsoft Windows Vista and Microsoft Works 8.0. Since I have installed GNU/Linux and declined the Windows license, I would like to make use of the refund option as described in the Windows and Works licenses.

I would like to inform how the refund procedure works, and would like to start it if possible.

Thanks in advance, With kind regards,

After a few days she received a reaction from Dell that stated that a refund would not be possible without returning the complete machine, because the license is inseparable from the hardware. In her answer she referred to previous cases where Dell Germany and Dell UK provided a refund to customers.

In the next reply a Dell representative answered that she was indeed eligible for a refund for both Windows Vista and Works. The combined refund is Euro 70 excluding tax. My conclusions:

• This provides no guarantee that Dell will give refunds to other customers. But at the very least they seem to be open to consumer choice for GNU/Linux (they have been providing GNU/Linux on servers and workstations for a longer time). They are slowly introducing some models with GNU/Linux in the EU, and in this case they also provided a refund.
• In the meanwhile I have heard from others that if you want a machine without Windows, it is often best to place an order by telephone to see if it is possible to order a machine without Windows, rather than using the website.
• From this refund and other stories, it seems that the per-machine “Microsoft-tax” is about Euro 70 (excluding tax). That’s quite much, try to get rid of it when you plan to erase any pre-installed system anyway. Aside the fact that it’s better for your wallet, purchasing or asking for machines without Windows shows that there is customer demand for choice.

Some time after the first post Daniel have another one i will post next.