For quite a long time the server which runs this very site has had some performance issues. This same server runs one or two instances of Mediawiki, and I have always just presumed that Mediawiki was the cause of the problems. I really didn't give it too much more thought, since the issues weren't causing many horrible user-facing performance issues. The server sort of hobbled along in the background, fairly loaded, but still managing to serve up pages decently. However, the problem most seriously manifested itself for me personally when working in a remote shell. Sometimes I'd go to save a file and the operation would take 10 or 15 seconds to complete. I ignored this, too, for some time, but it reached a point where I couldn't take it any longer.

I watched the output of top for a while, sorting on various metrics, and noticed that flush and kjournald were pegged at the top when sorted by process state, both being in a disk-wait ("D") state. This didn't make any sense to me, since the machine doesn't host any really busy sites and should have plenty of memory to handle what it has. I decided to do a web search for " linux flush kswapd " to see what it would turn up. As it turns out, the very first article returned in the search ended up indirectly shedding light on this issue, even though it turned out to be mostly unrelated to my own problem. However, what I did take away from it was learning of a utility that I didn't previous know about. Namely, perf , and specifically _perf top -a_.

What I discovered upon running this command was that the kernel was spending a huge amount of time (60% to 80%) running the function _acpi_pmread. A little investigation on this tracked it back to the kernel clocksource being set to _acpipm. The current, and available, clocksource(s) can be discovered by running the following, respectively:

$ cat /sys/devices/system/clocksource/clocksource0/current_clocksource
$ cat /sys/devices/system/clocksource/clocksource0/available_clocksource

I then went to another machine, also running Mediawiki, but one not having any performance issues, and found its clocksource to be hpet. After a little more research, some experiementing, and a few reboots, I found that adding the kernel parameter hpet=force to the variable _GRUB_CMDLINE_LINUXDEFAULT in /etc/default/grub and then running update-grub got the system using hpet as the clocksource. And this seems to have totally cleared up the issues on the machine. Processor usage is way down, memory usage is way down, processes in the disk-wait state are down, and our Mediawiki site is returning pages much faster that it ever has.

For reference, here are a few machine specifications which might be useful for others investigating this:

• OS: Debian Squeeze
• Processors: 2 x AMD Opteron 246