Measuring Linux Latency With LatencyTOP On Ubuntu 8.10 And Debian Lenny

Version 1.0
Author: Falko Timme <ft [at] falkotimme [dot] com>
Last edited 02/23/2009

LatencyTOP is a tool that lets you identify where in the system latency is happening, and what kind of operation/action is causing the latency to happen. This article shows how you can use LatencyTOP on Ubuntu 8.10 and Debian Lenny.

I do not issue any guarantee that this will work for you!

1 Ubuntu 8.10

1.1 Preliminary Note

LatencyTOP needs a kernel that was built with CONFIG_HAVE_LATENCYTOP_SUPPORT=y and CONFIG_LATENCYTOP=y which is the case for the standard Ubuntu 8.10 kernels (whereas on Debian Lenny, for example, only CONFIG_HAVE_LATENCYTOP_SUPPORT=y is set which means you'd have to rebuild the kernel to make LatencyTOP work on Debian Lenny).

grep -i latencytop /boot/config-`uname -r`

falko@falko-desktop:~$ grep -i latencytop /boot/config-`uname -r`
CONFIG_HAVE_LATENCYTOP_SUPPORT=y
CONFIG_LATENCYTOP=y
falko@falko-desktop:~$

1.2 Installing LatencyTOP

LatencyTOP is available as an Ubuntu package, so we can install it as follows:

sudo apt-get install latencytop

1.3 Usage

Usage is similar to the top program, you just type

sudo latencytop

and you will get a table of actions that cause latency:

Type q to leave LatencyTOP.

To learn more about LatencyTOP and additional options, take a look at its man page:

man latencytop

2 Debian Lenny

2.1 Preliminary Note

LatencyTOP needs a kernel that was built with CONFIG_HAVE_LATENCYTOP_SUPPORT=y and CONFIG_LATENCYTOP=y which is not the case for the standard Debian Lenny kernels. On Debian Lenny only CONFIG_HAVE_LATENCYTOP_SUPPORT=y is set which means we have to rebuild the kernel (see chapter 2.3) to make LatencyTOP work on Debian Lenny.

grep -i latencytop /boot/config-`uname -r`

server1:~# grep -i latencytop /boot/config-`uname -r`
CONFIG_HAVE_LATENCYTOP_SUPPORT=y
# CONFIG_LATENCYTOP is not set
server1:~#

2.2 Installing LatencyTOP

LatencyTOP is available as a Debian Lenny package, so we can install it as follows:

apt-get install latencytop

2.3 Rebuilding The Kernel

If you run LatencyTOP now, you will get an error message like this one:

server1:~# latencytop
Error writing to /proc/sys/kernel/latencytop: No such file or directory
Error writing to /proc/sys/kernel/latencytop: No such file or directory
Are you root?
Please enable the CONFIG_LATENCYTOP configuration in your kernel.
Exiting...
server1:~#

This means we must rebuild the kernel with CONFIG_LATENCYTOP=y. We can do this as follows:

apt-get build-dep linux-image-`uname -r`

apt-get install kernel-package libncurses5-dev fakeroot wget bzip2 build-essential

cd /usr/src/
apt-get install linux-source-2.6.26 linux-tree-2.6.26 linux-patch-debian-2.6.26

tar xjvf linux-source-2.6.26.tar.bz2
cd linux-source-2.6.26
make clean && make mrproper
cp /boot/config-`uname -r` ./.config
echo "CONFIG_LATENCYTOP=y" >> ./.config

Next we run

make menuconfig

where you can change the kernel configuration if necessary (but we've added CONFIG_LATENCYTOP=y to the configuration already in the previous step, so if the rest of the kernel configuration is ok for you, you can exit make menuconfig straight away without changing anything).

Now we build the new kernel:

make-kpkg clean
fakeroot make-kpkg --initrd --append-to-version=-custom kernel_image kernel_headers

This can take some time. Afterwards, we can find the new kernel in /usr/src...

cd /usr/src
ls -l

server1:/usr/src# ls -l
total 82448
drwxr-xr-x  3 root root     4096 2009-02-23 15:56 kernel-patches
-rw-r--r--  1 root src   9322496 2009-02-23 19:36 linux-headers-2.6.26-custom_2.6.26-custom-10.00.Custom_i386.deb
-rw-r--r--  1 root src  20574372 2009-02-23 19:28 linux-image-2.6.26-custom_2.6.26-custom-10.00.Custom_i386.deb
drwxr-xr-x 23 root root     4096 2009-02-23 19:30 linux-source-2.6.26
-rw-r--r--  1 root root 48398221 2009-01-10 16:37 linux-source-2.6.26.tar.bz2
drwxr-xr-x  4 root root     4096 2009-02-23 15:56 linux-support-2.6.26-1
server1:/usr/src#

... and install it as follows:

dpkg -i linux-headers-2.6.26-custom_2.6.26-custom-10.00.Custom_i386.deb linux-image-2.6.26-custom_2.6.26-custom-10.00.Custom_i386.deb

Open /boot/grub/menu.lst and make sure that the new kernel (2.6.26-custom) is the default one:

vi /boot/grub/menu.lst

[...]
default         2
[...]

Then reboot the system:

reboot

Afterwards, uname -r should show the new kernel:

uname -r

server1:~# uname -r
2.6.26-custom
server1:~#

2.4 Usage

Usage is similar to the top program, you just type

latencytop

and you will get a table of actions that cause latency.

Type q to leave LatenyTOP.

To learn more about LatencyTOP and additional options, take a look at its man page:

man latencytop

3 Links

• LatencyTOP: http://www.latencytop.org/
• Ubuntu: http://www.ubuntu.com/
• Debian: http://www.debian.org/