We've expanded our Linux news coverage and improved our search! Search for all things Linux across O'Reilly!

Testing SMP Kernel Modules with UML

by Jerry Cooperstein
03/03/2003

So you have just written, tested, debugged, and delivered a kernel module to your satisfied customer, who then distributed it to end users. Unfortunately, angry and frantic reports of system lockups, filesystem corruption, and other atrocities soon fill your mailbox. A little detective work quickly reveals that problems manifest themselves only on SMP systems. Perhaps it never even occurred to you that your module would be used on a multiprocessor box, so your testbeds all have single CPUs.

An ounce of prevention is worth a pound of cure. No competent kernel developer should make such an assumption. All kernel code should be written with SMP in mind as well, while making no assumptions about 32- vs. 64-bit, little-endian vs. big-endian systems, etc. Even so, if you haven't tested on an SMP system, you just can't be sure whether or not you've written a masterpiece or a bug-infested embarrassment.

One useful precaution is to turn on the kernel-preemption option, distributed as a patch for 2.4 kernels, and built into the 2.5 development series. Enabling full kernel-preemption can uncover some SMP bugs, as the scheduling path through kernel code is much less linear; i.e., after dealing with an interrupt, the kernel may pursue a different execution path than before. However, this check is partial and insufficient.

The obvious solution is to have an SMP system available for testing. Prices are indeed coming down to earth for dual-CPU motherboards and system components, although in these tight times you still may not be able to persuade your employer to pay for them. Furthermore, anything beyond a dual-CPU system is still very expensive, and there many be bugs and bottlenecks not visible in systems with one or two CPUs.

Related Reading Understanding the Linux Kernel By Daniel P. Bovet, Marco Cesati

User Mode Linux

Fortunately there is now a method of simulating a SMP system with a single CPU Linux system. Fantastically, it requires no financial investment. The tool is Jeff Dike's User Mode Linux (UML).

UML was developed as a new Linux architecture, although it doesn't have any associated hardware. It will eventually run on any physical platform. An instance of Linux--a full Linux kernel running with its own complete directory tree, device nodes, filesystems, etc., as needed--runs in non-privileged user mode as an application.

Multiple instances can run simultaneously, and can be networked to each other, their host system, and the rest of the universe. With a few caveats regarding networking, each instance (as well as the host) can be based on different kernels. In this respect, UML resembles VMWare and its counterparts, except that UML hosts and virtual machines must all be in the Linux family.

UML is very powerful and has many uses. Its SourceForge project page has abundant documentation. Here we only want to focus on the idea of using an SMP Linux kernel running as a UML instance on a single-CPU physical machine. Furthermore, we don't have the time here to consider the very interesting topic of how UML works.

At present, UML has one important limitation: its ability to debug real hardware devices is still a work in progress. Some work has already been done to drive physical devices on the host system from within UML; in particular, support for USB, sound, and PCI devices is under active development. See the UML web page for an up-to-the-minute status report.

However, even unfinished, UML is extremely useful. As pointed out on its web page:

Basically, anything in Linux that's not hardware-specific just works. This includes kernel modules, the filesystems, software devices, and network protocols.

Setting up UML

Experimenting with UML requires two major ingredients: a kernel and a filesystem, each of which can be downloaded in canned form. A pre-compiled kernel and pre-assembled filesystem will facilitate your first UML experience; eventually, you'll need to move past this. The UML home page provides a long menu of choices for both. Of course, I'm not aware of any pre-compiled SMP kernels, and you'll need the source anyway if you are going to insert and remove kernel modules of your own design.

For 2.4 kernels, you must obtain and apply the UML kernel patch. For 2.5 kernels, UML is fully incorporated, but the architectural implementation always lags somewhat behind Linus' official releases, and thus a kernel patch will still be required. To apply the patch, follow these steps (assuming the 2.4.19 kernel version):

$ cd /usr/src/linux-2.4.19
$ bunzip2 -c uml-patch-2.4.19-46.bz2 | patch --p1 
$ make ARCH=um xconfig
$ make ARCH=um linux

The kernel configuration step (make ARCH=um xconfig) can be a little tricky; not every possible choice of options will successfully compile. (This is after all, still an experimental feature.) Make sure you enable SMP. Other important options to turn on are host filesystem, virtual block device, virtual network device, and TUN/TAP transport. If you disable the kernel hacking options, you'll wind up with a much smaller and faster kernel that is harder to profile and debug. Once you have configured the kernel compilation, make ARCH=um linux to compile and link your UML binary, nominally called linux.

The second step is to generate a complete Linux filesystem, which will be stored in one big image file and mounted through the loopback mechanism. You can obtain a pre-manufactured image at the UML home page, and almost any one of them should work, with some minor twiddling. However, in the long run, you'll probably be more comfortable building your own with the tools and packages you really need. There are a number of available utilities for doing this. Your author became particularly enamored of Roger Binns' UML Builder program.

This utility has both a graphical and command line interface and excellent documentation. After installing the program, launch the GUI by typing umlbuilder_gui as an ordinary user. You must have available the installation media for an RPM-based Linux distribution on which to base the image. The first menu lets you pick from an assortment of Caldera, Conectiva, Mandrake, Redhat, SuSE, and TurboLinux releases (and a few other distributions, as well). After pointing to the RPM file location, select the packages you want and do some other configuration.

By default, uml_builder will give you a 1 GB image. Depending on what packages you install, a good fraction of that will be free space. I chose a very limited installation, which came to about 400 MB. If I weren't lazy, I would have pruned it down much further.

We should mention that uml_builder actually invokes a UML instance as it runs! You must have a UML binary to run it. You can install the UML RPM that comes with Red Hat, for instance, or obtain a binary from the web site. You'll use this just temporarily, as you'll use your SMP-enabled binary later.

It's a good idea to install the full UML package even if you don't use uml_builder, as it provides other useful utilities such as uml_net, which eases getting your UML instance to speak to the host. Otherwise, you will have some more work to do to get the TUN/TAP interface up and running on the host side. If you want to do everything from scratch, thoroughly read the documentation at the UML web site.

Once the filesystem is built, you can further tweak it (before running UML on it) by mounting it (as root) through the loopback mechanism with:

$ mount -o loop root_fs /tmp/mnt

We have assumed that the name of the image is root_fs. You can then navigate through the image and modify files. In particular, you may need to modify /etc/inittab. If you have a recent Red Hat distribution (as I do), make sure you have some uncommented lines pointing to virtual consoles, like:

1:2345:respawn:/sbin/mingetty tty1
2:2345:respawn:/sbin/mingetty tty2

or

1:2345:respawn:/sbin/mingetty ttys/1
2:2345:respawn:/sbin/mingetty ttys/2

which should work the same. You may need to comment out any serial line (starting with co:), depending on what you install and have in your host kernel. The above will give you two xterm-like windows when your kernel instance boots.

Running User Mode Linux

UML Builder comes with a complicated control script you can use to boot UML, but something as simple as:

#!/bin/bash

LINUX=./uml_linux ; ROOTFS=./rootfs ; SWAPFS=./swapfs ; MEM=64m ; NCPUS=4

$LINUX ncpus=$NCPUS mem=$MEM ubd0=$ROOTFS  root=/dev/ubd0  ubd7=$SWAPFS \
    eth0=tuntap,,,192.168.1.200

will do the job. In this example, we're giving our instance 64 MB of memory and pretending it has four processors. It will use the root filesystem and swap file images to which we have pointed. Note that ./swapfs should be a properly-prepared swap file, and that on your filesystem image, /etc/fstab, should point to it being mounted at /dev/ubd/7 in the above example. uml_builder takes care of these steps for you.

The TUN/TAP network interface (/dev/tap0) on the host side will be assigned to the address 192.168.1.200. During network configuration, I gave the UML instance the static address 192.168.1.201.

If you run the above command line, your Linux instance should start up, go through your distribution's usual system boot, and pop up two virtual console windows as xterms on your display. By default, with uml_builder you can log on as root with password=root. (You may want to change this, but remember that it is just a normal, unprivileged application.) You may see some annoying warnings during boot, but hopefully you are up and running.

I chose to do my development on the host machine rather than try to make sure I had all development tools properly installed on the more limited UML disk image. If you have networking installed properly, you can communicate and exchange data through usual methods (i.e., ftp, ssh, etc.), as well as do things like launch xterms on the host display. (Be sure you do xhost +192.168.1.201 on the host for our example.)

It's even easier to take advantage of the hostfs filesystem on the UML instance. With this you can simply do:

mount none -t hostfs /mnt/host -o /tmp/UMLTEST

which will mount /tmp/UMLTEST on the UML system at the /mnt/host mount point on the UML filesystem. You can just work in this directory simultaneously in both instances. If you put an appropriate entry in /etc/fstab, you can have this done automatically at boot.

Examples

Let's consider some simple tests to see how well UML can catch and highlight SMP bugs that would be invisible on uniprocessor systems. The full code for the kernel modules, testing programs, Makefiles, etc., can be found and downloaded here.

I: Data Corruption

We begin playing by writing a flawed module that abuses a global parameter by merrily letting multiple CPUs modify it simultaneously. To do this, we create an entry in the /proc filesystem that, when read, returns the CPU number and an unprotected global variable. This variable, pid_check, contains the invoking process' ID when the module entry point was entered. The relevant code fragment is:

cpu       = smp_processor_id ();
pid_check = current->pid;
do_something ();
rc = pid_check;
return (sprintf (page, "%3d %6d\n", cpu, rc));

where the delay routine

static void do_something ()
{
    int j = jiffies + HZ / 10;
    while (time_before (jiffies, j));
}

does an idiot's busy wait for a tenth of a second in order to give another CPU a chance to corrupt pid_check. It's important to do nothing during the delay that actually sleeps, as that would cause the process to get swapped out, and you'll get corruption even with one CPU.

If the testing program compares its own processor ID (from getpid()) with the returned value, it can check against corruption. On a single CPU system this will never happen; on an SMP system it will happen frequently, depending on CPU load. Fortunately, this bug is easily fixed by taking out a spinlock. The modified code is just:

cpu = smp_processor_id ();
spin_lock (&lock_A);
pid_check = current->pid;
do_something ();
rc = pid_check;
spin_unlock (&lock_A);
return (sprintf (page, "%3d %6d\n", cpu, rc));

This improvement avoids any data corruption on an SMP system, as you can verify with your UML instance.

II: Race Condition Leading to Deadlock

Now we'll use two nested spinlocks, and insert delays in between grabbing and releasing them:

cpu = smp_processor_id ();
spin_lock (&lock_A);
do_something ();
spin_lock (&lock_B);
pid_check = current->pid;
do_something ();
rc = pid_check;
spin_unlock (&lock_B);
do_something ();
spin_unlock (&lock_A);
return (sprintf (page, "%3d %6d\n", cpu, rc));

We create two /proc entries, one including the above code fragment, and one with the order of the locks reversed. We get a deadlock when a first process grabs the first lock, and, while the first is delayed in do_something(), a second process grabs the second lock. Each process will freeze while it waits for the other to release the lock it wants. By putting in a delay, we make it easy to trigger this deadlock; in real code such a bug may show itself only very rarely, when a particular combination of processes and load occurs. Discovering this problem can be difficult.

You should find no problem on a single CPU system, while the UML-SMP instance will deadlock. Note that you can't recover from this one without rebooting the instance. However, for convenience, the example code also provides two more /proc entries that can be used to unlock each of the locks, so you can manually remove the deadlock.

Both of these simple examples of stupidity can also be exposed by using a preemptive kernel. You may have some fun finding bugs that can be found by UML, but not by kernel-preemption.

Acknowledgment and Conclusion

I'd like to thank my Axian colleague, Bill Kerr, for some important suggestions.

User Mode Linux is a valuable tool for kernel development, and provides a cheap and enjoyable way to simulate SMP systems. Together with all of its other uses, it is likely to play an increasingly public role in Linux.

Jerry Cooperstein is a senior consultant and Linux training specialist at Axian Inc., in Beaverton Oregon, and lives in Corvallis, Oregon.

---

Return to the Linux DevCenter.