Write Emulator-Friendly Linux Code

By Peter Seebach
2005-04-16

Software Emulators

In the world of emulation, software emulators are where life gets interesting. A software emulator is not running your program on a virtual machine -- it's running it on the fly without a virtual machine. These programs work by setting up an environment in which a program's code can run normally, but attempts by the program to access the operating system get routed through an emulation layer of some sort. WINE is a great example (albeit for Windows), although it is officially not an emulator.

Some software emulators are explicitly invoked by the user, like the lxrun program available for SCO and Solaris systems. Others are built into a UNIX® kernel's support for loading binary images -- if a program doesn't look to be valid, it can be compared against a table of possible emulators that can look at it to see whether they can run it.

Software emulators often offer the best user experience. There's no special set up, no large disk images. The programs just run (most of the time). Access to system calls, shared libraries, and file system structures raise a number of issues, though, so we'll cover them next.

System calls
System calls are the easiest and the hardest part of emulation. A system call has a well-defined interface, and the calling mechanism can generally be easily detected and handled -- that's the easy part. The hard part is that the system call may be difficult or impossible to implement reasonably.

Traditionally, the big killer in Linux emulation was the clone() system call. This call provided a brute force way to get simple threading by creating two processes that shared a number of things that could include memory, file descriptors, signal handling -- in other words, anything and everything. Unfortunately, if your operating system didn't provide a good analogue to this, there was simply no way to implement the system call.

Worse, since clone() showed up when POSIX threads were not well or widely supported and was often used as a substitute for them, a lot of programs used it in a variety of exciting, complicated, and (need I say) unexpected ways.

If you want people to run your binaries, try to stay away from OS-specific system calls; favor standard POSIX system calls. This is a good practice in software development.

A kernel-based emulator traps the system calls when they reach it. A user-space emulator such as lxrun waits for the application to try to make a system call. Because the Linux system call facility is not the same as the system call facility on Solaris or SCO UNIX, the result is a segmentation fault. The lxrun program then acts like a debugger, correcting the fault and continuing -- but in fact, it has intercepted the system call, made a corresponding system call to the underlying operating system, and patched everything up. Clever!

File system structures
The problem with file systems is often more subtle. It's easy enough to access the file system. What's not easy is finding the files there that you expect.

If your program is running in emulation, the file system you access may be substantially different from the file system you had when you were developing the program. For instance, if your program uses the /proc file system (commonly used to get access to kernel status and information), it's possible that a feature common in more recent kernels will be absent on an older system.

Linux developers have a big advantage here over developers on proprietary systems, because different Linux distributions arrange files differently, so most programmers have a good sense of how to avoid being too dependent on file system layouts. Nonetheless -- sometimes -- a file name will have a perfectly good reason for being encoded in a program.

A solution to this dilemma, adopted in more than one emulator, is to set up an extra layer of interpretation for file system calls. For instance, in NetBSD's Linux emulation code, file accesses are checked first against the files in /emul/linux and only after that against the files in the system's real root directory. This allows the system to provide "overrides" for system files when Linux binaries won't work with the standard files.

In fact, the main use for this is in libraries and other support files, but a number of system binaries are provided as well. For instance, if a Linux binary were to try to call uname to get a kernel version, it would be very confused if it got back a NetBSD version number. Instead, it gets the Linux version numbers it's expecting.

Shared libraries
As mentioned above, shared libraries are a good candidate for being found by the emulated binaries but not by system binaries. Because the details of shared library formats and ABIs may vary from one system to another, you can't just assume that all the systems can share a given library. Names will clash -- for instance, the current NetBSD and SUSE 7.3 both have a file called libncurses.so.5. Gettting the right one of those is important.

Shared libraries bring up another point for developers. It's important to know what library version the different systems are using. Right now, NetBSD is using SUSE 7.3 shared libraries for its Linux emulation. There's code to grab the 9.1 shared libraries, but there's also a warning that they aren't stable with the kernel-level emulation.

Emulation packages tend to lag a bit behind the rest of the marketplace. Even if you think that most of your prospective users will have reasonably current Linux distributions, the emulator crowd will almost all be a bit behind the times.

Shared libraries bring up another concern -- not every system contains all of them. Emulation packages are often likely not to have every last shared library installed. And, to make it more fun, their users are less likely to be able to easily install a missing package.

In these cases, it's a good idea to minimize dependencies, both on new features and on non-core shared libraries. Emulator users are likely to run into these issues.

Don't get tricked into using static libraries as insurance against these problems. A static library can introduce its own new dependencies, and you can't check them as easily. It doesn't do any good to rework an algorithm to avoid an unportable system call if you statically link with a library that uses it. Dynamic linking allows you to build a program that will run on a much broader variety of systems.

Programs calling other programs
There is one special case that seems to bite people more than any other, especially with installers. On many systems, the default shell you get by calling /bin/sh is notbash. This means that scripts that assume bash extensions may not work on other systems.

This gets into an especially tricky bit of logic in the emulator. The operating system probably knows enough to check the Linux path for relevant Linux binaries when a binary is executed, and it will likely have a copy of bash installed there. But when you run a script, the kernel doesn't see this as a Linux binary; it sees a script with an interpreter path, and it's no longer running in emulation mode when it tries to load the interpreter.

Portable shell scripting techniques pay off here. This is one of the most common issues users face when running emulated applications. The installer may fail to run because it's a nonportable shell script.

Tutorial Pages:
» A Developer's Guide to Linux Emulators and How They Operate
» The Basic Emulator
» Emulators as a Distribution Format
» Full Hardware Emulators
» Partial Hardware Emulators
» Software Emulators
» Like Normal Development, Only More So
» Resources

First published by IBM DeveloperWorks

Related Tutorials:
» How to Install PHP 5 on Linux
» How to Install Apache 2 on Linux
» How to Install MySQL 5.0 on Linux
» SMB Caching
» Mound --Bind
» Tar Wild Card Interpretation