Installing Linux on Removable Drives
Obtaining an external drive is a great way to breathe new life into older hardware, or allow you to use Linux on machines on which you can’t (or don’t want to) alter the internal hard drives.
Say you want to use Linux in a dual-boot arrangement, but you don’t have any free space on your computer’s hard drive. One solution would be to use a “live” Linux distribution such as Knoppix, which can be run directly from CD. This is certainly viable for occasional use, but it has a number of serious drawbacks:
• You will still require some permanent storage for your data files. A floppy disk may be suitable if you only work with very small files, or a USB flash memory key may suffice for medium-sized files, but neither of these is ideal.
• When using a “live” CD, it is at best difficult — and at worst, impossible — to install your own applications, or to customize existing applications.
• There is a performance penalty to using live distributions, most notably on start-up while all of the devices are being detected — but also while running (as everything has to be loaded from CD, which is usually much slower than from the hard drive).
To be sure, other options also exist. For instance, you could buy another internal drive and install Linux on that. But often, you may not have any free drive bays in your case (this is particularly true of laptops, which typically only allow one internal hard drive).
Or, you could replace the current drive with a larger one, and install Linux in the resulting extra space. This, however, is a very time-intensive option, as it requires you to reinstall your existing OS on the new drive, reinstall and reconfigure all of your applications, and restore all of your data.
A much better solution is to purchase an external hard drive and install Linux on that. This lets you leave your existing hardware and software untouched and simply connect the external drive when you want to use Linux.
Removable Drive Options
The range of removable devices onto which Linux may be installed ranges from floppy drives, through USB-flash devices, on to USB/FireWire hard drives, and more.
While it is certainly possible to install Linux on a small-capacity device such as a 1.44MB floppy or a 32MB USB-key, these are usually (necessarily) specialized, cut-down distributions intended for, for instance, rescuing broken installations.
External hard drives, however, offer the most flexibility for a general purpose Linux distribution at a reasonable cost.
External drives are available in a large number of different sizes from a number of different manufacturers (Maxtor, Western Digital, and so on). These drives tend to consist of an external box, which holds a standard 3-1/2 inch or 2-1/2 inch IDE drive. These drives then are typically connected to the computer via USB or IEEE1394 (FireWire) connection.
USB comes in two main versions, 1.1 and 2.0. Version 1.1 has a maximum transfer speed of 12 Mbit/s (megabits per second), whereas version 2.0 supports transfer speeds of up to 480 Mbit/s. While most 2.0-compatible drives are backwardly compatible with 1.1, it is generally advisable to avoid using 1.1 unless there is no other option (due to its slow speed).
The FireWire standard also defines a number of different possible speeds, but in reality, whenever people say FireWire, they mean “FireWire400,” which supports transfers up to 400 Mbit/s.
There is little to choose between USB 2.0 and FireWire in terms of speed: although USB 2.0 has a higher quoted speed, in practice they both tend to be comparable due to differences in the protocols. If your machine has both, it is probably worth going for USB rather than FireWire (for reasons I’ll explain later), but if it has only FireWire, then of course you will go for that. For maximum flexibility, choose one of a number of drives that support both USB 2.0 and FireWire (such as the one I use later in this article).
For machines that do not have the required ports, PCI (for desktop) and PCMCIA (for laptop), FireWire and USB 2.0 cards are now available very inexpensively: for example, the PCMCIA FireWire card I use later in this article was purchased for approximately 10 GBP (under $20 US).
For the purposes of this article, I have purchased a 5-1/4 inch external drive enclosure. This is a very flexible enclosure, which is supplied with no drive and can be filled with any standard IDE device, including 3-1/2 inch hard drives and 5-1/4 inch IDE devices such as CD-RW/DVD-RW drives. The enclosure has both USB 2.0 and FireWire connections.
In order to connect the enclosure to my IBM Thinkpad T30 laptop, I also purchased a PCMCIA FireWire card (as the built-in USB ports only support USB 1.1).
Both the enclosure and the FireWire card were relatively inexpensive (approximately 50 GBP and 10 GBP, respectively).
For testing purposes, I fitted the enclosure with a 13GB 3-1/2 inch IDE drive I had laying around — for real world usage, I would buy a larger capacity drive, which again are now very cheap (approximately 50 GBP per gig!).
As you may expect, Linux support for these enclosures is very good indeed. Any device that adheres to the SBP (Serial Bus Protocol) standard for “Mass Storage Devices” can easily be used with Linux.
In general, to enable support for these devices you will need to have a number of things supported in your kernel (either directly compiled in or via modules).
For both USB and FireWire, SBP device support is implemented via SCSI emulation — that is to say, the devices appear to Linux as though they were SCSI disks. This is a common way to abstract storage devices within the Linux world (for instance, IDE CD/DVD drives are typically also connected using SCSI emulation). For this reason, the following kernel support is required:
• SCSI support
• SCSI emulation
• SCSI disk support
In addition, the following support will be required according to the connection method:
• For FireWire:
• IEEE1394 support
• OHCI1394 support
• RAW1394 support
• SBP-2 support
• For USB:
• (host-side) USB support
• OHCI support
• UHCI support
• USB mass-storage support
You will obviously have to have all the normal support for the rest of your hardware (graphics cards and so on), and may require some additional modules, depending on your exact hardware.
For instance, I am using a PCMCIA (cardbus) FireWire card, so I needed to add:
• PCMCIA support
• cardbus support
Now that we have our nice external drive, we shall start by installing Linux on it.
The easiest way to install Linux these days (certainly in my opinion), is to connect all of your hardware (in my case, this consisted of plugging in the PCMCIA FireWire card, attaching the FireWire cable to the PCMCIA card and the drive, and turning on the drive’s power switch); then boot up your machine with the installation CD from your distribution of choice.
My distribution of choice is Gentoo (see Resources for a link), so I used the latest “Universal” x86 Live CD (2004.1). The steps required for other distributions should be more or less comparable to those outlined here.
Once you have booted with the install CD, with a bit of luck it should have recognized your drive. The drive should appear as a disk under /dev/sdX, where X is a lowercase letter starting at “a.” On my system, the external drive was detected as /dev/sda, but this will vary if you have other SCSI disks (or emulated SCSI disks); in that case, it might be /dev/sdb or some other letter. If your drive is not detected automatically, some further steps may be required — for instance, you might have to pass boot options to enable FireWire or PCMCIA, or you might have to manually load some kernel modules, or other things of that sort (see Resources for links to troubleshooting guides).
Once the drive has been recognized, it should behave exactly like an internal hard drive as far as the rest of the installation is concerned; so you should be able to partition it as required and install Linux as normal.
One word of caution, however: care needs to be taken when deciding on where to install the boot loader (usually GRUB or LILO) — I would recommend not installing it in the Master Boot Record (MBR) (which is usually the default). Rather, it should be installed in the root partition (or boot partition, if you use a separate one) of the external drive.
Now that we have Linux installed on the device, we want to boot it up. This is where things can start to get a little tricky.
Before I discuss booting up your new drive, a little boot loader theory is required.
Boot loaders are usually installed in the MBR of the first hard disk in the machine. When the boot loader is invoked (the BIOS automatically executes the code in the MBR), it usually displays a menu of possible OSes to boot. Selecting a given OS causes it to boot up.
Two things should be noted about this scenario:
• The menu of OS choices is (usually) loaded from disk
• To boot the relevant OS, the bootloader needs to read the relevant kernel from disk
As the above takes place before the OS has been loaded, it means that all of the disk reading must take place by way of BIOS calls. This has a very serious implication: namely, that in order to boot the disk directly, your BIOS must support disks connected via FireWire or USB. This can typically be seen as a BIOS option to boot from these types of disk. FireWire BIOS support is currently very rare indeed, but USB support is becoming reasonably common. Therefore, if you are using USB on a relatively recent machine, it should be possible to boot the drive into Linux directly.
After installing GRUB in the MBR of the external drive, I was able to boot it directly when connected via USB. Simply enter the BIOS setup utility while booting with the disk connected. The external disk will appear as a regular hard drive: move it so it is before the internal drive in the boot order.
I was also able to install a boot loader in the MBR of the internal drive and use that to boot the USB drive (where it appeared as hd1 in GRUB). If you are using FireWire, chances are that your BIOS will not be able to boot the drive directly, and a little more work will be required.
Luckily, due to the flexibility of Linux, there is a fairly simple solution if you cannot boot directly (which is certainly the case in my situation, with a PCMCIA FireWire card!) You can perform the preliminary boot steps from a supported device such as a floppy drive, CD, USB key, or a tiny partition on the main drive, and then use the external drive for everything else.
Building the Boot Image
There are two methods we can use to boot:
• One-phase boot
The kernel boots, mounts the root filesystem, and continues initialization by calling the initialization scripts (usually /sbin/init)
• Two-phase (initrd) boot
The kernel boots, mounts an initial ram disk (initrd), performs further customizable initialization, then mounts the root filesystem and continues initialization (again, usually by calling /sbin/init)
Each of these methods has its own advantages and disadvantages.
In order to use the one-phase boot, we need to build a kernel that has all the drivers needed to mount the root file system built in (any other drivers can be built at modules that can be loaded from the root partition during normal initialization).
If we are attempting to boot from a very small device, such as a floppy disk, the best approach is to build a kernel with just enough drivers built in to allow us to mount the root external file system — and build everything else as modules. For example, I have the SCSI support, PCMCIA support, IEE1394, SBP, and like support built in, but everything else — including graphics card support, networking device support, and so on — is built as modules that are stored on the root partition (on the external drive), rather than on the floppy.
With a simple (one-phase) boot procedure, we should be able to build a kernel with the required support, put it on a floppy drive, install a boot loader on the floppy (I use GRUB, but there are other options, such as LILO), then boot with something similar to this (for GRUB):
kernel (fd0)/boot/bzImage root=/dev/sda1
This almost works, except for two problems:
1. Because of the fact that SBP support uses SCSI emulation, the emulated SCSI bus need to be “rescanned” in order to detect the disk and allow /dev/sda1 to be mounted. This scanning is performed with a couple of simple commands. Unfortunately, however, using a one-phase boot, we cannot run any commands until the kernel has finished booting, and the kernel cannot finish booting until the root file system is mounted — a classic Catch-22 situation. Thankfully, there is a patch available for 2.4 kernels that causes the SCSI bus to be scanned on startup (see Resources for more details). By applying this patch, I was able to have the external drive automatically detected by the kernel during bootup with no rescanning commands required. This leads us to the next problem.
2. There is a timing window within the kernel, which means that often, the kernel will proceed to try to mount the root device before it has had a chance to be properly detected or initialized. Again, there is a patch available for this (please see Resources for a link) that simply makes the kernel wait for a short period of time on startup, and retry if it fails to mount the root filesystem (to give the external drive time to be recognized).
By applying these two patches, I was successfully able to build a kernel on a bootable floppy disk, which would boot and then use the external FireWire drive as root.
The main problem with this approach is that it requires us to patch the kernel source — which is at best a pain (when new kernel versions are released) and at worst, a real problem (if the patches are not maintained to keep in step with the other changes occurring to the kernel).
It may have occurred to you that we can avoid these two problems if our BIOS supports USB or FireWire and we are booting directly. Unfortunately, this is not the case: while this method uses BIOS calls to access the disk during boot up, once the kernel begins to initialize, the BIOS is no longer used, and the disk is accessed using the kernel drivers — so the same problems are encountered.
As of kernel version 2.0.X, an interesting capability was added to the Linux kernel — the ability to use an “initial RAM disk” (or initrd) to give a two-phase boot process.
In a nutshell, the kernel is booted as normal; but instead of mounting the “real” root file system, a miniature root filesystem is created in RAM, and that is mounted. Any number of arbitrary steps can then be performed in this initial environment before the real root is mounted and we switch to using the real root and destroy the initial RAM disk.
This is useful in all sorts of circumstances, but for our purposes we will simply be using our mini environment to rescan the SCSI bus, wait for the external disk to be recognized, then switch to using this as our real root and continue to boot.
In order to use this method, we need to create two things, a kernel and an initrd image.
The kernel is just a regular kernel which has initrd support built in. The initrd image is a loopback filesystem image which contains our mini-root filesystem (this image can optionally compressed with gzip to reduce its size).
You can check the Resources section for more information on creating or customizing your own initrd image.
Within the initrd image, there is a file called linuxrc. This is the file that is executed when initrd is loaded, so make sure it has execute permission! For our purposes, the linuxrc is very simple:
Listing 1. initrd linuxrc
# mount the /proc filesystem
mount -t proc none /proc
# modprobe sd_mod
# modprobe pcmcia_core
# modprobe ieee1394
# modprobe ohci1394
# modprobe raw1394
# modprobe sbp2
# modprobe usbcore
# modprobe ohci-hcd
# modprobe uhci-hcd
# modprobe usb-storage
# loop rescanning the scsi bus + rerunning devfsd
until [ -e $REAL_ROOT ]
if [ $i -gt $retries ]
echo "Unable to mount real root ($REAL_ROOT) - Giving up!"
echo "Real root ($REAL_ROOT) not found, retrying ($i)"
echo "scsi add-single-device 0 0 0" > /proc/scsi/scsi
echo "scsi add-single-device 1 0 0" > /proc/scsi/scsi
echo "scsi add-single-device 2 0 0" > /proc/scsi/scsi
/bin/devfsd /dev -np
#umount /proc as it will be remounted by the normal init process
#now we simply exit, and the normal boot process should continue
All we are doing is loading the appropriate modules to support the external drive: they should be uncommented as required. (I built all the required support into my kernel, hence no modules are required.) We then loop, rescanning the SCSI bus (by echoing a command to a special file in the /proc pseudo-filesystem and calling devfsd) until the root device (/dev/sda1 in my case) is present. In my case, the emulated FireWire SCSI bus in question is 1 0 0, but it doesn’t hurt to try a few others — if you know which one you will be using, you can tailor the script. Also, if you have other SCSI devices (or emulated SCSI devices), the drive may have a different letter (for example, /dev/sdb1). And if you are not using the first partition on the external drive, you will need to use a different number (for example, /dev/sda2).
All we now need to do is copy the relevant files into the initrd image (you can mount the uncompressed image using the mount -o loop command). In particular, we need to ensure that we have the linuxrc file, all of the commands used therein, and any libraries that those commands rely on. The (unmounted) image can then be optionally compressed.
The kernel (bzImage) and the initrd image (initrd.gz) are then copied to the (bootable, ext3) floppy.
The final step is to install a boot loader on the floppy, and boot the kernel with the following options: kernel bzImage root=/dev/sda1 initrd=initrd.gz.
You should now be able to boot using the floppy: it will load the kernel from the floppy, load the initrd image into RAM, wait for the root device to be recognized, and then continue to boot as normal from there. From this point onwards, the floppy can be removed.
If a floppy is not appropriate (for example, if the machine has no floppy drive), then any device that can be booted by your BIOS can be used. Personally, I use a little 32Mb USB key for this purpose. Alternatively, if you don’t mind altering your internal hard drive, a small partition can be created there for more convenient booting.
• Martyn’s distribution of choice is Gentoo Linux .
• For tips on troubleshooting installation of USB mass storage devices on Linux, see Adding USB mass storage devices in Fedora Core .
• IEEE 1394 for Linux offers information and links on FireWire on Linux.
• For help in creating your own custom initrd image from scratch, see The Linux Bootdisk HOWTO . To tailor an existing initrd image to your own needs, see Brian Writes about His BOEL ( Linux Devices , 2002), which is based on tomsrtbt (see also BuildElToritoFloppyImage from the tomsrtbt wiki.
• ” Knoppix gives bootable, one-disk Linux ” ( developerWorks , February 2003) introduces this one-CD, bootable Linux distro.
• ” Access USB devices from Java applications ” ( developerWorks , September 2003) examines two projects which provide APIs through which Java applications can make use of USB devices.
• ” Burning CDs on Linux ” ( developerWorks , 2003) shows how to create your own .iso images using the full power of the command line.
• Previously on developerWorks , Martyn wrote ” Connecting KDE applications using DCOP ” ( developerWorks , February 2004), ” Postmortem memory profiling with PERL ” ( developerWorks , December 2003), and the Significant trace series of articles ( developerWorks , April 2003).
• Develop and test your Linux applications using the latest IBM tools and middleware with a developerWorks Subscription : you get IBM software from WebSphere, DB2, Lotus, Rational, and Tivoli, and a license to use the software for 12 months, all for less money than you might think.
• Download no-charge trial versions of selected developerWorks Subscription products that run on Linux, including WebSphere Studio Site Developer, WebSphere SDK for Web services, WebSphere Application Server, DB2 Universal Database Personal Developers Edition, Tivoli Access Manager, and Lotus Domino Server, from the Speed-start your Linux app section of developerWorks. For an even speedier start, help yourself to a product-by-product collection of how-to articles and tech support.