Server Clinic: A Must-Have Book for Linux Developers
Limoncelli and Hogan’s Theory of Professionalism Touches more than just Admin
The best thing you can do for your servers and the programs you host to buy your own copy of Thomas A. Limoncelli and Christine Hogan’s The Practice of System and Network Administration. This new book, though aimed at administrators, is full of valuable lessons for developers, too. While it mentions Linux only once in almost 800 pages — and obliquely at that — the principles it explains will help make more sense of your work every day. Share your thoughts on this article with the author and other readers in the discussion forum /a> by clicking Discuss at the top or bottom of the article.
Other reviewers are already touting the value of The Practice of System and Network Administration (called simply Practice, from here on). Moreover, “Server clinic’s” domain is programming, not administration; so what’s the point of discussing it here?
Not Just for Administrators
There are a couple of reasons to offer Practice to you in this column. First of all, the book has hidden benefits that I want to amplify: namely, its applicability to development, as opposed to administration alone.
And secondly, future installments of this column will refer to several of the resources mentioned in this installment. Expect, the subject of the previous column , and Practice, will both make appearances in future installments, and it’ll be best to start off with clear understandings of them.
How Practice Differs
For most of the last decade, the two standard references for UNIX system administrators have been the Nemeth et al. UNIX System Administration Handbook and, just behind it, Aeleen Frisch’s Essential System Administration. When you have, say, a printer queue that’s in a bad state, you can reasonably expect to pick up either of these and find the commands that’ll solve your problem. More recently, several books, both good and bad, specifically address Linux system administration.
Practice isn’t like any of these. It both complements and to some extent supersedes all of them. Most administration books are full of code; they have sendmail.cf examples, lots of shell scripts, explanations of top’s command-line arguments, and so on. Practice targets a slightly more abstract body of knowledge, roughly the same one as Mark Burgess’s Principles of Network and System Administration. The point of Practice is to elevate the “how” of administration as a too-often clerical chore into the “why” of a professional practice.
Practice does this with essentially zero references to specific programs or products. In fact, it doesn’t even commit itself to a particular operating system line; its ideas apply equally well to MacOS, UNIX, Windows, or other enterprise operating systems.
Does that seem extreme to you, a book on administration without any examples of administration? Opinion on Practice is polarized; most readers conclude they either can’t live without it, or dislike it and wouldn’t accept a copy as a donation. I’m obviously with the former camp. Ultimately, all of Practice’s theory is practical.
Chapter 19 on e-mail, for instance, gives no direct help when you have a mail loop or a wedged queue, or need to filter spam, limit inbox sizes, archive traffic, or automate boilerplate disclaimers. Instead, it lays out in simple language the ideas of reliability, scalability, manageability, and so on, that must be preserved for any particular tasks to have value.
How can the authors — and I — insist on calling “practical” a book that does not have a single recipe for solving any particular problem? The point is that, if all you know are recipes, you’re no better off once you leave the tiny domain where the recipes work. Practice assumes that you have a way to find recipes. It takes on the task of complementing those recipes with deeper understanding, so you can cope with more general and novel situations. And always, in every chapter, Practice emphasizes automation, adherence to standards, good documentation, and clear communication as fundamental to getting administration right.
This focus leads to another difference between Practice and an earlier generation of books on system administration. Traditional books on administration have been “references,” somewhat in the sense of encyclopedias: when you want a solution for a problem with IP assignment, you look up the chapter that covers that topic. In contrast, what many of Practice’s readers are doing is reading the book straight through — narratively, “like a novel.” They study it for coherent comprehension of all the principles of administration, then turn to the book’s Web site (see “EverythingSysadmin” in Resources, below) for links to particular “how-tos.”
This is probably part of a trend. I suspect there will be less and less of a need in the future for a “Purple Book” (the “street name” for Nemeth’s book) chock-full of code and “man pages.” More and more of us expect to get such information online. What we still need on paper, though, are the explanations, analysis, and advice based on real experience that pack Practice.
Implications for Server Programming
So what’s the point for the “Server clinic” readers who develop server-side applications? First, Practice is a good read. It has the information at just the right level of abstraction — How is a server different? How do you institutionalize policies? What’s involved in deployment? How can you manage your own career? — that you need for your own work. Also, you can do your work better if you understand how administrators think (or should think).
Beyond that, Limoncelli and Hogan’s specific approach to administration emphasizes themes that benefit programmers. They explicitly label six keys to administrative design and support: simplicity, clarity, generality, automation, communication, and “basics first.” Programmers need these at least as much as administrators.
Limoncelli appears to organize his whole life around make files. He used version control on the text of the book itself, or more properly on the “macros” he and Hogan used to generate the text. The point is not that make is a great language; it’s not. It is, however, one proven technique for managing change and ensuring reproducibility. As a programmer, much of your challenge is to cope with unceasing change: new protocols, application programming interfaces (APIs), customer requirements, hardware constraints, and so on. Put yourself in charge of all this variability by automating and generalizing. Practice has the right attitude about this, and plenty to teach.
In response to questions I asked them, Limoncelli and Hogan offered a few tips on thinking about servers beyond what’s in this book. The former emphasizes that, “Server programming is different because change management becomes a requirement.” He also likes to distinguish services from servers: “Any idiot with a checkbook can buy a server, but it takes a lot of brains to run a successful service.”
Those slogans inspire this month’s homework assignment. Many shops monitor servers, with alarms connected to paging systems and other notifiers when a server goes dark. What you really want, though, is a constellation of services; servers are just a means to that end. The natural conclusion, after study of Practice, is to build a service monitor. Although there are a few commercial ones, and many organizations run “home-grown” versions, I haven’t yet come across one that satisfies me.
Let’s sketch how it should work: given a table of /etc/service-style assignments (18.104.22.168 and 22.214.171.124 should honor HTTP, while 126.96.36.199 offers SMTP), the monitor checks all ports on monitored hosts. It ensures that defined services are present, but that no other ports are open. Results are logged (Practice shows good understanding of the value of logging), and the table of assignments is maintained under revision control (version control is another good habit of Hogan and Limoncelli). Do this, and the results are sure to surprise you. You’ll uncover plenty of vulnerabilities and “stress points” in your network. I invite you to participate in the forumfor this column (clicking Discuss at the top and bottom of this article is another way to access the forum) if you want to go into more detail.
Next month, “Server clinic” will offer working code that might help in your programming. This time, the best help the Clinic can give is advice: read Practice.
• Participate in the discussion forum on this article. (You can also click Discuss at the top or bottom of the article to access the forum.)
• Check out the other installments of Server clinic.
• EverythingSysadmin.com is Limoncelli and Hogan’s Web site for the Practice book. By the time you read this, it should include hundreds of sample code fragments for common administrative tasks.
• O’Reilly maintains a page on Essential System Administration, 2nd Edition.
• Prentice-Hall publishes the UNIX System Administration Handbook.
• Mark Burgess keeps his own page on Principles of Network and System Administration.
• Find more Linux articles in the developerWorks Linux zone.