It's a Cycle of Life Thing: Managing Linux Releases

Doing the Penguin Dance

One of the things I used look forward to about twice a year was a new release of Red Hat (RH) Linux. I'm pretty conservative on my main desktop system, where I write my articles and do my main development, and tend to run a pretty out-of-the-box configuration. I leave my hotrodding for my experimental system, where it doesn't matter if I accidentally trash a disk. Up until recently (say, Red Hat 7.2), upgrading my system wasn't a major hassle, because the overall rate of change of packages was relatively small and the improvements in system performance, as well as the look and feel of the user interface/experience, got to be more fun with each new release.

The release of the 2.4 series kernel made a lot more functionality available to developers, and the Linux community has taken advantage of it with wild abandon. With the release of Red Hat 7.3 (and SuSE 8.0, and most other Linux distributions from about mid-2001), I noticed a sudden bump in the number of applications available and a radical change in the dependencies in any given distribution, release after release.

For many users, upgrading a system or two is not a major hassle; they put in the CD-ROMs and go. However, the picture gets a bit more interesting if you're an enterprise using Linux for development or, dare I say it, as the corporate desktop standard for your users. A small change to a critical library (like glibc or gtk) can cause applications to break. A change in application functionality sprung on unwary users can cause support headaches galore. One of the great things about Linux is how quickly new software and services are becoming available--but it's also turning into a major headache for enterprise IT support people who have to keep track of all of these changes and software versions. Let's face it, the system itself (and its OS) is in reality only a small part of the Total Cost of Ownership (TCO) and one reason why end-users would rather fight than switch when it comes to operating systems and applications.

One of the selling points and competitive levers for commercial Linux distributions is how many applications they bring to the table for their various constituencies. For power programmers, it's all about the depth of the development environment. For those deploying web services, it's the degree of support for Apache and related technologies like Tomcat. Most recently, for the desktop user, it's been the ever-growing number of applications available on the desktop, like StarOffice and OpenOffice.org.

It's interesting to note that in the RH7.2 release there were perhaps 75 desktop GUI-based packages and utilities available. Under RH7.3, and now 8.0, a default, standard desktop system includes over 200 (yes, 200!) unique applications installed and available for the GNOME and KDE desktop environments. All of these programs have some level of dependence on other parts of the system, and as the depth of applications grow, so does the level of dependence.

What's Really in a Linux Distribution?

Before Red Hat developed the first serious package manager--the Redhat Package Manager, or "RPM"--for Linux, there was a list of packages that you needed to install in order to make a bootable system. This list consisted of the Linux Kernel, glibc, various "userland" utilities (e.g., the GNU binutils), networking tools like ftp and telnet, and a few shells. Everything else was optional. Fast-forward to 2003: the smallest possible automated installations (in terms of packages) from the major Linux vendors look like this:

Wow, that's a lot of stuff isn't it? That doesn't include X11, or any user-level applications like OpenOffice.org or even freecell. If you count just the basic packages that make up X11, window managers like GNOME or KDE, support libraries, and user-level productivity tools, you can add up to 100 more packages to this list. This isn't a bad thing, but it's clear that a minimum usable, networked system comprises a lot of software.

Related Reading Learning Red Hat Linux By Bill McCarty

Imagine that you run a large enterprise and support not only the base operating system across hundreds or thousands of machines, but also end-user applications ranging from office productivity to CRM, or order management, or customer-service applications. It also makes for a lot of packages for an IT manager to support.

What do Enterprises Need?

One of the hardest things for any enterprise to deal with is change. Back in the days of the Internet bubble when we thought (wrongly) that old paradigms didn't matter, no amount of disruption to the workflow of an organization was too radical. Now, we understand that change is good, but unexpected change can damage the ability of people to work together and get things done. In deploying IT solutions, the two hardest parts of any deployment are 1) consistent equipment configuration and 2) training the people who will use it in the software packages they need to get their jobs done.

Some people might suggest that this is a non-issue and that services such as the Red Hat Network allow IT managers to tame this complexity with automatic updating. This is true, provided you always want Red Hat's suggestions. What if installing a new version of OpenOffice.org or some other application would mean that your staff needs to be retrained, or a critical feature changes in some way that negatively affects your operations? What happens then?

The ability to upgrade systems in a logical way that minimizes system-level changes (except for bug fixes) and allows the enterprise to keep using older/existing versions of applications to help manage user expectations and constrain retraining is extremely important.

In the days of mainframes and mini-computers, upgrades were a lot more logical and well-contained because an upgrade applied to a system shared by a large number of users. With workstations and PCs, even, a small upgrade can cause issues far beyond the scope of a single package, as anyone who has ever had the pleasure of maintaining even a small gaggle of Microsoft Windows-based systems can attest. One of the things that made the non-PC world manageable was that, for the most part, hardware vendors made a very clean separation between the operating system and the applications.

Back in January at the NY LinuxWorld Expo, it was interesting to hear a developer from Dell opine onstage that the current "problem" with Linux distributions was all about the number of releases per year and that "things were settling down to a nice 18-month release cycle." I contend that the release cycle of distributions as a whole is less of a problem than the lack of a clean interface for layering products onto Linux systems. There can be no argument that the speed of Linux development--especially from a security perspective--is a good thing; a cleaner mechanism is needed in order to support multiple versions of software to help IT managers keep end-user-visible changes and disruptions under control.

Chuck Yerkes, a consultant specializing in Unix and sendmail, suggests that the Linux world could benefit from the experiences of the BSD world, where almost all non-core functionality is installed in a non-root tree. Instead of a package like OpenOffice living under /usr/bin, it would live in /packages or a similar directory that allows multiple versions of packages to be installed easily. Of course, this "re-rooting" can be done now with RPM and other package managers, but it has to be done manually.

One way to look at this is that, in the old parlance of Digital's VAX/VMS, the core of the operating system itself is a separate product from the "layered products" used by end users. It should be possible to install or upgrade the core of the Linux operating system without forcing the upgrade of the layered packages. In fact, it should be possible for a Linux system to support multiple versions of a package along with whatever underlying libraries are needed, without causing application dependencies to break. This might seem to add major complications to Linux distributions, since vendors would have to track and understand more details about every package they include in their distributions, but it would make the adoption of Linux by large and medium-sized organizations a lot easier by making their support mechanisms more clear-cut and predicable.

A Call to Distribution Vendors and the LSB

It seems to me that it's time for Linux distribution vendors, backed by the Linux Standards Base (LSB), to break the core OS out from the rest of the "layered products" that make up the bulk of their offerings. Make all of the end-user applications into a separately-selectable set of installable packages and develop a way to allow for multiple versions of packages to coexist on the same system. In the old days, under mainframes and minicomputers, this was done with "logical names" under VMS and virtual directories under VM and other OSes. The BSD world seems to have navigated this shoal; it shouldn't be to hard for Linux world to do so, either.

Acknowledgements

Deep thanks to Steve Jones of Crash Computing, Chuck Yerkes of SNEW.com, Stephen Degler, Andrew "Bobcat" Templin, and Richard Rognlie, "Sr. Lackey" for Gamerz.NET Enterprises, for helping me sift through bits and pieces of various Linux distributions and offering suggestions on how maintaining large Linux installations could be better handled.

David HM Spector is President & CEO of Really Fast Systems, LLC, an infrastructure consulting and product development company based in New York

Read more Linux in the Enterprise columns.

Return to the Linux DevCenter.

Good idea? Not taking it far enough? What's the best way to manage Linux upgrades?
You must be logged in to the O'Reilly Network to post a talkback.

Showing messages 1 through 11 of 11.

• The enterprise can layer the OS
  2003-03-25 05:04:11  anonymous2 [Reply | View]
  
  
  An enterprise can have RedHat in a minimal install using Kickstart. Then separate functionality can be layered over that for a desktop, document server, RDBMS server, DNs server, ... . Then there are only a few platforms that scale across the organization.
  
  This is very different from the 'personal computer' paradyme and much closer to the 'terminal of a mainframe' model. But until the enterprise limits the number of platforms that need to be tested and debugged, no maintenance effort will be able to scale.

• Breaking dependencies
  2003-03-25 01:31:48  anonymous2 [Reply | View]
  
  
  The original idea behind Unix-tools was to be
  small and stupid, only together the tools formed
  advanced functionality.
  The distributors work to provide coherence and
  see it as their product to maintain dependencies.
  That could be a key factor to find a solution
  on the "must update" problem.
  
  The older X11-GUI continues this tradition.
  The KDE/Gnome are monolithic mastodonts, except
  that you can actually run some of the modules
  single without windowsmanager and desktop. E.g.
  Open Office, big in itself, works nice in the
  heterogeneos environments - if you get it directly
  from OpenOffice.org, that is.
  
  RedHat has much flexibility in RHN-services, but
  of course you must invest some time there if you
  do not want to do up2date everything.
  
  Donald Axel, systems consultant.
  donald_j_axel@get2net.dk
  

• GENTOO
  2003-03-24 04:58:13  anonymous2 [Reply | View]
  
  
  doesn't the Gentoo distro do this kinda thing already? and I think this is just another arguement for the *BSD's

• i think its fascinating
  2003-03-23 20:16:49  patrick_darcy [Reply | View]
  
  
  red hat pr has put out the statements
  
  we own linux
  
  we hate mandrake
  
  if we wanted to kill kde we would kill kde
  
  and u put out a nice article drawing the
  people in.
  
  fascinating, keep up the good work
  
  
  
  

• On the right track
  2003-03-21 06:46:31  anonymous2 [Reply | View]
  
  
  I am going to try the following when I step up to Mandrake 9.1 . I am going to put /opt in its own partition and create a subdirectory /downloads. There is a program called STOW which I have read about recently which can do essentually what this article calls for. Unfortunately, it will not handle rpms but must use tgz source. Hopefully this cure my problem of wiping out my downloads every time I upgrade.

• /opt
  2003-03-21 06:14:34  anonymous2 [Reply | View]
  
  
  The FHS specifies "/opt" for the same function as BSD's "/packages". Now, if only the distributors would start to use it. I can understand not using it for GNOME and KDE, but OpenOffice and other end user applications should.

• packages vs OS
  2003-03-19 11:23:28  anonymous2 [Reply | View]
  
  
  I think the FreeBSD package system would be GREAT under Linux. It is indeed difficult to track what app dependencies are broken by a new os release.

• Re: It's a cycle of life thing
  2003-03-18 14:00:10  anonymous2 [Reply | View]
  
  
  Look hard at Debian - the other distributions
  have been forced into an "enterprise" Linux
  version at high cost which changes relatively
  slowly. For all the grouching about the fact that Debian
  Debian releases slowly, there are always three
  levels of packages available (stable,testing,unstable)
  ble) and you can mix and match. Want gcc 2.95,3.0,3.2 simultaneously
  can do.
  

• file system hierarchy
  2003-03-18 13:14:20  anonymous2 [Reply | View]
  
  
  You mentioned the BSDs being superior to Linux in this regard. FreeBSD basically puts *everything* that's not part of the core system in /usr/local. NetBSD does it better, they put stuff in ports (which they call pkgsrc) in the /usr/pkg tree, while leaving /usr/local clean for the sysadmin to add custom programs that aren't installed via a potentially upgrading pkgsrc. This makes upgrading of software a lot less disruptive under NetBSD than it can potentially get with FreeBSD; a sysadmin or user can rest assured that anything they added to the system is not interfered with, and if they want it to be interfered with (upgraded via "make update" in pkgsrc or upgraded via build.sh in /usr/src) it gets modified within it's own domain.

• encap can do some of this
  2003-03-18 11:06:21  anonymous2 [Reply | View]
  
  
  There are various projects that used the encap system. I'm not sure any are still around. Encap is a package management system and philosophy that takes a lot from the NextStep system. Each package is installed into its own directory. Different versions have different directories. Search Google for encap. I've never used it, but it appears to solve some of the problems raised in the article. Of course now you need a distro based on encap...

• Zlib again
  2003-03-18 09:17:06  anonymous2 [Reply | View]
  
  
  The only way to avoid dependencies is compile them together as one monster program. Then you are open to the zlib bug again and have to update multiple programs. Whereas a dynamic link to the library means one thing to update not many (including those that few people know about).
  
  This comes back to the benefits of code reuse. Often these new packages require new routines and it is the case that old programs break because of lack of backwards compatibility (either due to change in new libraries or original code).
  
  If you rely on a major distro, just wait for the next release especially if it is a major upgrade such as KDE 1 to 2 to 3 etc.