Enterprise distributions and free software

By Jonathan Corbet
LWN

March 7, 2011

The core Red Hat Enterprise Linux (RHEL) distribution is built entirely on free software (the supplementary repository includes a few third-party proprietary bits like Flash). The source for the distribution is freely available and freely distributable by any other party as long as trademark-relevant bits are stripped out. This freedom is exploited by free distributions like CentOS and Scientific Linux; it is also exploited by commercial competitors who are happy to charge money to support the distribution that Red Hat has made.

Naturally, it is the commercial competitors that are creating the most trouble for Red Hat, which invests a considerable amount of money into the development of both the software on which RHEL is based and the creation of RHEL itself. Companies like Oracle also contribute to upstream projects, but on a rather smaller scale, and they contribute little indeed to the development and maintenance of the RHEL distribution. Freed of that expense, these companies are able to offer support for Red Hat's distribution (or derivatives thereof) while charging much less. They are said to be actively courting Red Hat's existing customers, with a certain amount of success. It seems like a clear path to a successful business, even without the darker allusions, heard occasionally, that the real intent is simply to drive Red Hat out of business.

Given this situation, it is not entirely surprising that Red Hat has decided that it needs to respond; a company which makes life too easy for its competitors tends not to stay around for long. The company could try to make its distribution more proprietary to keep others from redistributing it. The use of proprietary kernel modules to this end is not unheard of, but it would be surprising indeed to see Red Hat take that approach. Proprietary user space components, like those used by Google to keep some control over Android, would have fewer licensing difficulties, but it still would be a major change for a (previously) free distribution. The good news is that Red Hat has resisted any such pressures - so far.

What Red Hat is doing is still seen by many as being something other than good news, though. Essentially, Red Hat is asserting stronger proprietary control over the metadata it possesses about the RHEL distribution. That metadata includes its "knowledge base" of problems and solutions, bugs and fixes, etc. What Red Hat wants to do is to exploit the advantage gained from having built and supported RHEL over all these years, and to make it harder for others to exploit that knowledge. The company is trying to compete by providing better support, and one way to have better support is to keep your competitors from making their own support offerings better through the use of your own information and experience.

Kernel source packaging

The move that brought all this to the community's attention was a change in the way that the source RPM file for the RHEL 6 kernel is packaged. Traditional source-packaging practice is to combine an unmodified upstream tarball, a set of distributor patches, and a script (the "spec file") which applies the patches and builds the software. The RHEL kernel package, instead, consists of a single tarball reflecting the kernel in its fully-patched form. There is a long list of patch titles in the spec file, but there is no other information which can be used to recover the specific patches which have been applied to the kernel.

The purpose of this change, evidently, is to make it harder for competitors to figure out why specific patches were made. Many changes applied to the RHEL kernel will have been done in response to specific customer problems; the understanding of those problems and how they were fixed will, at times, be useful in future support situations. By mixing all of its patches into a single, lossy tarball with no changelogs, Red Hat hopes to deprive others of this useful support information.

Will it work? The restriction of information in general should certainly deprive competitors of a useful resource, though the cynical might note that access to that information could be restored via an inexpensive low-end RHEL subscription. Whether broken-out patches for the kernel, in particular, are useful to competitors is not entirely clear. It may well be that Red Hat has irritated - and possibly made life harder for - the development community without bothering anybody else in any appreciable way.

Red Hat claims that the packaging change should not bother the development community at all, noting that the change had been in effect for four months before complaints were heard. One of the foundations of this claim is the fact that the RHEL "2.6.32" is far removed from anything called "2.6.32" anywhere else; the company once described [ http://press.redhat.com/2010/05/05/red-hat-enterprise-linux-6-kernel-an-overview-and-genealogy/ ] it this way:

The Red Hat Enterprise Linux 6 kernel includes numerous subsystems and enhancements from 2.6.34, as well as its predecessor versions. As a result, the Red Hat Enterprise Linux 6 kernel cannot be simply labeled as any particular upstream version. Rather, the Red Hat Enterprise Linux 6 kernel is a hybrid of the latest several kernel versions. And, as Red Hat provides regular updates over the lifecycle of the product, we expect that the Red Hat Enterprise Linux 6 kernel will incorporate selected features from future upstream kernels that have yet to be developed.

This kernel is a sort of Frankenstein's monster of fixes and backports; it is a distant relation to any sort of upstream kernel. Given that, Red Hat has said, most patches to this kernel do not apply in any useful way to the kernels the rest of us are working with. In other words, they claim, there is no value to the community in most of the RHEL kernel patches.

More importantly, though, Red Hat insists that it is a strong believer in working upstream. Any patches developed by Red Hat which have any relevance to upstream kernels are promptly sent upstream for merging. This line of reasoning suggests that there will not be anything of interest to the community in RHEL kernels; everything of value will have already been submitted for merging. The company's history backs up this claim; its history of contributions is matched by few others, and, by all accounts, the practice of contributing upstream is wired deeply into the company's culture.

That said, maintainers of the community stable kernels (and some other developers as well) have been heard to complain that Red Hat's practice is making work harder for them. Beyond that, though, this practice leaves the community with a bit of a bad taste in its mouth. The development of a significant fork of the Linux kernel has become more closed in a possibly ineffective attempt to support corporate interests. The fact that those corporate interests are in the personal interests of everybody whose salary is paid by that company and by everybody who benefits from that company's contributions perhaps softens the blow, but it does not make everything better. Many people are left wondering what the next step will be.

The enterprise distribution business

They may be right to wonder. The "enterprise distribution" business model, as it is currently implemented, is a strange distortion of the development community that supports it. It is said by some to violate the GPL, or, at least, to stretch its spirit to the breaking point; that is a subject which will be examined in another article. Of interest here is the fact that this model also led to the creation of kernels so divergent that its simple fixes no longer apply to the mainline. Your editor can't help thinking of Andrew Morton's classic keynote [ http://www.groklaw.net/article.php?story=20040802115731932 ] at the 2004 Ottawa Linux Symposium:

I have a few words for the Linux vendors here. It's understandably tempting for Linux vendors of various forms to seek to differentiate their products by adding extra features to their kernels. OK. But moving down this path does mean that there will be incompatible versions of Linux released to the public. This will, by design, lock some of our users into a particular vendor's implementation of Linux. And this practice exposes the entire Linux industry to charges of being fragmented. And it exposes us to the charge that we are headed along the same path as that down which the proprietary Unixes are deservedly vanishing... [A]s a person who has some responsibility for Linux as a whole, I see the perfectly understandable vendor strategy of offering product differentiation as being in direct conflict with the long-term interests of Linux.

The RHEL kernel, which is certainly differentiated from any other kernel out there, would seem to be well described by Andrew's words. Red Hat's move to lock down information about this kernel is an admitted attempt to increase customer lock-in; they are trying to make it harder to move to another provider of support. Is this good for Linux?

Corporate moves do not happen in a vacuum; Red Hat's competitors will likely come up with responses of their own. Perhaps they will just develop other sources for the information they need. If, instead, they adopt a strategy like Red Hat's, we could end up with a fragmented range of incompatible Linux distributions, each of which tries to lock in customers with its own combination of unique features and proprietary information. Veterans of the Unix wars can be forgiven for shuddering a bit at that thought.

An alternative might be to move away from the idea of "frozen" kernels in enterprise distributions. The truth of the matter is that these kernels are far from frozen; RHEL's "2.6.32" kernel has features from 2.6.38 - and probably beyond. Nobody wants a kernel which was set in stone years before, so these "stable" kernels get no end of features backported into them. The result is a combination which has never been seen before in nature and which may have stability problems of its own. It is also expensive to create and expensive to maintain.

Perhaps enterprise distributions should stick closer to mainline kernels and even move to new releases occasionally? Oracle's "Unbreakable Enterprise Kernel" [ http://lwn.net/Articles/406199/ ] looks like a move in that direction. Novell did something similar with its SP1 update to SUSE Linux Enterprise 11. This approach would seem to offer some real advantages: users would get new features and fixes from upstream, and distributors would not have to pay the cost of creating and maintaining a highly divergent kernel. There would also be less need to treat related information as proprietary; companies would compete on their ability to support current mainline software, rather than a vendor-specific kernel. That seems like an environment in which Red Hat, which does so much to create mainline kernels, could compete most effectively.

Of course, anybody who listens to business advice from your editor has a strong chance of learning all about the bankruptcy process in short order. Beyond that, this is not a new idea; it was discussed here [ http://lwn.net/Articles/239036/ ] in 2007. The ultra-stable enterprise distribution model seems to be in no danger of going away, in any case, even though some vendors seem to be experimenting with some changes. There are customers, it seems, so there will be vendors.

In the end, companies need to make money or they disappear. Red Hat has, so far, found ways to keep its stockholders happy while supporting the development community which makes it all possible. One can only hope that, as this market becomes more competitive, the companies involved will continue to find a way to balance those important interests; the alternative, certainly, will be destructive to both.

Copyright 2011