Red Hat: 'Yes, we undercut Oracle with hidden Linux patches'

But CentOS will live, CTO tells El Reg

By Cade Metz in San Francisco
The Register

March 4, 2011

Red Hat has changed the way it distributes Enterprise Linux kernel code in an effort to prevent Oracle and Novell from stealing its customers, making it more difficult for these competitors to understand which patches have been applied where.

Some have speculated that the change is designed to make it harder for Oracle as well as the open source CentOS project to build their own Linux distributions. But Stevens says this is not the case. He says the change is meant to hamper Oracle and Novell's ability to offer support to customers who are already running Red Hat Enterprise Linux (RHEL).

"We made the change, quite honestly, because we are absolutely making a set of steps that make it more difficult for competitors that wish to provide support services on top of Red Hat Enterprise Linux," Red Hat chief technology officer Brian Stevens tells The Register, before naming those competitors. "Today, there are two competitors that I'm aware of that go to our customers directly, offering to support RHEL directly for them...Oracle and Novell."

In essence, Red Hat is trying to hide information from these competitors that is essential to providing support for RHEL specifically. "What we're trying to impede is competitors that come to customers who are already running RHEL under subscription from Red Hat and saying 'Don't pay Red Hat anymore, pay us, and don't make any changes to your systems'," Stevens says.

He insists that the change does not violate either the letter or the spirit of RHEL's GPL open source license. "We were very careful that what we've done does not impede what our customers need to accomplish or what the community needs to accomplish." And he says that the change would not really hamper the development of other Linux distros, including CentOS.

"We haven't at all restricted CentOS's ability to grab source code and recompile it and clean-out trademarks and package it. It's just some of the knowledge of the insides that we're hiding," he explains. One longtime CentOS developer agrees.

"I'll not lose sleep on the matter," CentOS co-founder Russ Herold tells The Reg.

In November, with the release of Red Hat Enterprise Linux 6, the company released its kernel package with all patches pre-applied. "In the past, we distributed the kernel as a base file and then a set of add-on patches that accompany it. Then when you did a build, the build process automatically applied all those patches to the kernel file," Stevens says. "Now, we integrate those patch files directly into that kernel. We do the first part of the build process prior to distribution."

This was recently noticed by kernel-community member and LWN editor Jonathan Corbet [ http://lwn.net/Articles/430098/ ], who took issue with the change, calling Red Hat's package "obfuscated" kernel source code.

"Distribution in this form should satisfy the GPL, but it makes life hard for anybody else wanting to see what has been done with this kernel," Corbet wrote. "Hopefully it is simply a mistake which will be corrected soon." Others speculated [ http://www.h-online.com/open/news/item/Controversy-surrounds-Red-Hat-s-obfuscated-source-code-release-1200554.html ] that the move would undermine not only Oracle's Unbreakable Linux, but also CentOS. Both are based on RHEL.

CentOS is meant to be a RHEL clone. Whereas the compiled bits of Red Hat Enterprise Linux are only available under a Red Hat paid subscription, CentOS is completely free.

"The changes will make work harder for distributions such as CentOS, the community-built Linux distribution ... based on Red Hat's sources," H Online said. "CentOS is built from the RHEL source by a limited number of volunteers and Red Hat's change in policy will mean more work for them unless more volunteers or other companies step in and provide them with assistance."

We heard similar noises from an experienced Linux kernel developer. He said that Red Hat's change was like shuffling all the cards in an old fashioned Dewey Decimal library file system – the card you want is still there, but finding it is no easy task – and that this would cause problems for CentOS, which is an economic threat to Red Hat.

But CentOS founder Russ Herold insists the change is not a big issue. "Private local trial builds of the released RHEL 6 sources by me and others have proceeded with no major problems. I just do not see that the changes as some earth-shattering change. I just think [the patches will be] incrementally more difficult to figure out," he says.

"Nothing in Red Hat's new approach prevents a person from running a local version-control system, containing the pristine kernel at point A, and the Red Hat variant which we might call point B. Then one runs a 'diff' in that version-control system between A and B, and starts reading the diffs to see what is happening. Over time, both the pristine kernel, and the patched Red Hat versions will vary, and one will get a sense for which 'diff' parts matter, and which are cosmetic cleanups."

Other distros will not be affected, Red Hat's Stevens says, because the company distributes its kernel changes upstream as well. "The work that we've done should not impede companies from building their own versions of Linux and supporting those for their customers," he says. "All the code we deliver through RHEL is out there. In most cases, the changes that go into RHEL. We already distribute into the upstream kernel. We have an upstream-first policy, where we're developing openly and then later integrating into our tree and then delivering it. So it shouldn't at all impede the community or anybody that's in the business of competing on that."

Red Hat, he reiterates, is trying to keep RHEL-specific knowledge away from Oracle and Novell. With past RHEL kernel-code distributions, the patches mapped to articles in Red Hat's knowledge base. "It makes competitors do heavy lifting," he says. "If you want to support RHEL, remove the trademarks, and do some heavy lifting. If nothing else, it causes competitors to have to invest."

This won't hamper CentOS, he says, because CentOS isn't in the support business. "The code is still available. It's just more difficult to support the distro as a commercial entity. CentOS is not in the support business."

Oracle and Novell are in the support business. And whatever collateral damage was caused by Red Hat's change in policy, one thing is for sure. On some level, it will indeed be more difficult for Oracle and Novell to pilfer the company's customers.