The VMware house of cards

VentureCake
Mike MacCana

August 13, 2007

Bloomberg believe VMware’s IPO today may the largest technology offering since Google. But doubts have been cast over the company’s supposedly proprietary ESX product, which may be derived from Linux.

Updated:


For those short on time, this article is not about VMware providing source for the Linux kernel used by ESX - this can already be downloaded from VMware [ http://www.vmware.com/download/open_source.html ].

It is about whether vmkernel, a proprietary blob that can only be loaded by a Linux kernel, can be considered a derived work of Linux.

VMware recreated the modern virtualization industry, resurging an idea last heard of in the 80s to allow multiple instances of an Operating System run simultaneously on the same hardware, on today’s commodity systems.

VMware has 55% market share [ http://weblog.infoworld.com/virtualization/archives/2006/07/survey_suggests.html ] and few competitors: Microsoft’s competing Veridian virtualization originally scheduled for this years Windows Server release has slipped to 2009, and Xen, supported by IBM, Red Hat and Sun, has yet to get into the hands of their customers en masse.

VMware owns this market right now, and even if VMware’s share shrinks - and nobody’s saying that - Garner estimates VMs will grow from 5 to 40% of new Operating System installs [ http://www.adtmag.com/article.aspx?id=18694 ] will be on VMs by 2009. Their IPO this week promises to be one of the biggest since the early 2000s.

VMware’s main product is ESX Server. Unlike desktop Virtualization products, ESX is installed directly on to the hardware. ESX is an OS designed only to run VMs and tuned to run them as fast as possible. The VMs live on storage shared between ESX servers, and it’s possible to move running VMs betweens ESX servers. Need some more processing power? Plug a new server in, install ESX, and drag some VMs there to start using it – without ever stopping the apps.

It’s simple to use, avoids all kinds of downtime, and consolidates hardware assets to better utilise them. It’s not cheap, though, and neither are the SANs made by VMware’s parent company EMC.

VMware leads a rapidly growing market with a proprietary product. That’s the basis of estimated value of at least 10 billion [ http://blogs.barrons.com/techtraderdaily/2007/05/02/emc-more-guesses-on-what-vmware-is-worth-a-virtualization-primer/ ] dollars. But is VMware ESX really proprietary?

VMware and Linux

Looking at an ESX server, you’ll find what looks like a Linux OS. This isn’t a secret – VMware call this the ‘console OS’. Is ESX server based on Linux? The VMware ESX FAQ provides an answer:

‘Q. Does ESX Server Run on Linux? On Windows?
A. ESX Server runs natively on server hardware, without a host operating system. ‘

Ok, so ESX doesn’t need a host OS. It’s pretty clear that ESX installs directly on the hardware without needing Windows, Linux or any other OS installed first – ESX itself is the OS. The question then is whether the ESX OS is based on Linux. The FAQ continues:

‘The ESX Server virtualization layer is a highly compact and efficient operating system kernel entirely developed by VMware for optimum virtual machine performance … ‘

Or, as the Wikipedia discussion for this issue states:
‘ESX’s kernel is definitely NOT Linux’

Let’s hear it from the horse’s mouth. VMware employee Jim Troyer further states [ http://en.wikipedia.org/wiki/Talk:VMware ] in the discussion for this article:
‘ESX Server’s vmkernel is definitely not Linux or derived from Linux. ‘

A kernel is the core of an Operating System. Kernels do basic things like give memory and CPU time to apps. Windows has a kernel called NTOSKRNL, Linux distributions use the Linux kernel. And apparently VMware ESX uses VMWare’s own kernel, called VMkernel.
So ESX does not use Linux. Matter settled.
Or does it?
Back to that FAQ:

‘ESX Server also incorporates a service console based on a Linux 2.4 kernel that is used to boot the ESX Server virtualization layer. It also runs ESX Server administration applications.’

The OS with Two Kernels

So according to VMware, ESX actually has two kernels that run directly on hardware - the vmkernel, and a Linux kernel. This sounds a bit odd, given a computer can only run one kernel at a given time – otherwise which one determines who gets access to the CPU, memory, and other hardware?

But VMware explain there is a Linux kernel, but it’s only used to boot the other kernel. In this interview [ http://www.rtfm-ed.co.uk/?m=200612 ] Michael DiPetrillo, VMware’s Senior Systems Engineer, states at 4:44 ‘In the 2.xs (the Service Console) was home to a reasonably large number of things, bootloader, it had it’s own networking‘.

Jim Troyer, Senior Product Manager of the VMware Technology Network, mentions on Wikipedia’s discussion page:

‘A good citation for the relationship between the vmkernel and the Service Console is Chapter 2 of Oglesby, Herrod, and Laverick’s ESX Server Advanced Technical Design Guide‘ [ http://www.vi3book.com/esxatdg_ch2.pdf ]

Which also states that Linux ‘acts as the bootstrap for the VMkernel’ (bootstrap is another term for bootloader.

A Big Bootloader

A bootloader is a small piece of software used to start a kernel. Every OS had one:

But wait a moment: this might be usual, but it’s not unique - Novell Netware used the similarly large MS DOS kernel to load itself. And the document goes on to state that ‘Once the COS has loaded ESX, the VMkernel will assuming the role of primary operating system… then load the COS and several other “helper worlds” as privileged VMs.’

Hrm, it seems that Linux is used to load vmkernel, and then vmkernel virtualizes Linux.

[ http://video.google.com/googleplayer.swf?docId=-8007161406494547506&hl=en-AU ] Let’s see how that’s done.

The VMKernel Driver and Linux

vmkernel is loaded by vmkmod, which itself is loaded from the ‘vmware’ service at boot time. Linux services are started via scripts. The specific line in the vmware service script that loads vmkernel runs:

insmod vmkmod

‘insmod’ installs (loads) Linux driver modules into memory. It doesn’t load anything else but driver modules.

When ESX boots, Linux is ESXs kernel: vmkmod is a driver, and vmkernel a large piece of software loaded by that driver that functions in kernelspace.

Linux isn’t DOS, and ESX isn’t Netware

The license for the Linux kernel is quite different to the licensing for DOS that allowed Netware to use it for a bootloader. Proprietary drivers for Linux kernel have an interesting licensing situation. Unlike the license for higher level libraries, which allow those libraries to be used by both Open Source and proprietary software, the license for the Linux kernel specifies that software based on the Linux kernel must be licensed under the same terms.

As a result of that, most drivers for Linux are Open Source: whether created by coders working for Linux companies like Red Hat or IBM, hardware companies like Intel, or enthusiasts who want to make sure their hardware works well under the OS, most Linux drivers are Open Source.

But some proprietary modules do exist, and they do that on one premise: Linus Torvalds (the copyright holder for the Linux kernel) has repeatedly stated that he doesn’t consider drivers ported from other operating systems to be derived works of Linux. After all, if something can load without Linux, it can’t really be considered a derived work. NVIDIAs Linux video card driver shares most of its code with it’s Windows equivalent – NVIDIA’s Linux driver is proprietary, and according to Torvalds, that’s fine - since the driver clearly doesn’t need Linux to start it.

Witness Alexandro Rubini’s interview with Linus Torvalds, published in the September 1998 issue of Linux Gazette.

Alessandro: ‘What is your position about the availability of Linux modules in binary-only form?’
Linus: ‘I kind of accept them, but I never support them and I don’t like them. The reason I accept binary-only modules at all is that, in many cases, you have, for example, a device driver that is not written for Linux at all, but, for example, works on … other operating systems, and the manufacturer … wants to port that driver to Linux. But because that driver was obviously not derived from Linux (it had a life of its own regardless of any Linux development), I didn’t feel that I had the moral right to require that it be put under the GPL, so the binary-only module interface allows those kinds of modules to exist and work with Linux.

That doesn’t mean that I would accept just any kind of binary-only module: there are cases where something would be so obviously Linux-specific that it simply wouldn’t make sense without the Linux kernel. In those cases, it would also obviously be a derived work, and as such the above excuses don’t really apply any more, and it falls under the GPL license.’

VMWare’s desktop products don’t use the driver, as they don’t need the advanced storage functions that the vmkernel does. The only way to load vmkernel is by vmkmod, a driver that requires Linux. Take away Linux and there’s no way to load vmkmod and start vmkernel.

It’s possible to ditch, remove, or crash Linux after vmkernel has virtualized it - but you wouldn’t be able to get to that stage without Linux being used to load vmkernel.

But wait - perhaps vmkmod might be a derived work, but the vmkernel it loads is stored in a separate file. Lets check out Torvalds again, from 19 Oct 2001 [ http://www.google.com.au/search?q=torvalds+%22I+personally+consider+anything+%22&ie=utf-8&oe=utf-8&aq=t&rls=com.ubuntu:en-US:official&client=firefox-a ]:

‘I personally consider anything a “derived work” that needs special hooks in the kernel to function with Linux (i.e., it is not acceptable to make a small piece of GPL-code as a hook for the larger piece), as that obviously implies that the bigger module needs “help” from the main kernel.’

ESX Server 2.5 architecture, from the VMware Advanced Technical Design Guide. VMkernel uses the vmkmod hook to work with Linux (Console OS). According to Linus Torvalds, such applications are derived works of Linux.

So why hasn’t this issue come up before?

“A derived work of the Linux kernel that’s not legally redistributable”

Christopher Helwig is the Linux SCSI storage maintainer, and one of the top 10 contributors to the Linux kernel. He has been pursuing VMware over the issue for a year. Why hasn’t Hellwig pursued VMware legally? Hellwig answers in a post to iSCSI mailing list [ http://groups.google.com/group/open-iscsi/browse_thread/thread/d06c452c8a2958e6 ]:
‘VMware uses a badly hacked 2.4 kernel with a big binary blob hooked into it, giving a derived work of the Linux kernel that’s not legally redistributable. I unfortunately don’t have enough copyrights on that particular version to sue them. I do object to use of any open-iscsi code of my origin to be used with it, though.’

Do VMware know about this? They should: they employ a number of developers who work on the Linux kernel. And they do: witness the following from last August - replying to a post from VMware’s Zachary Amsden to the Linux kernel mailing list, Hellwig wrote to Zachary [ http://lkml.org/lkml/2006/8/4/112 ]:

‘Until you stop violating our copyrights with the VMWare ESX support nothing is going to be supported. So could you please stop abusing the Linux code illegally in your project so I don’t have to sue you, or at least piss off and don’t expect us to support you in violating our copyrights.

I know this isn’t your fault, but please get the VMware/EMC legal department to fix it up first.’

Zach did not post a reply, and no mention of the topic has come from VMware anywhere, despite Hellwig’s continued claim on the list in front of VMware staff. From May 6 this year [ http://lkml.org/lkml/2007/5/6/40 ]: ‘VMware… are on the almost black side of the illegality’

Is Hellwig right, and is VMware ESX a derived product of Linux? Unless vmkernel can be loaded without the Linux kernel, it would appear so. VMware was developed from another, long ago OS created as a research project, but it’s unclear whether vmkernel was ported from that OS or rewritten as the Linux-requiring binary blob.

What’s more of an issue is that VMware had these serious questions posed directly to them a year ago, repeated in a public forum many times since, but have yet to respond at all.

ESX is VMware’s main product. EMC are hoping to raise 10 billion dollars from its promise. They have a duty to defend it, and clarify the licensing situation for all concerned.

11:44 pm

Alan Cox [ http://www.linux.org.uk/ ]

August 15, 2007

I’ve no official opinion on Christophs view at this point but let me correct one important detail. Linus has said he’s okay with some binary modules. However he owns but a few percent of the copyright and others, me included, have quite specifically said we are not. Its entirely possible nvidia’s code is not a derived work - the caselaw is very limited, but the only question that matters is “is this a derived work”, not “what Linus thinks”

09:14

Zachary Amsden

August 18, 2007

The VMware vmkernel is most definitely NOT a derived work of Linux.

I have seen three arguments that ESX is in violation of the GPL. None of them hold any water in my opinion. As for an official statement, we may make some statement in the future. What I say now is merely my professional opinion backed by hard facts.

The first argument I have seen is that ESX uses a Linux distribution during operation. It certainly does. This old Linux kernel, which is freely available under the GPL, runs an entity called the console OS. This is based on a standard Linux install. You can watch it in operation in the ESX boot video above. It uses RPM based packaging, and many userspace programs from existing Linux distributions in addition to many proprietary userspace programs. There is no GPL violation in using proprietary software on a Linux operating system. Any argument that ESX is in violation of the GPL for relying on said Linux OS falls flat on its face, so much so as to publicly humiliate the accuser.

First, many complex (proprietary) systems rely on Linux in one form or another, perhaps running on a different machine in a corporate IT network, perhaps running inside a virtual machine in a single box solution. This is no argument for GPL violation. Second, these proprietary software programs are simple UNIX programs, complying to a well established system interface. They could just as well be run on FreeBSD, Solaris, or even on Windows using UNIX emulation libraries. The only thing that matters, as Alan Cox points out, is whether the derived work argument applies. To the third and final point. Since the Linux kernel specifically grants exception to userspace programs, and allows for the creation of proprietary software which relies on and depends on the Linux kernel, there is no possible argument that any of the ESX components which run in userspace on the Console OS are derived works of Linux.

The second argument that I have seen is that the vmkernel itself (the core part of ESX) is a derived work of Linux. The argument goes - with variations - since the vmkernel is loaded by Linux, it is a binary module, and thus a violation of the GPL; alternatively, since the binary vmkernel module uses Linux symbols (standard hooks that are exposed to other binary modules), it must be in violation of the GPL. I will not debate the contentious issue of binary modules here. But I will say this argument is wrong on at least three counts.

First, the vmkernel is not a Linux kernel module. The vmkernel is a completely isolated and separate piece of software which is loaded by a module called vmnix. The vmkernel has no knowledge or understanding of Linux data structures or symbols, and as a necessary result, does not depend on the Linux kernel for any services whatsoever.

Second, the vmkernel does not run inside or as part of the Linux kernel. It simply takes over control of the CPU and switches into a completely alien operating mode - one where Linux itself no longer exists. The former kernel used to boot the systems is still alive, but to switch back to it is a complex and involved process, similar to the well-defined copyright boundary of switching between two user processes, which are completely separate programs, running in their own address spaces. The vmkernel and the console OS Linux kernel are two completely separate entities, and the process of going from one to another is even a stronger separation than that given to user processes - more like rebooting the processor and re-creating the entire world. There is also no reason this can’t be done on any other operating system, in fact, this is exactly what we do on Mac OS and Windows to run our hosted products. An extremely strong argument for an independent, rather than a derived work.

Finally, there is no source code from Linux present in the vmkernel. The most clear way to become a derived work of another code base is to copy and include part of the source. ESX does no such thing. None of the code in the vmkernel is derived or copied in any way from GPL code.

The third argument I have seen that ESX violates the GPL is complaints about the way the driver layer operates (”that VMware are loading a GPL licensed scsi module … into vmkernel”). As usual, I will dissect and dispel the claim that this violates the GPL in three co-tangential ways.

First, the vmkernel, just like any other kernel is free to run and load any other object code it desires. You can write a driver or any other piece of software under any license whatsoever, and it goes nowhere towards enforcing anything about the execution environment of said software unless it is explicitly stated in the license. Copyright merely gives the authors control over how a work is copied and distributed, not how it is run (as in, you can’t copyright your DVD under the Foo. Corp license, which gives the user permission only to play said DVD on Foo. Corp DVD players; you can however prevent people from copying your DVD and selling it unless they agree to run it only on Foo. Corp DVD players). The GPLv2 does not make any such assertions about the execution environment of the software; indeed, if it did, the following scenarios would be ludicrously possible:

1) GPL software could not be run on proprietary operating systems
2) GPL kernels could not be run on proprietary hardware, which includes software BIOS components
3) GPL drivers loaded and run in non-GPL kernels would taint those kernels and bring them under the scope of the GPL.

This concept is simply ludicrous. It would mean that loading a GPL driver in Windows would subject the Windows kernel to the GPL. Check out the OSSWin project - they have GPL drivers for Windows.

One might ask, if the vendor of a proprietary kernel were prohibited from distributing GPL software along with their proprietary product. The answer is no. In fact, in this scenario, the GPL actually specifically exempts the distributors from any such requirements. To quote, verbatim

“In addition, mere aggregation of another work not based on the Program with the Program (or with a work based on the Program) on a volume of a storage or distribution medium does not bring the other work under the scope of this License.”

Again, we see the only valid argument is whether another work is a derived work of a GPL licensed work. To tackle this argument head on, one must look at the question of derived work.

For my second response to the third argument. It is accepted among the mainstream that drivers ported from other operating systems to run on Linux are not subject to the GPL; indeed, Linux provides services to these binary driver modules that are similar among all operating systems - facilities to set timers, plug into event chains, register new device or filesystem types. Use of these type of basic facilities does not imply a derived work. And by extension, nor does emulation of these basic facilities imply a derived work. Linux has established a well defined, binary interface through which driver modules and the kernel may interact, with well defined copyright separation. In fact, in newer versions of Linux, some more internal Linux specific functions are exported only to modules which are licensed under the GPL. If there is such a clear and well defined boundary of copyright separation, there is no way to reason that being on either side of this interface creates a derived work of the other side. So emulating Linux, Windows, or any other operating system interfaces which have this clearly defined separation is not a derived work argument.

This has been constantly found to be true in similar cases, and in fact was vehemently argued correct by many in the open source community when faced with the threat that Linux was a derived work of SCO UNIX because the user / kernel interfaces of Linux were based on the traditional UNIX interfaces (and that argument is well settled). Linux itself crosses the same lines as ESX’s vmkernel. Linux can emulate the Windows NDIS layer to load network drivers, and has all sort of shims to allow loading and running binary firmware and drivers. Does this make Linux a derived work on Windows? Certainly not. And if your argument is about to be — well, ndiswrapper is technically ok, because it is not part of the kernel, it wraps Windows drivers inside a module, which is separate from the kernel — think again. The vmkernel can load drivers from Linux and Windows, and it does so by wrapping those drivers inside a module, which is separate from the vmkernel. You cannot simply choose an position and apply it only when it is convenient. In this case, there is no way for Linux to take any other stand.

For my third desconstruction of this, keep in mind that copyright only applies to the copyright holders. VMware distributes, in binary form, old drivers, which for the most part consist of drivers written by the original hardware manufacturers themselves. Not only does the GPL explicitly give us permission to redistribute these drivers in binary form, and not only are we free from any taint of derived work, but those vocal advocates such as Christoph, who are so offended by this, do not even own any copyright on the code we are distributing.

I am not a lawyer, and this is not an official statement, simply my plain and common sense reasoning. I would not stand for any GPL violation and I feel compelled to defend my position as a respectable member of the open source community. So in summary, I most sincerely believe:

1) There is no argument that depending on a Linux console OS makes any part of ESX subject to GPL that is not already opene sourced. We have open sourced those GPL components of the system which we have modified and given the changes back to the community.

2) There is no argument that loading the vmkernel by bootstrapping it with Linux makes the vmkernel subject to GPL. In fact, in the funniest counter-example, Linux itself used to be boostrapped under MS-DOS by running the command loadlin (note the 7 character name, since DOS would not allow enough characters for load-linux).

3) There is no argument that distributing GPL binary drivers makes any piece of software distributed with them or using them subject to GPL. In fact the converse appears to be explicitly stated in the GPL itself.

So I find that there is no compelling evidence nor any doubt in my mind whatsoever that VMware is most definitely not violating the GPL in any fashion, nor even infringing on any gray areas, and that those vocal complaints we have seen are completely unjustified.

Zach

Ed: Zach, thanks for posting.

1. That argument hasn’t been made anywhere in this article, which mentions that the source to the kernel used by the Console OS is available for download in the first paragraph. Neither is this the argument made by Hellwig. However I do appreciate and understand why you would address it, as many people are still confused by it.

2. VMkernel is started by the vmkmod hook - ie, from a Linux kernel module. As mentioned in the article, what happens after that module is loaded is irrelevant - the point is that vmkernel is started from Linux. The loadlin example is also irrelevant, as the userspace land of MS-DOS does not have the same copyright restrictions as the kernel space of Linux. Could you specifically address this? If vmkernel is not ported from another OS and indeed depends on Linux to load (not run), by Linus Torvalds’ logic vmkernel is a derived work. Whether vmkernel continues to depend on Linux after it has loaded or not is irrelevant, if Linux copyright had to be infringed to start vmkernel in the first place.

Could you clarify whether you are stating vmkernel has been ported from another OS, and which state any OS vmkernel was running on prior to being ported to Linux? This would happily address the issue.

3. Again, that argument hasn’t been made anywhere in this article. VMware can emulate the Linux driver API all it wants. However, again it’s understandable why you would address it.

08:33

Copyright 2007 http://www.venturecake.com/the-vmware-house-of-cards/