Date Thu, 03 Aug 2006 03:14:21 -0700
From Zachary Amsden <>
Subject A proposal - binary
I would like to propose an interface for linking a GPL kernel, specifically,
Linux, against binary blobs. Stop. Before you condemn this as evil, let me
explain in precise detail what a binary blob is.
First, there are two kinds of binary blobs. There are the evil, malignant
kind, that use manipulative and despicable techniques to subvert the GPL by
copying code into Linux and, the most evil kind, which uses a GPL wrapper to
export GPL-only symbols to the binary blob. This is unconditionally wrong. I
do not support this kind of use. These evil blobs are used to lock people into
a particular type of protocol or proprietary hardware interface. In my
personal opinion, they should be unconditionally banned, or at least phased out
rapidly from any GPL kernel. I have been frustrated by this in the past, where
binary filesystem modules would not allow me access to AFS when upgrading to a
new kernel version. I do not wish that on anyone.
But there is also another kind of binary blob. These are not the evil, nasty
subversion type blobs, but a benign kind that actually exists in binary form
not to close off choice, but to open it. This is exactly the kind of binary
interface we are proposing with the VMI design. Only with a specific
ABI can you guarantee future compatibility. This is exactly the same thing
I believe some hardware vendors are trying to do. When you have a fairly
complex interaction with the hardware layer, you have a lot of code which
suddenly becomes hardware dependent. When that hardware is actually changing
rapidly, you have a serious problem keeping released drivers for that hardware
in sync. Software release cycles are becoming much longer, and delivering new
capabilities to average consumers outside of that software release cycle is a
very difficult problem to solve. As a result, vendors build some smarts into
the hardware - a firmware piece that can be loaded and actually run on the
processor. This firmware allows the same driver to be used for many different
versions of the hardware, allowing a single software release to support
multiple versions of yet to be released hardware. It is only natural to run
this privileged software inside a privileged context - the kernel.
In our case, this "hardware" is actually the virtual machine. We must deal
with changes to the underlying hardware, as they are happening rapidly, and we
must support future compatibility for customers that decide to start using a
virtual machine in 2006 - it is a virtual machine, after all, and it should
continue running in 2016, no matter what the underlying hardware at that time
will look like. In this sense, we have an even larger future compatibility
problem to solve than most hardware vendors. So it is critical to get an
interface that works now.
The essence of our interface is a separation between the kernel, and the
hypervisor compatibility layer, which we call the VMI. This layer is
completely separate from the kernel, and absolutely cannot be compiled into the
kernel. Why? Because doing so negates all of the benefits this layer is
supposed to provide. It is separate from the kernel not to lock anyone into a
proprietary design or prevent anyone from distributing a working kernel. It is
separate to allow the hypervisor backend to change radically without
introducing any changes whatsoever into the kernel. This is absolutely
required for future compatibility - with new versions of each hypervisor being
released regularly, and new competing hypervisors emerging, it is a necessity.
This allows the hypervisor development process, as well as the Linux kernel
development process, to continue unimpeded in the face of rapid change on each
side. Having an open binary interface encourages growth and competition in
this area, rather than locking anyone into a proprietary design. It also does
not stop anyone from distributing a working, fully source compiled kernel in
any circumstance I can imagine. If you don't have the firmware for a VE-10TB
network card compiled into your kernel, but also don't have a VE-10TB network
card, you haven't been deprived of anything, and have very little to complain
about. Provided, of course, that when you do buy a VE-10TB network card, it
happily provides the required firmware to you at boot time.
On the other hand, the GPL is not friendly to this type of linking against
binary fragments that come from firmware. But they really, absolutely, must be
separate from the kernel. There is no argument against this from a feature
point of view. But there is also no reason that they must be binary-only. The
interface between the two components surely must be binary - just as the
interface between userspace and the kernel, or between the apps and glibc must
be binary. This means the code from one layer is visible to the other purely
as a binary "blob". But not an evil one. And by NO circumstances, is it
required to be a CLOSED source binary blob. In fact, why can't it be open? In
the event of a firmware bug, in fact, it is very desirable to have this
software be open so that it can be fixed - and reflashed onto the card, where
the firmware belongs.
Let me illustrate the major differences between an "evil" binary blob, a
typical vendor designed hardware support layer, a well designed, open binary
interface, and a traditional ROM or extensible firmware layer. I think you
will see why our VMI layer is quite similar to a traditional ROM, and very
dissimilar to an evil GPL-circumvention device. I can't truly speak for the
video card vendors who have large binary modules in the kernel, but I would
imagine I'm not far off in my guesses, and they can correct me if I am wrong.
EVIL VENDOR VMI ROM
Module runs at kernel privilege level: YES YES YES
MAYBE (*)
Module is copied into the kernel: YES MAYBE NO NO
Module is part of kernel address space: YES YES NO(+) ??
Module has hooks back into kernel: YES MAYBE NO NO
Kernel has hooks into module: YES YES YES YES
Module has proprietary 3rd party code: MAYBE MAYBE(?) NO YES
Module has significant software IP: YES MAYBE(?) NO
MAYBE (?)
Module is open source: NO MAYBE MAYBE NO
(*) In the ROM case, sometimes these are run in V8086 mode, not at full
hardware privilege level, and whether the < 1MB physical ROM region is part of
the "kernel" address space proper, or just happens to appear in kernel address
space as a result of linear mapping of physical space is a debatable matter.
(+) The VMI layer is not technically part of the kernel address space. It is
never mapped by the kernel, and merely exists magically hanging out in virtual
address space above the top of the fixmap, in hypervisor address space. But it
can be read and called into by the kernel, so whether this constitutes being
part of the same address space is a dicey matter of precise definition. I would
propose that only supervisor level pages that are allocated, mapped and
controlled by the kernel constitute the kernel address space, or alternately,
the kernel address space consists of the linear range of virtual address space
for which it can create supervisor-level mappings.
(?) There are only two reasonable objections I can see to open sourcing the
binary layer. One is revealing IP by letting people see the code. This is
really a selfish concern, not a justification for keeping the code binary only,
while still allowing it the privilege of running in the kernel address space.
The other objection I see is if that code has 3rd party pieces in it that are
unable to be licensed under an open software license. This really is a hard
stopper for open sourcing the code, as the vendor doesn't actually own the
copyright, and thus can't redistribute that code under a different license. We
don't have any such restrictions, as we wrote all our code ourselves, but many
ROMs and firmware layers do have such problems. ROMs might also have some IP
in the form of trade secrets protecting power management or other features run
in SMM mode - but this is just a guess. And IANAL - so take all this with a
grain of salt, it is purely my uninformed personal opinion.
This brings me to the essence of my proposal - why not allow binary "blobs" to
be linked inside Linux, in exactly the same way modules are linked? If these
binary modules are actually open sourced, but separate from the kernel, is
there no reason they can't actually link against GPL symbols within Linux?
What if these modules exposed an ELF header which had exactly the same
information as a typical module? In this case, kernel version is not relevant,
as these modules are truly kernel independent, but device/module version is.
The biggest issue is that the source and build environment to these modules is
not the standard environment -- in fact many of these binary modules might have
extremely bizarre build requirements. We certainly do. But still there
remains no reason that a well encapsulated build environment and open source
distribution of these modules cannot exist. We are actively working on this for
our VMI layer. Perhaps a good solution to this problem would be to provide a
link embedded in the binary which points to a URL where this environment can be
downloaded and built - or even fully buildable compressed source within the
binary itself for most friendly binaries with plenty of space to burn.
There may be other issues which I may not be aware of on our end, but that
has no bearing on finding out what the Linux and open source community
wants.
I propose this as a solution because I would like to see binary (only) blobs go
away, and I would never again like to see hardware vendors design stupid code
which relies on firmware in the operating system to initialize a hardware
device (can I say WinModem?) which is not published and open code. The point
of an interface like this is to open and standardize things in a way that
vendors can benefit from a hardware abstraction layer, and to make sure that
the GPL is not violated in the process of doing so. I would very much like to
see Linux come up with a long term proposal that can accommodate open firmware
which actually runs in kernel mode, while at the same time assuring that this
code is authorized within the license boundaries afforded to it and available
for use by any operating system.
Thank you very much for your patience - I can get verbose, and I've already
gone too long. This is still a very controversial issue, and I wanted to
clarify several points that merit attention before dooming myself to enduring
the yet to come flames. Again, I must say IANAL, and I can't promise
that VMware or any other hardware vendor that wants to use a binary
interface will agree to everything I have proposed. But I would like to
open the issue for discussion and get feedback from the open source community
on this issue, which I think will become more important in the future.
Zach
Date Thu, 3 Aug 2006 12:06:05 -0700
From Greg KH <>
Subject Re: A proposal - binary
On Thu, Aug 03, 2006 at 03:14:21AM -0700, Zachary Amsden wrote:
> I would like to propose an interface for linking a GPL kernel,
> specifically, Linux, against binary blobs.
Sorry, but we aren't lawyers here, we are programmers. Do you have a
patch that shows what you are trying to describe here? Care to post it?
How does this differ with the way that the Xen developers are proposing?
Why haven't you worked with them to find a solution that everyone likes?
And what about Rusty's proposal that is supposed to be the "middle
ground" between the two competing camps? How does this differ from
that? Why don't you like Rusty's proposal?
Please, start posting code and work together with the other people that
are wanting to achive the same end goal as you are. That is what really
matters here.
thanks,
greg k-h
Date Thu, 03 Aug 2006 12:26:16 -0700
From Zachary Amsden <>
Subject Re: A proposal - binary
Greg KH wrote:
> On Thu, Aug 03, 2006 at 03:14:21AM -0700, Zachary Amsden wrote:
>
>> I would like to propose an interface for linking a GPL kernel,
>> specifically, Linux, against binary blobs.
>>
>
> Sorry, but we aren't lawyers here, we are programmers. Do you have a
> patch that shows what you are trying to describe here? Care to post it?
>
<Posts kernel/module.c unmodified>
> How does this differ with the way that the Xen developers are proposing?
> Why haven't you worked with them to find a solution that everyone likes?
>
We want our backend to provide a greater degree of stability than a pure
source level API as the Xen folks have proposed. We have tried to
convince them that an ABI is in their best interest, but they are
reluctant to commit to one or codesign one at this time.
> And what about Rusty's proposal that is supposed to be the "middle
> ground" between the two competing camps? How does this differ from
> that? Why don't you like Rusty's proposal?
>
Who said that? Please smack them on the head with a broom. We are all
actively working on implementing Rusty's paravirt-ops proposal. It
makes the API vs ABI discussion moot, as it allow for both.
> Please, start posting code and work together with the other people that
> are wanting to achive the same end goal as you are. That is what really
> matters here.
>
We have already started upstreaming patches. Jeremy, Rusty and I have
or will send out sets yesterday / today. We haven't been vocal on LKML,
as we'd just be adding noise. We are working with Rusty and the Xen
developers, and you can see our patchset here:
http://ozlabs.org/~rusty/paravirt/?cl=tip
And follow our development discussions here:
http://lists.osdl.org/pipermail/virtualization/
Zach
Date Thu, 3 Aug 2006 13:01:36 -0700
From Greg KH <>
Subject Re: A proposal - binary
On Thu, Aug 03, 2006 at 12:26:16PM -0700, Zachary Amsden wrote:
> Greg KH wrote:
> >On Thu, Aug 03, 2006 at 03:14:21AM -0700, Zachary Amsden wrote:
> >
> >>I would like to propose an interface for linking a GPL kernel,
> >>specifically, Linux, against binary blobs.
> >>
> >
> >Sorry, but we aren't lawyers here, we are programmers. Do you have a
> >patch that shows what you are trying to describe here? Care to post it?
> >
>
> < Posts kernel/module.c unmodified>
If you want to stick with the current kernel module interface, I don't
see why you even need to bring this up, there are no arguments about
that API being in constant flux :)
> >How does this differ with the way that the Xen developers are proposing?
> >Why haven't you worked with them to find a solution that everyone likes?
> >
>
> We want our backend to provide a greater degree of stability than a pure
> source level API as the Xen folks have proposed. We have tried to
> convince them that an ABI is in their best interest, but they are
> reluctant to commit to one or codesign one at this time.
Don't you feel it's a bit early to "commit" to anything yet when we
don't have a working implementation? Things change over time, and it's
one of the main reasons Linux is so successful.
> >And what about Rusty's proposal that is supposed to be the "middle
> >ground" between the two competing camps? How does this differ from
> >that? Why don't you like Rusty's proposal?
> >
>
> Who said that? Please smack them on the head with a broom. We are all
> actively working on implementing Rusty's paravirt-ops proposal. It
> makes the API vs ABI discussion moot, as it allow for both.
So everyone is still skirting the issue, oh great :)
> >Please, start posting code and work together with the other people that
> >are wanting to achive the same end goal as you are. That is what really
> >matters here.
> >
>
> We have already started upstreaming patches. Jeremy, Rusty and I have
> or will send out sets yesterday / today. We haven't been vocal on LKML,
> as we'd just be adding noise. We are working with Rusty and the Xen
> developers, and you can see our patchset here:
>
> http://ozlabs.org/~rusty/paravirt/?cl=tip
>
> And follow our development discussions here:
>
> http://lists.osdl.org/pipermail/virtualization/
I really don't want to follow the discussion unless necessary. I trust
Chris and Rusty to do the right thing in this area...
thanks,
greg k-h
Date Thu, 03 Aug 2006 14:41:27 -0700
From Zachary Amsden <>
Subject Re: A proposal - binary
Greg KH wrote:
> On Thu, Aug 03, 2006 at 12:26:16PM -0700, Zachary Amsden wrote:
>
>> Greg KH wrote:
>>
>>> Sorry, but we aren't lawyers here, we are programmers. Do you have a
>>> patch that shows what you are trying to describe here? Care to post it?
>>>
>>>
>> <Posts kernel/module.c unmodified>
>>
>
> If you want to stick with the current kernel module interface, I don't
> see why you even need to bring this up, there are no arguments about
> that API being in constant flux :)
>
Hence my point follows. Using source compiled with the kernel as a
module does nothing to provide a stable interface to the backend
hardware / hypervisor implementation.
>
>>> How does this differ with the way that the Xen developers are proposing?
>>> Why haven't you worked with them to find a solution that everyone likes?
>>>
>>>
>> We want our backend to provide a greater degree of stability than a pure
>> source level API as the Xen folks have proposed. We have tried to
>> convince them that an ABI is in their best interest, but they are
>> reluctant to commit to one or codesign one at this time.
>>
>
> Don't you feel it's a bit early to "commit" to anything yet when we
> don't have a working implementation? Things change over time, and it's
> one of the main reasons Linux is so successful.
>
We have a working implementation of an ABI that interfaces to both ESX
and Xen. I have no argument that things change over time, and that
helps Linux to be successful and adaptive. But the fact that things
change so much over time is the problem. Distributing a hypervisor that
runs only one particular version of some hacked up kernel is almost
useless from a commercial standpoint. Most end users get their kernels
from some distro, and these kernels have a long lifetime, while the
release cycle for an entire OS distribution is getting much longer.
During that time, the hypervisor and the kernel will diverge. If is not
a question of if - it is a question of how much. A well designed ABI
slows that divergence to a pace that allows some hope of compatibility.
An ad-hoc source layer compatibility does not.
>>> And what about Rusty's proposal that is supposed to be the "middle
>>> ground" between the two competing camps? How does this differ from
>>> that? Why don't you like Rusty's proposal?
>>>
>>>
>> Who said that? Please smack them on the head with a broom. We are all
>> actively working on implementing Rusty's paravirt-ops proposal. It
>> makes the API vs ABI discussion moot, as it allow for both.
>>
>
> So everyone is still skirting the issue, oh great :)
>
There are many nice things about Rusty's proposal that allow for a
simpler and cleaner interface to Linux - for example, getting rid of the
need to have a special hypervisor sub-architecture, and allowing
non-performance critical operations to be indirected for flexibility of
implementation.
The Xen ABI as it stands today is not cleanly done - it assumes too many
details about how the hypervisor will operate, and is purely a
hypervisor specific ABI. We and other vendors would have to bend over
backwards and jump through flaming hoops while holding our breath in a
cloud of sulfurous gas to implement that interface. And there is no
guarantee that the interface will remain stable and compatible. So it
is really a non-starter.
VMI as it stands today is hypervisor independent, and provides a much
more stable and compatible interface. Although I believe it is possible
that it could work for other vendors than just VMware and Xen, those
other vendors could have their own, proprietary, single purpose ABI that
is either deliberately hypervisor specific or accidentally so from a
lack of forethought. <Zach removes soapbox>.
As you mention, Linux is adaptive and will change going forward.
Rusty's proposal allows a way to accommodate that change until such a
time as a true vendor independent hypervisor agnostic ABI evolves.
Hopefully in time it will - but that is not the case today, despite our
sincere efforts to make it happen. In fairness, we could have been more
public and forthcoming about our interface, but we also have not
received significant cooperation or collaboration on our efforts until
the work on paravirt-ops began. With the code coming into public
scrutiny and the goal of working together, perhaps our model can serve
as a blueprint, or at least a rough draft of what a long term stable ABI
will look like. But that is neither here nor there until the code is
working and ready to go. Paravirt-ops provides a nice house for this -
it lets us all speak the same language in Linux, even if we have to
phone our hypervisor in Sanskrit and Xen speaks in Latin. Creating a
common lingua franca is still an excellent goal, but we can't predict
the future. Hopefully, nobody starts using smoke signals. In the end,
paravirt-ops allows us all to play on the same field, and either a
unified hypervisor independent solution will win, or the hypervisor
interfaces will fragment, and Linux will still have an interface that
allows it to run on multiple hypervisors. Either way, Linux gets more
functionality and better performance in more environments, which is the
real win.
Zach
Date Fri, 4 Aug 2006 09:56:26 +0100
From Christoph Hellwig <>
Subject Re: A proposal - binary
On Thu, Aug 03, 2006 at 02:41:27PM -0700, Zachary Amsden wrote:
> We have a working implementation of an ABI that interfaces to both ESX
> and Xen.
Until you stop violating our copyrights with the VMWare ESX support nothing
is going to be supported. So could you please stop abusing the Linux code
illegally in your project so I don't have to sue you, or at least piss off
and don't expect us to support you in violating our copyrights. I know this
isn't you fault, but please get the VMware/EMC legal department to fix it
up first.
Copyright 2006 http://lkml.org/