Author Eric Raymond avidly preaches the benefits of giving away source code. The software world just might be ready to listen

By Daniel P. Dern
Computerworld

July 13, 1998

When Netscape Communications Corp. recently announced it would publish the source code for its Navigator browser, people at the company credited independent programmer, author and speaker Eric Raymond for building the momentum that led to the decision. His paper "The Cathedral and the Bazaar" had become the rallying point for the new "open source" software initiative, which holds that the best, fastest way to develop software is to publish the source code and let the world have at it.

This is Raymond's second "15 minutes of fame," as he puts it; he was the editor who helped turn the old Arpanet Jargon File of humorous, historical computer word derivations and definitions into The New Hacker's Dictionary (MIT Press), now in its third edition.

Daniel Dern, who usually runs in to Raymond at science fiction conventions, spoke to the prolific author at his home in Malvern, Pa., near Kennett Square, one of the mushroom-producing capitals of the world.

CW: How did you come to write your paper?

Raymond: I came to write "The Cathedral and the Bazaar" out of my feeling of profound shock when I first encountered the Linux operating system.

CW: Shock in what sense?

Raymond: This was in 1993. I had been writing software for 15 years, and I thought I understood how to produce high-quality software. I'd absorbed all the conventional engineering wisdom we know from Fred Brooks [author of The Mythical Man-Month] and other writers in the software engineering tradition: Keep your project group small, keep your objectives well-defined, keep everything tightly controlled, obsess about bugs and so on.

So I knew all these things, and then I encountered Linux and discovered that here was a development community that was violating basically every one of Fred Brooks' rules and somehow getting away with it to produce an OS which has no peers, as far as I'm aware... it's the best, the best in the world. And I found this so astonishing that I immediately decided two things: One, I'm going to be a part of this community, and two, I'm going to figure out how it works.

The process of figuring out how it works took me about three years.

CW: As your paper's title suggests, you ended up using cathedral and bazaar metaphors for different approaches to software development, with Microsoft as a major example of the former and initiatives such as Linux as examples of the latter. What did you conclude were some of the problems with the cathedral method?

Raymond: In the cathedral approach, you have a very closed, centralized, controlled style of development. In my paper, I point out that it appears that the underlying premise of the cathedral approach is that bugs in software are such deep and tricky phenomena that everything else about the process has to be subordinated to... find all the bugs.

This leads to the insulation of development groups, it leads to the long release cycles, it leads to the rigidly defined objectives and central control, and it leads to all the other things we think of as characteristic of proprietary software development.

The bazaar model is based on the premise that bugs are a fundamentally shallow phenomenon — that if you have enough people poring over the code and testing it in different ways, [bugs] become characterized and fixed extremely rapidly. And in order to maximize this effect, you need to release early and often, and grow your development community as rapidly as possible.

It's a 180-degree shift in perspective and management style, but the empirical evidence is that large projects that are run on a cathedral model have very poor reliability and develop uncontrollable complexities. On the other hand, large projects developed on the bazaar model... seem in general to succeed very well.

CW: You suggest that Linux was the first project to make a conscious and successful effort to use the entire world as its talent pool.

Raymond: That's correct. [This approach] gives you access to a talent pool that few if any companies will ever be able to attract in the classic closed approach.

Most importantly, most critically to the entire process, you have to be willing to publish source. Let the sources be seen.

CW: By "source" you mean...?

Raymond: Modifiable source code for the software. Not the opaque block of bits that is normally delivered — not the compiled executables or whatever.

Now where is the central incentive for doing this? It's the reliability problem. The central problem of software engineering has always been that our reliability has been horrible. Awful, just completely unacceptable by the standards of any other branch of engineering. The reason is, we don't do what other kinds of research or engineering do: massive independent peer review.

In large-scale civil engineering or architecture, you would not trust a design for a bridge or a building that had not been independently peer-reviewed and seemed to conform to various standards of robustness and integrity by engineers outside the original construction group. Well, it's just completely insane to trust software — especially mission-critical software and infrastructure software — that has not been massively and independently peer-reviewed by other software engineers.

And that's why whenever reliability is a concern, sources must be open.

CW: Your sequel to "Cathedral" is called "Homesteading the Noosphere." OK, so what's a "noosphere"?

Raymond: It's a term of art from some schools of philosophy and mysticism... it basically means the "sphere of ideas." I wanted to make a distinction from "cyberspace," which is a much overused and rather horrible term.

In talking about the noosphere, I wanted to talk about the sphere of ideas — or more accurately, the sphere of possible projects — and I wanted to focus on and highlight and describe the processes by which volunteer groups and open source groups, whether commercial or volunteer, carve out new pieces of territory and how people essentially acquire a sort of community ownership of bits of that noosphere.

And that turns out to be a tremendously interesting thing to look at if you're interested in how to organize the development of software for maximum creative work and the highest quality of creative work.

CW: Your paper delineates some distinctions and differences between the open source model and the free software model. A lot of people in the corporate and commercial world exhibit legitimate concern when they hear "free software" — which sounds to them a lot like "no support."

Raymond: And they have a right to be concerned. Here's the deal: In the past, most of the arguments that Internet hackers have made for this style of development have tended to be essentially a priori and moral ones. "You should share software because it's the right thing to do." "You should share software because intellectual property is a bad thing." "You should this, you should that... follow my saintly example." This is the classic mode of evangelism that we associate with Richard Stallman and the Free Software Foundation. And there's a level on which that worked really well, in that Richard was successful in recruiting a great many programmers to do high-quality free software work and make those results available to the world.

[However,] those arguments are completely useless, even counterproductive, when retailed to corporate CIOs and venture capitalists who, quite properly, are interested in questions like, "How do I make my company grow and make a whole lot more money and get better reliability and more satisfied customers?"

When the Netscape release went down, I was invited to consult with them for a day on some of the major issues [the company would face in giving away its source code] — licensing, community support programs, how they could erect a firewall around their development group so it wouldn't get mugged by their own corporate culture. The day after I talked with them, a number of free software and Linux people from Silicon Valley got together, and we talked about where do we go from here. One of the things we realized was that nobody had really tried to make the case for this way of doing things — which we knew was technically superior — that was palatable to Joe Average CIO. So we thought seriously about how to make that case.

One of the conclusions we came to is that the free software label is a net loss. The technical ideas behind it are really good in the sense that openness and code-sharing lead to high reliability and better quality and so forth. But all the political and moral baggage associated with it was just turning people off.

CW: You're saying open source isn't necessarily free, and vice versa.

Raymond: Yes. You can have proprietary software that's not commercial. That's the classic shareware model, where people give out binaries but you don't get the source. Conversely, you can have open software that's quite thoroughly commercial, stuff for which the source is released but for which the company makes money by selling service on it. So these are two different axes, and we decided there was a need to convince the corporate world that open source is a good idea, that there are excellent selfish economic reasons to do it.

I'm not going to claim that open source is always the way to go. In fact, I discuss the unusual cases in which closed source is the way to go, on the Open Source [ http://www.opensource.org/ ] site. But in general, the idea that you're protecting a revenue stream by closing software is an illusion.

The industry has this model of itself as a manufacturing industry, where you make your product and then compile it into bits and shove it out the door and try to ignore your customers as much as possible because after-sales support is a cost center, not a profit center. This is fatally out of line with the cost structure of software development and the expectations and legitimate needs of consumers.

'I coulda been a contendah' — Eric Raymond's version

"Back around 1989-1990, I worked with a guy named Ray Gardner on... a special-purpose little program we called VH [for Volks Hypertext]. You could page through the file and also follow references in the dictionary entries, back out of those references, and so forth and so on.... It did a significant fraction of what we now associate with the Web.

"A couple of years after the Web broke big, I was looking through my back E-mail, cleaning it out, and discovered I had a message in my mailbox from Tim Berners-Lee in 1991 — which I had forgotten because nobody knew who he was at the time.

"And he said, 'I looked at your VH, and it looks pretty interesting. I'm doing some work on hypertext, shall we collaborate a little bit?' And I wrote back and said, 'Sure, I'm really interested in this problem.'

"And he never got back to me. [Laughs.] It looks like because of one piece of lost mail from Tim Berners-Lee, I just missed being in on the beginning of the Web."

Resources

Eric Raymond's home page [ http://sagan.earthspace.net/~esr ]
This page holds links to Raymond's papers, "The Cathedral and the Bazaar" and "Homesteading the Noosphere."

The Jargon File online [ http://sagan.earthspace.net/jargon ]

Open Source [ http://www.opensource.org/ ]

Dern is an independent author, speaker and consultant who focuses on Internet/intranet benefits, strategies and concerns for businesses and end users. He can be reached at www.dern.com or via E-mail at ddern@world.std.com.

Copyright 1998