Climbing the Peak: Agony and Ecstasy Of 200 Code Writers Beget Windows NT

Badgered by a Driven Guru, They Fight Fatigue, Fear To Build Grand Synthesis --- Dread of an Unfixable Bug

By G. Pascal Zachary, Staff Reporter
The Wall Street Journal

Redmond, Wash. -- May 26, 1993 -- David Cutler, wearing white Reeboks, white trousers and a T-shirt with the legend "Over the Line," blasts through the door leading to Microsoft Corp.'s Build Lab. Mr. Cutler, a stocky, balding man who shares some character traits with Captains Ahab and Bligh, is checking on the progress of the biggest, most complex and possibly most important program ever designed for a desktop computer. He is not happy.

It is 10:20 Monday morning, May 3, and the daily "build" of the program, called Windows NT, isn't finished yet. As leader of the NT team, Mr. Cutler insists that a new build, or test version, of NT be stitched together electronically each morning so that his programmers can, as he puts it, "eat their own dog food." Mr. Cutler is angry about the delay, angry about a botched test the day before, angry at the world. After glaring at a computer screen he storms out of the lab, leaving a distinct chill. Two builders, Kyle Shannon and Arden White, dip into a king-size jar of Rolaids, popping one each. The day has soured early.

Minutes later, Mr. Cutler returns to the Build Lab looking even more upset. "You're wasting the whole damn morning by not having this damn thing ready," he snaps.

The NT leader doesn't believe in bottling up emotion. "The way you let off stress is to let it out," he says. He means it. Mr. Shannon points to a circle drawn on the wall near the door; it marks the spot where Mr. Cutler once unleashed a violent kick, cracking through the wall and injuring his toe. For his 50th birthday a few months ago, Mr. Cutler's team gave him a framed piece of his office wall, which he had destroyed in a moment of rage ignited by the failure of a new build to run on his computer.

Code Red

Mr. Cutler's impatience is ill-mannered but understandable. Time is about the only thing that matters to him and the roughly 200 developers and testers who are bringing NT to life. The product, already six months late, was officially announced on Monday, and Microsoft plans to begin shipping hundreds of thousands of copies to customers within 60 days. Failing to meet that deadline -- or shipping a bug-riddled NT -- would mar the reputation of a computer program considered by many to be the most ambitious ever tried.

Consisting of a staggering 4.3 million lines of code and costing more than $150 million to develop, NT is designed to conquer new worlds for Microsoft and its allies. NT's supporters hope they can use the program to empower not just speedier personal computers but also new kinds of PC networks that can displace far more expensive mainframes and minicomputers.

With NT, Digital Equipment Corp. aims to become a leading PC maker, Compaq Computer Corp. a workstation maker and American Telephone & Telegraph Co.'s NCR a maker of inexpensive mainframe substitutes. Microsoft's chief, William Gates III, fondly hopes NT can wipe out the franchise of another hugely successful software pioneer, Novell Inc., by making Novell's networking programs superfluous. NT also opens a crack in Intel Corp.'s near-monopoly on microprocessors for machines that run PC software.

Immense stakes are riding on NT. But the making of NT also illustrates how innovation, in an age of technological complexity, usually requires dogged system-building and an ability to hold large teams together while allowing -- even cultivating -- conflict. Conflict lies at the core of innovation "because there isn't always a unique way to solve a problem," says Emanuel R. Piore, a former chief scientist for International Business Machines Corp. "When there's no conflict, a lab is no good."

The King of Code

The story of NT begins in the late 1980s, when Microsoft held a grip on the basic operating system, called DOS, that had come to dominate personal computing. An operating system, or OS, controls the basic functions of a computer, providing its distinct personality and powers. DOS's success sent Mr. Gates on his way to becoming the richest man in America.

But he and his top aides dreamed of a much more advanced operating system, one that could become, in a sense, personal computing's lingua franca for years to come. It would run at greatly enhanced speeds, processing 32 bits of information at a time rather than DOS's 16, permitting the use of greatly enhanced graphics. It would contain extraordinary security features, which are especially important to corporate customers. It would still be able to run the many thousands of so-called application programs (such as Lotus 1-2-3) that have been written for DOS. It could perform more than one task at a time, and it would link many different kinds of computers and hundreds of different kinds of printers and other gear in a single, seamless command system.

The system they envisioned would also be "portable," or able to run on different classes of microprocessors, making NT ubiquitous on machines such as workstations, "server" machines that link networks and other midrange computers. And despite its inevitable complexity, the system would be rigorously reliable.

Something like this already existed, actually: a high-end, complex, expensive system called Unix, used mostly by scientists and engineers. At the urging of Nathan Myhrvold, a key aide, Mr. Gates aimed to bring most of its features to the PC masses. In 1988, he found "the man for the mission" in Mr. Cutler.

Mr. Cutler was a luminary at Digital, the programmer who had led the team that wrote VMS, the operating system that fueled Digital's growth in the late 1970s by linking together a diverse family of computers. This was a feat IBM, Digital's main rival, had never accomplished, and it enabled Digital to blow a substantial hole in IBM's commercial computer sales.

But after one more big success at Digital, the company killed Mr. Cutler's next big project, prompting him to quit in 1988. He was working on a deal to start his own company when Mr. Gates called. Mr. Gates felt Mr. Cutler might have the ideal set of skills for creating a "superambitious" operating system. The Microsoft chief met with Mr. Cutler and came away impressed with his know-how and intrigued by his gruff personality.

"He's known more for his code than his charm," Mr. Gates said. That is just what Mr. Gates, no pushover himself, wanted. As it happened, Mr. Cutler's style meshed well with a Microsoft corporate culture that thrives on confrontation.

For Mr. Cutler, anger and passion were bound up together. He would get mad easily, menacing colleagues with expressions such as "I'm going to walk through your wall." But his zeal to finish a job infected his colleagues, and his frequent confrontations served a function often missing in big projects: instant feedback incapable of misinterpretation.

And nobody doubted that he was a gifted programmer, which is a curious thing to be. Writing code, far from being an exercise in rational reductionism, is more like hitting a baseball, programmers say. Two equally brainy people may sit down and try it; why one succeeds and another fails is as mysterious as art. Some of the best programmers have never been to college.

Soon after his arrival, Mr. Cutler presented his vision for NT at Mr. Gates's exclusive annual retreat in November 1988. As Mr. Cutler talked well into the night, "we sat in awe, listening," recalls Jonathan Lazarus, vice president for systems marketing.

At the same time, Mr. Cutler was the sort of leader who slogged through the mud with his troops. "You really don't take Mr. Cutler's outbursts personally," said Mitchell Duncan, his chief builder. "He's not attacking your character. Dave gets in at 6 a.m. He's there every weekend. He's there in the trenches."

Creative Tension

In those trenches were divisions of engineers, often pitted against one another in a deliberately cacophonous search for answers to programming questions that grew more complex with each additional line of code (eventually the NT program reached the equivalent of 100,000 single-spaced pages of type). First among equals were the developers, or "code jockeys" -- mostly men and usually terribly sure of themselves -- who wrote basic instructions. Next in the pecking order came testers, who tried to make those instructions fail. Then came product managers, who thought about what NT ought to do, and builders, who put the program together.

To start work on NT, Mr. Cutler created three broad units: one to create base language that monitors and runs the program; one to create its networking ability; and one to run all the features a user actually sees. Each was in turn broken down into five to 10 subgroups as small as three or four people each. "The smaller the better," said Mr. Cutler, who wanted to delegate responsibility, eliminate supervisors and discourage the bureaucracy that often grows up alongside large projects.

With an organization in place, he began assembling a prototype of an operating system that could run on different microprocessors. One of Mr. Cutler's important early decisions, in January 1990, was to put NT on a powerful new chip made by upstart Mips Computer Systems Inc., as well as Intel's 386 chip and its successors. The move potentially threatened the profits of mighty Intel, Microsoft's longtime partner, by ensuring that the huge body of PC software would run on chips from rivals.

Free at Last

NT soon presented Mr. Gates with a huge political problem. He had previously forged an agreement with IBM under which the two companies would jointly develop another operating system that would offer some advanced features while still running DOS. It was called OS/2. To mollify IBM, Mr. Cutler was told to get NT to run applications written for OS/2. Mr. Gates was insisting on such compatibility because it was important to IBM, but it was a bundle of unattractive compromises.

The NT team saw OS/2 as "a bag of dirt," says Steven Wood, the first Microsoft programmer to join NT. As it turned out, however, NT helped precipitate a timely blowup between Microsoft and IBM.

By August 1990, Mr. Gates's new graphical command system that made DOS programs easier to use, called Windows, had taken the PC world by storm, selling 500,000 copies a month. The runaway success led Mr. Gates to a bold stroke -- marrying Windows with NT, which opened a rift with IBM that couldn't be healed. In effect, this move meant Mr. Cutler's new mission would be to develop NT so it ran applications written for Windows and DOS, not OS/2 and DOS.

IBM got the message. The company in effect divorced Microsoft, taking OS/2 in settlement. (Windows has since outsold OS/2 by a 10-to-1 ratio.) Mr. Gates had hoped his attraction to Windows would not threaten his longstanding association with IBM, but when the rupture became inevitable he wrote associates gleefully that Microsoft no longer had to accept IBM's "poor code, poor design and other overhead."

The split lifted the importance of NT and breathed fresh life into the team. Suddenly, their work moved from the periphery to the heart of Mr. Gates's vision of Microsoft, where "everything we do should focus on making Windows more successful."

With the pace quickening, Mr. Cutler's daily briefing shifted from a seminar on imaginative software solutions to a discussion of hard deadlines. Arguments erupted as chiefs from various parts of the sprawling organization grappled with Mr. Cutler's chief concerns: reliability; portability and compatibility.

Testers and coders waged some of the biggest battles over reliability. Every night the testers would subject the fresh NT code to ordeals on 200 computers, stresses that would rarely occur in the real world. Testers might ask NT to read and write, continuously and simultaneously, in separate files all night long. At the same time they might instruct the program to create a graphic on screen and then wipe it out, over and over again. A report would be generated in the morning, and any computers that couldn't complete the test -- that crashed or halted -- would be investigated. On a great night, only 2% of the computers would fail.

Programmers were happy when the tests exposed mistakes, but they complained loudly when the testers seemed too brutal and revealed too many flaws. Mr. Cutler, while convinced that a good way to improve a system was to break it, at times resisted the advice of testers, feeling that their rigor was too much. One tester, who attended the daily meeting, subtly retaliated by repeatedly sitting in Mr. Cutler's preferred chair at the conference table.

Pursuing the goal of portability also tested patience and discipline. Mr. Cutler would harangue developers when they failed to give equal attention to both the Intel and the Mips versions. His credo was: Treat Mips the same as Intel and make both versions plenty fast. At one point, Mr. Duncan, NT's chief builder, violated a Cutler policy against splitting the build into Mips and Intel versions, because programmers, eager to get their hands on the latest Intel code, demanded that he do it. When Mr. Cutler learned his prime directive had been violated, "he just laid into me," Mr. Duncan said. At a meeting the next morning, NT's leader upbraided him for making "a unilateral decision" and promptly overruled it.

The Death March

NT, everyone at Microsoft well understood, could succeed only if outside software developers flocked to the system, so a crucial part of the team was devoted to promoting NT to the thousands of large and small companies that would write programs for it. In October 1991, the team released a package of code guidelines to help people write programs for the Intel version of NT. In December, a similar package for Mips was issued. These preliminary kits, however, were just appetizers. Eager to start writing programs for NT, software developers craved the main meal -- a finished kit. Microsoft promised to supply it on July 7, 1992, at a conference in San Francisco. Expectations ran high. Never before had so much trouble been taken to raise the curtain on an unfinished operating system.

But NT was so complex it had thousands of defects, or bugs, raising fears that the kit would be a disgrace. In an all-out effort to clean up NT before July 7, the NT team went on "a death march," as one member put it. The workload increased, the briefings and builds were extended to Saturdays and Sundays, dinners were brought in, and a fair number began sleeping at work -- under desks, in lounges and on the floor.

Many on the NT team actually had never faced such a deadline in their lives; for 18% of the staff, this was the first job out of college. Johanne Caron, a 28-year-old French Canadian who was writing part of the "DOS shell," or the layer of instructions that allowed existing DOS programs to run on NT, found herself on the edge. Her basic code affected a crucial, if somewhat pedestrian, part of NT, and it drew scrutiny. If anyone criticized her, she said, "I'd jump down their throat." Meanwhile, her marriage was falling apart. "I put all my energy in my job," she says. "I didn't even try to save the marriage."

Spouses were not the only ones who resented the killer pace. One Saturday morning in June, Jonathan Manheim dropped his seven-year-old son off at a soccer game on his way to Mr. Cutler's morning meeting. As he left the car, the boy said, "I would throw away all of my toys if you would be here after the game."

The words hit Mr. Manheim, who was struggling to make a Windows graphics program compatible with NT, "like a knife in my heart. I don't remember what I said to my son, but I didn't go to work that day."

Mr. Cutler kept the pressure high. In most software-development programs, the various pieces under development are stitched together only once in a while. By contrast, Mr. Cutler insisted on a fresh build practically every day, meaning that each piece of NT, some containing hundreds of changes, had to be reassembled into a whole, which was then subjected to a new round of testing. This scramble for a fresh build led to more skirmishes, these pitting programmers against builders. "We have all of these cowboy developers, just slinging code like crazy, calling out: We need another build, we need another build," grumbled Mr. Duncan, the chief builder.

Agitated colleagues even assailed Mr. Duncan while he slept under his desk during those nights he never made it home. He began hanging a sign on his door: "Do Not Disturb. I'm sleeping under my desk." During the day he grew defensive, buying construction hardhats for himself and his staff. When he wore his hat, programmers often slapped it.

A few people burned out, eliciting little sympathy from some colleagues. Walter Moore, 36 years old, wrote code that made graphics appear on the most widely used type of computer screen. The cleverer the code, the faster the graphics flashed on screen. This is basic, unglamorous work, but if poorly done it can cause much customer unhappiness. During the death march, Mr. Moore found himself "falling further and further behind." When faced with an arduous task, he felt a strong urge to play video games. A sympathetic manager covered for him, but his performance got so bad that co-workers began referring to his debilitating condition as Waltheimers.

"Sooner or later, you lose it," Mr. Moore said when people asked what happened. Finally, the head of Mr. Moore's unit told him: "I don't think you're going to pull out of this." In October, Mr. Moore quit.

Despite the NT team's casualties, the software developers' kit was delivered on time, and the three-day conference "was a turning point," said Peter Winston, a developer who was there. "It moved NT from being all talk to being a viable product." Fifty thousand kits were sold at $69 each, an unheard-of number for an unfinished product.

Bug Hunt

Mr. Gates still worried about the NT project. Some potential customers who previewed it told him they were far from sold on NT. It was too slow, required too much extra internal memory and had not proved it could run Windows applications, they said. In late September last year, Mr. Gates was compelled to announce that NT would miss its year-end shipping date. Before long, the date was formalized as "in the first half of 1993."

On Oct. 12, the first test version of NT was released to actual customers. Mr. Gates was not happy with parts of it. In December, behind closed doors, he met with the NT team's leaders and sternly reminded them that while brand-new features are admirable in a software program, nothing bothers customers more than seeing a familiar program run more slowly than it did before.

The point was clear. "Bill basically prioritized our work for us," said Moshe Dunie, who oversees testing. "That's his genius. He can discern the weak points."

Most of those weak points were attributable to NT's enormous size. NT was like some giant interactive jigsaw puzzle, in which every added piece could affect the pieces that had gone before, often in unpredictable ways. NT was designed to work with more than 600 kinds of printers. It had to be able to recognize and communicate with modems, pointing devices and, of course, other computers. It had to be able to run popular Windows programs -- originally written in another language altogether -- flawlessly. Teams could learn a lot by writing code, building programs and punishing them with tests, but even Mr. Cutler conceded that no one person "can grasp the complexity of NT."

As the March release of a second test version approached, anxieties flared. Performance was improving, features were set and an astonishing number of existing applications and peripheral hardware already meshed with NT. But compatibility with Windows programs -- a critically important goal -- was a continuing headache.

It turned out not to be possible, as some had once hoped, to create a generic translator to handle Windows programs. So a team of nine programmers led by Mark Felton produced "workaround code," embedding individual translators into NT that would wake up and go to work when they sensed that NT was being called upon to run a particular Windows program.

Mr. Felton's team aimed to adapt 200 Windows programs to NT, but a large percentage proved balky. Top managers fretted that some customers might dismiss NT altogether if it didn't run Windows programs well. The pressure grew.

Mr. Felton was a confident programmer; both of his parents had been pioneering software writers in his native England after the second world war. Mr. Felton's progress, however, wasn't always straight-line. "From time to time, things would go backward," he said, as certain Windows programs actually ran worse on the latest and supposedly improved build. And as more "workarounds" were needed to get Windows programs to run, NT, already huge, kept growing. "It would be nice if we were smaller," Mr. Felton said, because fatter code requires more memory and more memory means costlier computers.

Meanwhile, thousands of bugs, or coding errors, menaced NT. Bugs were ranked in priority, with "priority one" and "showstoppers" -- the kind that could crash a system or swallow up data -- receiving the most attention. Fixes for the bugs often themselves spawned more bugs.

Even a programmer as seasoned as Mr. Cutler would not dismiss the possibility that NT somehow could become trapped in a weird infinite loop, wherein fixes spawned their own paralyzing bugs. It had happened to others. The history of software is littered with notorious projects that were abandoned in disgust. Such defeats caused careers to dissolve in ridicule. When a bug is discovered, "Things . . . can look so bad," Mr. Cutler says. "Then someone fixes a bug and everything just turns around." Adds David Thomson, one of his top aides, "Every morning we look at bug count. If the count increases for too long, we are doomed."

Patience thinned. One Saturday evening Mr. Cutler chided Ms. Caron for not filing a fix earlier in the day so that it could be subjected to stress testing. He asked her to explain the dereliction. "I was at karate class kicking butt," she shot back, eliciting a stunned "OK" from Mr. Cutler.

Compromise, Not Perfection

As the NT team pounded down the home stretch, it became clear that the project would yield a messy if successful compromise, rather than some paragon of coding perfection. In the last week of April, for instance, the team fixed a staggering 1,132 bugs, but found 713 new bugs serious enough to warrant fixing. The NT program just unveiled contained some bugs, and a few will almost certainly remain when the final release is shipped within 60 days.

The choices are stark now, said Mark Lucovsky, one of the first NT code writers: "Is it a problem we fix? Is it a problem we postpone? Is it a problem we work around?"

The most serious problems involve incompatibilities between NT and well-known Windows programs. At the moment, for instance, a popular desktop publishing program called Aldus Pagemaker won't print under certain circumstances. Because glitches like this are likely to be pounced upon, in recent weeks more testers and bug-fixers have been assigned to the problem of Windows compatibility. As of early May, Mr. Felton's cleanup crew had 80% of the 200 selected Windows applications turning in "good" or "excellent" performances on NT, but the other 20% were classed as "bad," "OK," or "bomb." Still, Paul Maritz, the Microsoft senior vice president overseeing the NT project, says that "we've got the problem licked."

Mr. Gates, for his part, thinks NT "absolutely met the goals we set for it," though he wishes the code had not grown so large. He wanted NT to run PCs with eight megabytes of random-access memory, but concedes that in most cases it will demand 12 to 16 megabytes.

Compromise or not, NT is viewed as a hit by industry observers, because it accomplishes enough of Microsoft's major goals for networking power, reliability, compatibility and portability to make life difficult for its rivals and provide an operating system that will last through the end of the decade. Mr. Gates predicts that one million copies of NT will be sold over the next year, at an initial list price of $295 for owners of Windows.

"You'd have to say that NT is a sure thing," says Stewart Alsop, editor of the trade weekly Infoworld. "The only question is how sure? Is it the whole future of personal and organizational computing, or just a part?"

Mr. Cutler is withholding judgment on the result. He says he is "very adamant" about not taking stock yet. "You don't take stock when you haven't shipped. We're not done until NT is on the retail shelves."

Others say Mr. Cutler can already take a bow. Gordon Bell, a renowned computer designer and one of Mr. Cutler's mentors, thinks that while the NT chief's methods were harsh, "It's probably the only style that's going to make it in a project like this. I don't think there's anybody else in the world who could've built NT. No one else has the incredible focus and stamina required."

As for Mr. Cutler's troops, they are divided in their feelings about the future. Many eagerly anticipate working on the next version of NT, while others sense that a chapter of their lives has ended. Some are tired, almost to the breaking point. After nearly two years in "ship mode," with one deadline after another looming, "People are really worn out," says Charles Whitmer, a graphics programmer. "A lot of people are angry, tired and burned out."

Copyright Dow Jones & Company Inc