Behind the scenes at the IBM Linux Technology Center
A look at who they are and what they doMaya Stodte (mstodte@us.ibm.com)
Contributing editor, developerWorks
June 2001
Maya Stodte takes a look at the IBM Linux Technology Center. She talks with Sheila Harnett, who founded the center with Dan Frye, about what they are and what they do. She also takes a look at some of the other key players at the LTC and the overall agenda and where it's headed.
Mirroring the growth of open source projects and the expansion of the free software community, IBM's own contributions to the community started small but have grown at a rapid pace. At the forefront of many of IBM's technical contributions has been the Linux Technology Center. It started out as a kind of skunkworks project in its own right and has grown to encompass as many as 200 devoted developers the world over. These days, the members of the LTC work on Linux projects ranging from patches to structural kernel changes to file systems and internationalization efforts. Together they form one of the larger corporate groups of open source developers.
The LTC Web site features an ever growing set of twenty-five Linux open source projects and over fifteen collections of patches in which IBM is directly working with the Linux development community. The LTC's mission is to work directly with the Linux development community to further a shared vision of making Linux succeed both as the basis for enterprise-level customer solutions and as an operating system.
The roots of the LTC go back to the Fall of 1998 when IBM kicked off a corporate task force to investigate Linux technology and open source and to articulate a strategy for IBM. That task force recommended the aspects of the Linux strategy that were later adopted by IBM and put into place in early 1999. One of those recommendations was that IBM create a team of developers to work with the open source community, as partners, in implementing the capabilities that were important to meet enterprise customer requirements. Sheila Harnett, one of the representatives on that task force, later came together with Dan Frye to form the LTC, with the goal of establishing a presence in the open source and Linux community as a center of competency and development, and to make increasing contributions to community development efforts. The original mission statement of the LTC, which was formally established in the summer of 1999, was to contribute technology to Linux with the specific objective of addressing enterprise functionality. "Or," as Sheila says, "simply, to help make Linux better, specifically for enterprise applications, workloads, and hardware."
In addition to serving as IBM's open source development team within the Linux community, the LTC also serves an important role internally as a technical resource and an enabler for Linux development teams in other areas of IBM. The first step was creating an internal presence and communications hub for IBM's growing internal Linux community.
The LTC started within IBM as an internal network. "In the summer and fall of 1999," according to Sheila, "we had one person doing Web administration and we needed a presence internally. So we started an internal Web site and download site. To foster the spirit of open source we had to do that internally."
The next step was to begin participating in external development projects. The first development contributions to the Linux community resulting from this effort were in the area of network drivers and the PCI TokenRing adapter. "There was no driver available for this specific TokenRing adapter on Linux, and although we didn't move the bar of capability on to enterprise with that contribution," says Sheila, "it did give us our first toehold in the community and an opportunity to show that we could contribute. The external Linux community came back to Mike Sullivan, a developer who was working on the project, and asked him to make other changes." At that time, the LTC also included a small team of performance experts who began investigating and assessing Linux's capabilities and identifying other areas in which IBM could contribute from a performance standpoint. At the time the LTC staff numbered only about five or six developers. Since then, the ranks have swelled to around 200.
Although the LTC has been known to hire non-IBMers, most of the developers come from within IBM. The team works to recruit people with the skills that they need. "We go after people who have good OS development skills or platform hardware skills in the areas that we want to have a presence," Sheila Harnett explains. "In growing our team, we try to identify areas where we want to make a difference and go after people with the necessary skills."
At the end of 1999, IBM made another significant contribution in the form of patches to enable Linux to run on the S/390 mainframe. The LTC's first major project, announced at the LinuxWorld Conference in February of 2000, was the JFS project led by Steve Best. Two months of work by the internal team preceded the announcement, to make sure they understood what would qualify as a base level load of functionality for the first code drop of JFS. "We were trying to determine what the first entry point into the open source community should be," Sheila notes. "And in the spirit of 'release early, release often', over thirty code drops in 15 months followed."
As is par for the course in the open source community, Dan and Sheila had worked together for ten months on starting the LTC before they had a chance to meet in person. They finally crossed paths in August of 1999 at LinuxWorld in San Jose. At that time, Sheila was trying to get more people on board, while Dan was trying to secure a budget. "Why don't we just start hiring them?" Sheila remembers asking him. And so they did.
Since then the team has grown to reach what they consider to be close to its maximum capacity (200) while maintaining an efficient work ethic with its members spread so widely apart.
"It's exciting," says Sheila, "to work inside IBM and be a part of the open source development community. Our team consists of developers located across the US, England, Israel, Australia, Japan, India, Germany, and several places in between. The team is practicing the open source model from an internal management perspective because people are not centrally located. We use mailing lists, Web-based tools, and IRC chat channels to communicate. By doing this, it has allowed us to go out to where the skills are without having to move people in to any one location."
The LTC's organizational structure is what you might expect in a typical server OS development team: small groups of developers working on kernel networking, I/O and storage, systems management, security, internationalization, runtime libraries, printing, RAS capabilities, performance, testing, standards, etc. "The neat thing," Sheila notes, "is that many times a project like improving scalability will weave its way through multiple areas of those groups, tapping and involving the expertise and skills of folks from all those disciplines."
The agenda
The main agenda of the LTC has been to establish an IBM presence in the Linux and open source community, and to mutually contribute to and benefit from that work. "When we first began," recalls Sheila, "we wanted to make sure we could demonstrate that open source community problems were our problems as well, so some of the projects that we worked on may have seemed a little spread out. Now that we have demonstrated that we can contribute code and development and expertise, we more deliberately choose projects to work on where we think IBM has technology to contribute."
The LTC is focusing on many Linux open source projects; here's a sampling:
The LTC is also working on volume management technology in the form of the Enterprise Volume Management System (EVMS), where the result of the effort is aimed providing at more volume management capability and interoperability in the kernel.
Finally, The LTC is working with other folks in the community on test and documentation. "Rather than have multiple groups test in isolation, we can collectively harness all of our efforts to more effectively test the OS in the way that any commercial product is tested before it's sent out to the customer. If everyone used similar technology we might have broader coverage. We're trying to help drive common test tooling and test cases through a lot of different groups." Sheila said.
Modes of operation
"It is important," according to Sheila, "to work with the community to bring Linux up to par in order to make it the optimal operating system to sell on a particular piece of hardware for a particular enterprise workload. We are trying to expand the possible types of workloads and solutions for which Linux plays a role in our customer's IT environments." As an open and unbiased contributor to the open source community, the LTC has received a pretty positive reception so far. Although the attitude of the team still takes into account that there is a constant need to prove themselves every time they put out code, they seem to have reached a stage of comfortable equilibrium with the community. "We have to continually earn our position in the community. People like what we are doing," says Sheila. "They view us as peers, and recognize that we have a lot of common objectives with other folks in the open source community. We're out there like everyone else getting our hands dirty."
Success at the LTC, like that of most open source initiatives, is measured mostly in the popularity of the code they write and the number of patches that are being accepted into mainstream components. Collectively, LTC contributions are also measured by how well they help Linux extend to handle more and more complex enterprise applications and hardware requirements. Unlike commercial development, where success can be measured in terms of sales and product popularity, work coming out of the LTC is more dynamically gauged along with the needs and directions of the community. "We want to work important, key projects, but that doesn't mean we won't sometimes take a controversial position by challenging existing design points. That is the lifeblood of innovation fostered by open source development," Sheila says. "Not only do we want the code to be technically good, but we have high coding standards and we see the need to advance the operating system as the community grows."
The future
The future for the LTC seems bright as does the future of Linux and the open source community. News updates from LTC projects are posted by the LTC several times a day on its Web site, and it's the cumulative direction of these projects and their evolution that point to the future of the LTC. "In order to prioritize our open source development activity, we are focusing on improving Linux capability and performance for workloads like Webserving, Database backend serving, and Telco carrier network servers," says Sheila. "As we improve Linux for those workloads, we will begin to tackle even more challenging workload requirements."
Key members and project leaders at the LTC
Bill Abt, a Senior Software Engineer at the LTC, heads up the Next Generation POSIX Threading Project. NGTP aims to solve problems associated with the pthreads library on Linux. Bill, in his own words, "has been a Linux hacker since 1996, and a Java hacker since the beginning. At it (software engineering) now for over twenty years, right after leaving the Navy. My first language was Fortran on punched cards!!!! I worked on OS/2, Lotus eSuite, and at the JTC on the VM before working on NGPT."
Steve Best heads up the Journaled File System project at the LTC, currently used in IBM's enterprise servers. Steve works in the Linux Technology Center of IBM in Austin, Texas. He is a member of the File System development department and is currently working on the port of Journaled File System (JFS) to Linux. Steve has done extensive work in operating system development with a focus in the areas of file systems, internationalization, and security.
Richard Ferri started the LUI (Linux Utility for cluster Install) project at the LTC two and a half years ago, to provide Linux with a cluster installation tool similar to NIM, the Network Installation Manager in AIX. He has been working on AIX and Linux for 10 years, first porting applications to AIX/ESA, and then designing and developing systems management code for the RS/6000 SP. He now works on LUI, SIS, and OSCAR, a consortium open source clustering initiative, of which he was a founding member.
"LUI was first released under the GPL (GNU Public License) in April 2000," according to Richard, "and distinguishes itself by concentrating on heterogeneous cluster install -- installation when the nodes require different kernels, file systems, sets of RPMs (Red Hat installable packages) and have different size hard drives and numbers of processors." There are now several programmers from the LTC that work nearly full time with Richard on LUI and its successor, SIS (System Installation Suite).
Richard Ferri and Brian Finley, the curator of the VA SystemImager project from VA Linux, are now working on merging technologies and building the SIS project. "SystemImager," according to Richard, "excels at installation when nodes are homogeneous (similar hardware and software requirements). This new installation tool will provide installation methods for all the major distributions of Linux, on the major architectures (PowerPC, Intel-32, Itanium, and Alpha). It will retain aspects of the original LUI technology, and will provide the community with an integrated solution for both hetero- and homogeneous. " The SIS project will be hosted on SourceForge.
Daniel D. Frye, Director of the Linux Technology Center (LTC), is responsible for overseeing IBM's Linux technical strategy and participation in the open source Linux development community.
Prior to his current responsibilities, Dr. Frye was a member of IBM's Emerging Technologies and Business Opportunities team. In this role, he was responsible for company-wide technical strategies that predicted future trends and transitions in the IT industry. It was during this time that Dr. Frye co-authored the original IBM corporate strategies for Linux and open source.
Dr. Frye has an M.A. in Physics from The Johns Hopkins University (1982) and a B.A. in Physics from the University of Idaho (1979). He also received his Ph.D. in Theoretical Atomic Physics from The Johns Hopkins University (1985). Dr. Frye is a core team member of the U.S. Government Presidential Information Technology Advisory Committee on open source.
Sheila Harnett, co-founder of the LTC, is a Distinguished Engineer at the LTC. Her current role is System Architect for Linux, although she has been involved with the definition and execution of IBM's software strategy for Linux since 1998. Since joining IBM in 1992, Sheila has devoted her time to operating systems design and development, Internet technologies, and object-based technologies.
Richard Moore, RAS architect at the LTC, is responsible for leading Serviceability initiatives for Linux and is a leader of the DProbes Project. DProbes is a debugging facility designed to work under extreme or inaccessible conditions. It gathers diagnostic information by dynamically firing probes into executing code modules, relying on user written probe-handlers (programs written in assembly-like code based on Reversed Polish Notation). Richard also works on Flexible Dump and Error Logging. He graduated in pure mathematics from the University of Kingston upon Hull and joined IBM in 1984 as a Networking Systems Programmer. His career has been devoted to problem determination and developing PD techniques. He has worked in a wide range of operating system environments including MVS and OS/2. Under OS/2 he co-authored the OS/2 Debugging Handbooks and was responsible for bringing enhanced tracing and dumping capabilities to OS/2.
Mark VanderWiele, Senior Technical Staff Member at the LTC, works on the Omni Print Project. Omni edits old device description files in order to create new devices and allows for the subclassing of previous device features, which simplifies the print driver. Mark is also currently working as a member of the LTC Technical Review Board. He has been with the Omni Print Project for one and a half years. He has ten years of experience in print systems development, two years of experience in desktops/user interfaces, and six years of experience in DOS and UNIX development prior to joining IBM.
Resources
About the author
Maya Stodte is a contributing writer and editor for developerWorks. She can be reached at mstodte@us.ibm.com.
Copyright 2001