Mindstorms Not Just a Kid's Toy

Grownups are gleefully turning smart Lego bricks into autonomous gadgets

By Paul Wallich, Contributing Editor
IEEE Spectrum

September 2001

Robotic roller-coasters...mobile intelligent cat toys...a robotic disk jockey...an autonomous self-propelled video camera...a tiny mobile image scanner that builds up pictures by crawling millimeter by millimeter across a page: all these are gadgets produced by an R&D organization that few know even exists.

A Mindstorms robot's brain--a programmable Lego brick.

The inventors are not part of a big research university or at some forward-looking computer company, but a loosely knit group of engineers who build gizmos out of Lego bricks. Developed by the Billund-based Danish toy company, some bricks contain microcomputer-controlled sensors and motors, and all are components of a three-year-old product called Lego Mindstorms.

Originally intended for use by children at home or in the classroom, Mindstorms has instead enraptured tens of thousands of adults. Engineers, academic researchers, and hobbyists have reverse-engineered its firmware, hacked together new programming environments, and shared recipes for unanticipated ways of connecting it to the outside world.

While Lego Co. may have been taken aback by the extent to which Mindstorms had transcended the child's toy category, the company is happy to have the millions of dollars in extra revenues it brings in. In response, it is expanding its range of computer-controlled toys, publishing some of Mindstorms' internal documentation, and enlisting fans as advisors and testers for new versions of the product. It even sponsors symposia to discuss Mindstorms theory and applications, the most recent of which was held in August in Fort Worth, Texas.

Bricks that think

Lego and the engineering community share a long history. As far back as the early 1980s, automotive engineers reportedly used the brightly colored interlocking bricks and hinges--then years away from built-in microcomputers--to make scale models of proposed vehicle assembly lines. In 1984, the company began working with researchers at the Massachusetts Institute of Technology (MIT), in Cambridge, to adapt its motors, switches, and other parts for computer control. Indeed, the RCX controller that serves as the brain of every Mindstorms robotic gizmo is based on what the company learned by sponsoring the development at the MIT Media Laboratory of a "programmable brick."

RCX stands for robotic command explorer. It is an oversized Lego brick [left], a little larger than two stacked decks of playing cards. On the outside are a liquid-crystal display (LCD), four control buttons, and six wiring connectors. Inside are a Hitachi H8 central processing unit (CPU) with 32KB of RAM, space for batteries, and an infrared transmitter and receiver, which communicate with a base station plugged into a PC port.

Three of the external wiring connectors are outputs, each supplying a user-controlled voltage to a motor, a lamp, or anything else that will run off half a dozen AA batteries. The remaining connectors can read data from a variety of sensors (one light and two touch sensors, all housed in Lego-style bricks, are included with the basic kit, along with two stepper motors). The LCD can display motor settings, sensor readings, or diagnostic information about the status of the RCX and its software.

To construct a working gizmo [see photo], a Mindstormer typically builds the mechanism out of the RCX, motors, sensors, wire, and the roughly 700 bricks, beams, gears, axles, and wheels included with the basic kit, which lists for US $199. Then, to program the RCX, an aspiring inventor runs a graphical environment provided by Lego for the PC. By connecting blocks representing commands such as READ SENSOR LEVEL or TURN MOTOR ON, he or she creates a program, downloads it to the RCX by an infrared link, and pushes the appropriate control button to start it running.

The PC sends the program to the RCX in the form of byte-codes. These are sequences of 8-bit instructions that specify such actions as turning on a motor, waiting for input from a sensor, or jumping to another point in the program. A little more than half the RCX's 32 KB of RAM is taken up by firmware--downloaded from the PC when the RCX is first powered on--that interprets these byte codes and executes the H8 machine code that actually makes things happen.

Mindstorms Madness
A few of the offbeat inventions created by Mindstorms devotees include [counter-clockwise from top left] a toilet bowl scrubber by Andrew Evans; a robot that simulates an airplane's motion in flight by Mario and Giulio Ferrari; a lightbulb changer by Evans; and a skier, also by the Ferraris.
Top left & bottom right: Andrew Evans; Top right & bottom left: Mario & Giulio Ferrari.

From these relatively simple parts, hobbyists have managed to build a mind-boggling array of widgets. The list includes robotic trash trucks that pick up and deposit miniature dumpsters, rolling and walking robots, a light-bulb changer, and maze-following robots [see pictures]. Hundreds of Web pages put up by fans around the globe supply the details for building these creations.

That Mindstorms is such a hit with adults does not strike Fred Martin as strange. Said the former MIT researcher, who played a pivotal role in the programmable brick development, "It's the toy I would have wanted to have" when he was younger, and he still wants it today. The goal of MIT's research, Martin said, was to create "materials that make ideas accessible," and the programmable brick does exactly that for robotics and for computer programming. People who use Mindstorms learn by building something tangible.

Above all, Mindstorms makes ideas about robotics accessible to people without sophisticated mechanical-engineering skills or access to a machine shop. "Lego has changed from a childhood toy to a viable mechanical prototyping system," said hobbyist John Barnes, who lives in upstate New York. With Mindstorms, anyone can build computer-controlled mechanisms by simply snapping parts together.

Where an earlier generation of hobbyists struggled to build parts to the submillimeter tolerances required for a working robot arm, "Lego parts are made to exact lengths," with an accuracy measured in tens of micrometers, noted widget-builder J.P. Brown of Chicago.

A hacker's dream

Pleased as many would-be roboticists were with the idea of Mindstorms, they were often less than satisfied with the reality. Among their complaints is Lego's simplistic programming language. It allows a mere 16 variables for holding sensor values or other data, and gives only coarse control over motor speeds. Its subroutines for handling repetitive tasks such as navigating around obstacles are quite limited. Multitasking is slow, as is response to sensor inputs.

Unlike youngsters, who might well toss a disappointing new toy back in the closet, some adults had the tools and training to bring Mindstorms more in line with their desires. Besides, they had the hacker ethos: if it doesn't work, fix it; if it does work, take it apart.

Kekoa Proudfoot, a graduate student at Stanford University in California, was the first to pry off the cover of his RCX and tell the world what was inside. He quickly decoded what commands its built-in firmware could carry out. David Baum, in real life a software engineer at Motorola Inc., cracked the communications protocol for the IR link. Mindstormers exchanged information through mailing lists hosted by hobbyist Russell Nelson, and through LUGnet, Lego Users Group Network, which had already been in operation for some years.

Development during those early days was fast and heady, recalls Markus Noga, now a doctoral candidate in computer science at the University of Karlsruhe, in Germany. In one rapid exchange, he and David Baum posted versions of a new program download protocol on the Internet every few hours, culminating two days later in a version that was four times the speed of the original.

Baum also quickly devised a new programming system to develop RCX code in a dialect of the C language he called Not Quite C (NQC). Instead of programming by manipulating graphical blocks within Lego's PC software, Mindstorms' hackers using NQC could write code with a conventional text editor in a familiar language. Baum's compiler then produced byte-codes that could be downloaded and interpreted by the RCX firmware. Besides making it easier to write and exchange code, NQC offered somewhat better control over the RCX hardware than did Lego's simple programming blocks.

Meanwhile, Ph.D. student Noga decided to ditch Lego's firmware and write his own. He devised a set of more efficient routines for controlling the brick's behavior, which he called LegOS. It can be used to write C programs that can be compiled to H8 machine code and downloaded for direct execution by the CPU, potentially running hundreds of times faster than the interpreted byte-codes.

In another application, Ralph Hempel, an embedded-systems engineer in Owen Sound, Ont., ported a variant of the Forth language to the RCX. He called it pbForth. Unusually for a Mindstorms programming system, pbForth does not create a program on a PC and download it to the RCX. Instead, the PC is treated as a simple terminal connected to the Forth development system running on the Hitachi H8 CPU inside the RCX brick. The system takes less than 10 kB of memory, leaving plenty of room for user programs. Hempel gave two reasons for writing the system: Forth is well suited for programming CPUs that communicate directly with the real world, and other fun projects (like reverse-engineering the brick or writing LegOS) were already taken.

Misjudging the market

Pent-up demand from would-be robot hackers transformed a toy originally designed for the 12-year-old-and-up home and educational market into one with more adult than adolescent buyers. Lego was not entirely surprised by the prevalence of Adult Fans of Lego (or AFOLs, as they're known within and without the company) or by its voracious appetite for the $200 toy, said Russell Stoll, creative director of Lego's community Web site. Still, the depth and breadth of the fans' continuing interest has been remarkable.

In response, the company is offering extra documentation, such as the details of firmware inner workings in its robotics Software Development Kit (SDK) and a Web site on which AFOLs can exchange information or post pictures and text describing their inventions.

Some Mindstorms hackers were unprepared for this corporate support, which included Lego's sponsorship of the first Mindfest conference, a gathering of academics and hobbyists at MIT in October 1999 to celebrate the product's first anniversary. As David Baum told IEEE Spectrum, for the first few months after he released NQC, he half-expected to find a letter from Lego's lawyers in his mailbox, telling him to cease and desist from meddling in the company's intellectual property. Instead, he ended up meeting Lego designers and forging friendships with some of them.

That might not have been the case earlier. Lego has a history of aggressively defending its designs and trademarks, particularly when other companies have produced plastic bricks that interlock with Lego's own. At the Mindfest conference, said Noga, one Lego official commented that even a few years earlier, the company might have considered legal action against the hackers, instead of responding with open arms and a Web site.

Besides encouraging adult inventiveness, Lego has also responded with new releases that it hopes will be easier for children to use, said Lego public relations manager Michael McNally. As was pointed out by hobbyist and engineer Michael Gasperi in Milwaukee, children often find the current product difficult. He recalls buying the Mindstorms kit in hopes of helping his eight-year-old daughter understand what he did at work every day. The two worked through the initial tutorials together, but by the time their first robotic creation was ready to be debugged, said Gasperi, "I was alone."

The newest RCX programming system, according to McNally, contains more sophisticated functions: for example, a single command will move an RCX-powered vehicle forward or back, right or left, rather than requiring a user to send individual forward or reverse commands to the RCX motor outputs. The company also has released two series of similar but simpler products, the Cybermaster and Scout, which offer built-in motors, less flexible programming, and hence perhaps a less bewildering array of choices. The idea, said McNally, is to provide a compelling "out-of-box" experience for both sides of Lego's diverse market.

These new products do not water down the original Lego design, according to Lego's Web site director Stoll. He pointed out that the next release of the RCX programming environment, RIS 2.0, also includes a text-based scripting language, so that users can type programs without first downloading onto their PC an entirely different set of programs (such as NQC or LegOS). The company has also released the Vision Commander, a combination of camera and PC software that can send messages to the RCX over its IR link, based on changing events within the camera's field of view.

Meanwhile, the hacker community has been producing new hardware as well. Gasperi, for one, has published on the Web designs for a series of simple homemade sensors for conditions ranging from touch and temperature to relative humidity and skin conductivity. John Barnes, an image-processing programmer in real life, has made a small business of supplying more-complex input devices, all built into hollowed-out Lego bricks so that they will snap together with other Lego parts (and so that Lego's legal department will have less of an incentive to come after him).

One of Barnes's staples is an ultrasonic range finder that plugs into an RCX input port and tells the RCX how far it is from the nearest solid object. At the top end of his line, he has built a prototype camera brick that tracks an object in its field of view and reports x and y coordinates back to the RCX. That unit, he notes, includes a microprocessor and several 50-MHz field-programmable gate arrays with more than 10 times the computational horsepower of the H8. "It can be a bit of the tail wagging the dog," he told Spectrum.

Barnes has also begun work on possible replacements for the RCX. He is using 18 I/O ports instead of three, a 1-Mb/s RF connection to a PC instead of a lowly IR link, and external power connections so that users will not be limited by the life of AA batteries. "Power is a real challenge," he gripes, even as he agrees with others that the RCX designers probably made the best tradeoff available among ports, battery life, and weight. Barnes has also eliminated the LCD, trading it for a PC-based virtual front panel connected to his brick by RF.

While Barnes and his ilk are taking the complexity of Mindstorms to new heights, others are working on simplifying the programmable brick even further. Former MIT researcher Fred Martin has moved to a spin-off company in hopes of commercializing one of his more recent research projects, a gizmo known as Cricket. Martin believes an 8-bit processor with 32KB of RAM is overkill. His Crickets use a less powerful controller with only a few hundred bytes of RAM to do almost the same job at a fraction of the price.

The Cricket design has only two sensor inputs and two motor outputs, but individual Crickets communicate with one another through IR links, so that any number of them could be used in a single system. For example, Martin said, a toy robotic arm might have a Cricket-style controller at each joint and several at the gripper, rather than a single massive CPU trying to control all the motions and drive all the motors.

Building tools or building toys?

Ironically, many builders of the most complex software and hardware additions to Mindstorms have found that answering questions and exchanging information with other AFOLs leaves them with little time to build new Lego creations. Baum, for example, has spent his spare time re-engineering NQC to improve its performance and make it possible to add new features requested by his fellow hackers. Hempel, Gasperi, and Barnes report similar experiences. Barnes, in fact, is trying to find a company to take over his sensor designs, which he currently handcrafts from printed-circuit boards made to order by an Internet-based prototyping service.

Six-legged pet built by Jin Sato includes a home-built light-emitting diode array for displaying different eye expressions.
Jin Sato.

But there are plenty of AFOLs out there willing to make use of new Mindstorms tools to build gadgets of ever-greater complexity. Consider Bob Fay, a Southern California technical worker who has spent his career repairing numerical controllers and other aerospace manufacturing equipment. He now builds lathes, milling machines, and solar-powered trucks out of Lego parts. (Fay's milling machine cuts nothing harder than dry toast, but it's a worthy successor to the pen-plotters and printers others have built.) Or there's Mike Onorato on Long Island, N.Y., whose RCX-based vending machine dispenses cans of real soda in exchange for coins or bills, which it stores in a cash box locked with a Lego key. And Japan's Jin Sato has created a pet that displays various eye expressions [photo].

[ Image ]
With J.P. Brown's Rubik's Cube-solving robot, a camera photos the cube's configuration and a PC works out the shortest number of moves needed to solve the cube. The solution is then transmitted to the Mindstorms robot.
Jean-Marc Giboux/Liason Agency Inc.

Another inventor is J.P. Brown, whose Rubik's Cube-solver takes two RCX modules and thousands of pieces [photo]. Although Brown had last looked at a Lego brick when he was 10 years old, he caught the bug again two years ago after running into a 14-year-old Mindstorms addict at a party in Research Triangle Park, N.C. On his return home to Chicago, he checked out the toy, discovered NQC, and promptly built a robot to solve the Towers of Hanoi problem. In this computer-science chestnut, a series of disks of different sizes must be moved from one "tower" to another (by way of a third "tower") without a larger disk ever being placed atop a smaller one. Then, after building a few walking robots, Brown was in a neighbor's apartment and noticed a disarranged Rubik's Cube. The sight triggered a memory of an article he had once seen in Byte Magazine about solving the cube in 60 moves, and he was off.

The cube-solver illustrates many of the issues that an aspiring builder must solve to create working widgets out of child-grade plastic. For example, when building the robotic grippers that hold the cube in place and turn it to realign its faces, Brown had to trade speed for torque to avoid wrenching his Lego bricks apart. Each turn of the cube's faces takes 30 seconds.

Even with the machine moving that slowly, Brown said, he went through six iterations of the gripper design before he found one that could hold the cube securely. In addition, the robotic arm flexes too much to give accurate readings at the "elbow," so that the sensors that tell when a turn is completed had to be mounted at the "wrist," right next to the gripper.

On the computing side, the RCX is relegated to controlling the grippers. Brown's PC calculates the shortest possible sequence of moves--usually less than 20--using a table of cube permutations that requires several megabytes of memory. (A camera zeroes in on the cube's faces to determine which of the millions of possible disarranged configurations the cube is displaying.) Brown commented that a 60-move solver would probably fit on the RCX, but "at 30 seconds per move, no one is going to watch the thing."

Is the cube-solver of any use? According to Brown, several acquaintances who have heard about his device have brought him their old cubes to be set shipshape.

Søren Rolighed's Party Optimizer [top] watches a dance floor. If no one is dancing, the robot keeps trying new CD tracks until it spots people dancing. His typewriter [bottom] uses a plotter to print 5-by-7-pixel characters on a roll of paper.
Søren Rolighed.

Similarly whimsical is Denmark-based Søren Rolighed's Party Optimizer [photo]. This Mindstorms robot uses the vision module to study an adjacent dance floor and determine whether anyone is moving on it. If not, a subrobot moves across the face of Rolighed's CD player to skip to the next track; continued inactivity leads it to change the CD for another, hopefully easier-to-dance-to selection.

Rolighed, a software engineer and database administrator who, like most Danes, has been playing with Legos almost since birth, has also built an RCX-controlled typewriter [previous page] and an RCX-controlled slot machine. The slot machine, he said, had been on his list to build from conventional Lego parts for many years, but it was only when the RCX came out that he could make it work. The typewriter uses a combination of fixed and moving sensors to figure out which key has been pressed, with a rudimentary pen plotter and paper feed to print out letters in a 5-by-7-dot matrix. (The slot machine is somewhat simpler, because it must merely sense the position of its three wheels when they stop rotating, and then count the coins it pays out accordingly.)

Feeding community spirit

From Lego's point of view, encouraging the Mindstorms community to invent makes good marketing and product development sense. The new RCX firmware benefited from more than a year of thorough examination by beta testers, who put it through paces undreamed of by its designers. And Lego marketing executive McNally points out that the company is releasing a new builder's kit with which customers may recreate at least a couple of the gadgets that Lego's master designers put together for trade shows and marketing tours.

But the individual participants get a great deal back as well: information, recognition, and support. Brown laughs about his "odd, god-like status" among a select group of 15-year-olds, and said it was "a little scary" when the son of a fellow Lego hacker suddenly recognized him as the creator of the legendary cube-solving machine.

When Rolighed found people like Brown, Baum, and Noga on-line, his immediate reaction was relief that he was "not the only not-so-young person still playing with Lego." He quickly became part of the on-line community, impressed with the time that older, more experienced builders spent with their younger compatriots.

Indeed, when there's new information to be exchanged, dozens of articles a day may be posted at Lego's Forums, or at the 1100-member LUGnet site, or in the usenet rec.toys.lego newsgroup. Tens of thousands of users have downloaded the NQC compiler, and even as arcane a programming language as pbForth finds hundreds of new users every month. What's more, user groups meet in cities around the world, from Seattle and Toronto to Tokyo and Sydney.

Some Mindstorms gurus have even found that their work on software and hardware for a like-minded community has changed how they approach their real jobs. NQC taught David Baum that his interests lay more in software research than in product development. "I realized that I like to try new things and see what takes," he told Spectrum. Repeated redesigns of the program's internals have also honed his software-engineering intuitions.

Hempel reports a similar evolution. Writing pbForth and supporting the people who want to use it, he said, has turned him from a design engineer into an educator. Using none other than the RCX, he is training other software engineers in embedded-systems design. (Even as a learning tool, Mindstorms seems to have found a firmer niche in college-level and graduate education than in secondary schools.)

Looking for convergence

Where will Mindstorms go from here? With version 2.0 of the RCX firmware due this fall, the original hardware may have gone as far as its designers can take it, and what directions it should take next are unclear. Lego is mum about its detailed plans. Even so, marketing executive McNally suggests that more RF communications links are in the offing, while Web site director Stoll is already looking forward to Mindstorms 3.0, which he claims will make even the most demanding professional users happy.

LegOS inventor Markus Noga has abandoned the RCX for a career in XML software tools, but hopes that hobbyists will soon see an RCX with an expansion bus of the kind once found in the Acorn BBC Model B or Amiga home computers. That way, he said, third-party developers would have full access to the internal architecture of a programmable brick.

Some Mindstormers, meanwhile, like the RCX just the way it is. The fun part of designing Mindstorms projects, said Baum, is figuring out how to do what you want despite the brick's constraints. Rolighed goes even further, limiting himself strictly to Lego-produced parts and eschewing homebrew sensors or motors. He has always made it "a point of honor," he said, "to use only the facilities available." /p>

As Barnes puts it, there are all levels of observance among adherents of the Lego religion, from the ultra-orthodox to the agnostic. Some of his products even blur the line between computer control and conventional Lego wires and switches: he has built radio-controlled modules that contain tiny microprocessors, and making them programmable would be a simple step. (Lego has announced that it, too, would produce generic radio-controlled bricks.)

Perhaps, as the price of microcontrollers and communications links continues to fall, the boundary between dumb and smart components will blur even further. It would be a strange twist of fate if the "ubiquitous computing" future that computer pundits have predicted for the past 20 years makes its most pervasive initial appearance in the form of intelligent, interlocking toy bricks.