A.T.&T. Pinpoints Cause of Phone Disruption

By Calvin Sims
The New York Times

January 17, 1990

The American Telephone and Telegraph Company said yesterday that it had traced the huge disruption in long-distance service that occurred Monday to a single computer program that malfunctioned and caused a chain reaction in the network.

Company executives said the malfuction occurred in a program at a giant computer switching station in the New York metropolitan area, causing the computer to send out alarm messages to other switching stations.

As a result, many of the 114 switching stations across the A.T.&T. network were essentially frozen, and for much of Monday afternoon and night about half of the national and international long-distance calls were not connected.

By disclosing the problem and saying they had solved it, company officials sought to allay their customers' concerns about the network's reliability. The officials also offered business and residential customers some minimal reimbursment for losses that may have resulted from the service disruption.

Businesses that rely on the telephone to conduct their affairs reported that they lost tens of millions of dollars in missed sales when the switching system failure disrupting most of the long-distance, international and toll-free 800 calls across the country.

For residential consumers, the breakdown of the company's network, which is regarded as the most sophisticated telephone systems in the world, caused confusion and frustration.

In a meeting with reporters at the company's network operations center in Bedminster, N.J., Robert Allen, A.T.&T.'s chairman, said company engineers worked through the night to resolve the problem and that he doubted that it would occur again.

Mr. Allen said that A.T.&T. had received many complaints about lost revenue from business customers who lease toll-free 800 and outgoing long-distance lines. But he said the company is not liable for business losses that result from service disruptions. He said that the company would honor service agreements that require A.T.&T. to pay business customers $20 or $50, depending on the type of toll-free service, for each billed line that was out of service.

'Didn't Live Up to Standards'

To console its residential and business customers who had difficulty completing long-distance calls on Monday, A.T.&T. said that it would seek permission from the Federal Communications Commission to offer customers a day of discount calling, comparable to the price cuts offered on any holiday. Monday was a national holiday commemorating the birth of the Rev. Dr. Martin Luther King Jr. ''Even though it was a one-time hit to the network, it was certainly the most far-reaching service problem we've ever experienced,'' Mr. Allen said. ''We didn't live up to our own standards of quality. We didn't live up to our customers' standards of quality. It's as simple as that. That's not acceptable to us.''

The network failure is a major embarrassment for A.T.&T., the nation's largest long-distance carrier, which has tried to differentiate its service from that of its competitors by advertising the quality and dependability of its network. Telephone industry analysts said the long-distance disruption was a major blow to the company's reputation and gives the company's rivals, the MCI Communcations Corporation and the U S Sprint Communications Company, new firepower in their battle to gain long-distance customers.

'Aura of Invulnerability'

''The conventional wisdom among telecommunications managers is that you don't get fired for choosing A.T.&T. because it has this aura of invulnerability,'' said Frank Governali, an analyst with First Boston. ''That wisdom changed yesterday. The failure of A.T.&T.'s network helps give the perception that the other guys can do just as well.''

MCI, based in Washington, was quick to take advantage of A.T.&T.'s mishap. ''People are starting to review whether they want to have their service with only one carrier,'' said Kathleen Keegan, an MCI spokeswoman. ''We are seeing a slight increase in calls from customers who want to sign up or just ask questions.''

Richard Firestone, chief of the common carrier bureau of the Federal Communications Commission, said that the agency had been closely monitoring the situation and was pleased that A.T.&T. had acted quickly to restore service. ''The commission has long been encouraging competition in the long-distance industry and one of the rationales is that it provides consumers with alternatives in case there are problems like A.T.&T.'s,'' Mr. Firestone said. ''This situation has served as a learning experience for us all.''

A.T.&T. said that although its operators initially refused, even when asked, to tell subscribers that they could place their calls with other long-distance carriers by dialing access codes, the operators were later instructed to give the codes to residential customers who requested them.

The telephone company also said that at the request of a business customer, it attempted on Monday to switch the customer's toll-free 800 service over to a competitor's network but that its engineers were unable to make the connection. A.T.&T. said that it had informed both MCI and U S Sprint of the difficulties it had experienced on its network.

Problem Is Pinpointed

The problem turned out to be an error written into the original program that instructs the switching computers where to route the telephone calls. A.T.&T. computer scientists wrote the program, which finds the most efficent path for long-distance calls to travel over the network. The company said that the program had been installed in all of its switching stations and that there had been no major problems with it until Monday.

Company officials said that the malfuction could have occurred at any of the switching stations and that they did not know yet why it occurred in New York. But once it did, it caused all the other programs to start generating the alarm signals.

Once the problem was pinpointed, A.T.&T. officials said, they were able to override the problem program with one that was error-free. They said it would be several days before they would have a permanent solution to the problem.

Asked if the company was certain that the problem was not caused by a rogue computer program like a virus or worm planted by a computer hacker, Walter Murphy, an A.T.&T. spokesman, said, ''We can't categorically rule it out but we remain confident that it was not sabotage that caused this disruption.''

The switching problem occurred at about 2:25 P.M. on Monday and was resolved at about 11:30 P.M. The giant telephone company said that of the 148 million calls that were dialed on its network Monday, about 56 percent, or 83 million calls were completed.

One company, Idelman Telemarketing Inc., based in Omaha, estimates that it lost $65,000 to $75,000 in service revenue when most of its outgoing long-distance telephone lines crashed, forcing it to send about 800 service representatives home early for the day. Idelman, one of the nation's largest marketers of products by telephone, estimated that its clients lost $100,000 to $200,000 in merchandise sales.

Copyright 1990