E.B. Writers of Dallas

HostingTech Magazine, December 2001

By Esther M. Bauer, Correspondent

The fragility of the world’s strongest financial district was laid bare when the airborne destruction of Sept. 11 claimed thousands of lives and erased the towering landmarks of the New York City skyline.

The scope of the problem was apparent almost immediately after the first jetliner slammed into World Trade Center tower No. 1 and then a second jet plowed into tower No. 2.

Nasdaq executives were in a routine planning meeting in their offices at 1 Liberty Plaza across from the twin towers when they saw the first plane hit. “They were already in a crisis mode and in a decision-making process when the second plane struck,” said company spokesman Mike DeMeo.

Even as U.S. Securities and Exchange Commission officials evacuated their regional offices at the World Trade Center and Nasdaq employees fled their building, a collective decision together with the New York Stock Exchange (NYSE) and Nasdaq’s sister company, the American Stock Exchange, was made not to open for business that day.

The decision would keep the securities markets closed until the following Monday due not to loss of data caused by the terrorist attack, ensuing inferno, and tumbling buildings but rather to the widespread loss of electronic connectivity and telecommunications among the exchanges, brokerages, traders, and participating firms—regardless of actual damage to their buildings or data centers.

Damage to the American Stock Exchange, for instance, was so severe that Amex traders were given space on the trading floor of the NYSE, which suffered no physical damage. Its point-of-sale applications were untouched and fully operational.

On outward appearance the NYSE could have conducted trading that same day, save for the interdependent connectivity that’s as critical to the wellbeing of the stock market as blood is to humans.

More than 90 percent of the NYSE’s buy and sell orders arrive over data lines, and without high-speed broadband connection to its members, the exchange’s wired realm of 200 miles of fiber optic cable and 8,000 telephone circuits might as well be an electronic island unto itself.

Nasdaq’s electronic infrastructure was untouched, too. All of its trading is done electronically, and its data center is safely in Connecticut. Yet, connectivity to many of its participant firms was gone—roughly 60 percent were located in Lower Manhattan’s decimated financial district.

Instinet, an electronic stock trading network spawned by Reuters, closed its operations in the U.S. and Canada after losing its offices and a data center in tower No. 1 along with connectivity to most of its northeastern clients who represent 50 percent of its business. Also lost were two of its three top senior technology managers. The two, responsible for managing 70 percent of the firm’s technology operations, were attending seminars on the tower’s uppermost floor and were among the thousands killed that day.

The facilities of the National Board of Trade, a commodities trader of coffee, sugar, cotton, cocoa and orange juice, were demolished along with all other offices in 4 World Trade Center, which was squashed when tower No. 2 crumbled.

That same day the Board of Trade relocated to its backup site in nearby Long Island City, but it, too, faced the telecommunications and connectivity problems of the other securities markets.

Verizon’s major switching center near the World Trade Center was inoperative due to debris from the fallen towers. Some 200,000 phone lines and three million corporate data lines were impacted, and all direct lines beneath the rubble of the World Trade Center were now useless. The damage to major switching lines and Central Offices of telecommunications carriers effectively put the securities markets out of business until connectivity circuits could be remapped and rerouted.

Although each securities market had multiple combinations of backup plans, data recovery sites, dual connectivity links with traders and participating firms, none had considered the extent of the connectivity loss.

“You’re only as strong as your weakest link . . .what makes this market and this industry work is the connectivity . . . major introducers of order flow were either not connected to their customers and couldn’t get orders in the first place or were not connected to market execution centers and couldn’t deliver those orders for execution,” Bob Britz, a NYSE group executive vice president, said following the market’s reopening.

At the NYSE and other markets, the interim six days before the stock market reopened were filled with round-the-clock collaboration among technology teams, telecommunications carriers and vendors. Determining which lines were workable and which had to be rerouted was conducted before the markets could even begin to think about orchestrating connectivity tests with individual brokerage houses, client firms and other market centers.

Nasdaq’s two emergency command centers logged more than 675 calls concerned mostly with connectivity or operations problems. A dozen of its member firms had resided at the World Trade Center complex and were given special attention to help get their backup sites operational and connected, said Steve Randich, chief information officer and vice president of Operations and Technology.

Final testing for market readiness included ping tests on a firm-by-firm basis, and by Saturday, approximately 98 percent of Nasdaq’s share volume was operational for the market’s reopening, Randich said.

At Instinet, backup at sites in Boston and New Jersey ensured that no data was lost, but the connectivity problem with client firms was desperate.

Most of Instinet’s 600 impacted clients had been dually linked to the alternate New Jersey data center. It was a fail-safe plan that failed because the clients’ connection to that site also ran through the same telecommunications switching centers that were destroyed.

“The fact that our alternate data center was fine didn’t necessarily mean we could actually communicate with our clients on Wall Street. We had redundancy on our end, but if the client is based on Wall Street and that line goes through the grid that’s down, it’s like having no line at all,” said Jean-Marc Bouhelier, executive vice president of Instinet’s U.S. Institutional and Professional Business, Technology and Global Product Development.

“The crisis made it clear. If you can process data, but your client cannot connect to you, you’re basically out of business. We’re an electronic broker, so losing connectivity to 600 clients was obviously a big challenge to our business,” he said.

Instinet began trying to reconnect client firms by calling them one at a time to find out the location of their disaster recovery sites and if necessary repoint the data circuits to their new location.

But calling them was no easy feat, considering that voice transmission lines throughout the area were quickly overwhelmed, and the destruction of the towers, which had also been wireless antenna sites, had rendered cell phones useless. In many cases, the only method of contact was through Instinet’s sales staff, who had the clients’ home telephone numbers.

Even then, Instinet was still unable to reach at least 150 client firms by the eve of the market’s reopening, and when the market did reopen Monday, only 92 percent of the client firms were in a position to trade. It wasn’t until the end of that week that nearly 100 percent were ready.

The disaster also revealed a particularly unsettling problem for some clients who only thought they had a second line directed to Instinet’s alternate data center in New Jersey. Yes, they had dual lines, but both erroneously pointed to the destroyed Manhattan data center. The only immediately available solution was to offer them an ISDN connection to the alternate site.

“It at least gave them a way to reconnect to us,” recalled Bouhelier.

Radianz, a global IP (Internet Protocol) network services provider for the financial industry, played a key role in solving Instinet’s connectivity problems. The firm, which is also an affiliate of Reuters, will be a major player in preventing future connectivity problems, said Bouhelier, who expects the firm and other vendors to help devise network redundancies that guarantee uninterrupted service levels to clients.

The New York Board of Trade was in the fortunate position of being the only open-outcry exchange in the world with an alternate trading floor facility where traders stand in a pit, yelling their buy and sell orders.

The Board’s Long Island City backup facility is supported by Comdisco, a computer leasing and data-recovery company, and had always been a hot site. From a technology standpoint, nothing had to be recovered, because everything was already mirrored and operating as a dark or invisible site.

But as soon as the Board’s staff of 263 moved in to serve its 1,000-seat membership, the insufficiency of landline voice and data capabilities was immediately apparent though a solution wasn’t. The government had temporarily banned commercial and private aircraft from flying to prevent additional terrorism, but the ban also made delivery of the needed switching equipment impossible prior to the market’s reopening—or so it seemed.

Siemens, a provider of communication network solutions, gained the help of the U.S. Army to fly in the needed switch from Texas, recalled Steve Bass, senior vice president of Information Technology for the Board of Trade.

“We had it set up and ready for Monday . . . it was critical to reestablishing communications both at the voice level and the data level with the rest of the world that we do business with,” Bass said.

Still, it may be a year before normalcy returns. The Board must either expand its cramped Long Island City location or acquire space in a larger building. Currently, the Board’s five agricultural markets are each trading one and one-half hours daily instead of the previous practice of simultaneously trading four to five hours per day.

Bass compared conducting business in the temporary quarters to “having a summer cabin in the mountains where you would normally spend a week but wind up spending a year. You’re not half as comfortable as you thought you’d be, but you manage. That’s what we’re doing here, getting the job done. We’ve accomplished our goal of saving the business.”

The fact that the twin towers which symbolized this nation’s financial might are now gone has been made less unnerving for many in the financial services industry by the collaborative spirit that enabled the resilient stock market to reopen less than a week after the terrorist attack.

“It was really something to see,” said Bass, a 35-year veteran of the technology business who confesses continuing amazement at the collaborative energy and resourcefulness. “It’s a testimony to the people involved. This certainly has to be the all-time challenge in my experience. This is definitely the one, but if another one comes along that’s bigger, I think I’d rather not be here,” he said.

Now that he’s gone through the crisis, Bass doesn’t believe any amount of planning could have averted the communications disruption or that a foolproof strategy is within the scope of the business community.

“The technological infrastructure it would take to overcome this kind of problem, to have it sit idle waiting for this type of situation to occur again would be too expensive. If everybody was putting out the money to do something like that, it would tie up a lot of capital probably better spent elsewhere,” he said.

Locating the financial district over a broader geographic area is one solution being tossed around, but its practicality also is in doubt.

Bouhelier of Instinet believes significant lessons must be learned from the disaster.

“We followed the data center strategy, which would allow us to be up and running if we lost one center, but that wasn’t enough for an incident of this magnitude. You have to be okay on your side. The client has to be okay on their side; and the communications provider, the infrastructure, has to be okay, too,” he said.

“The biggest single problem for all of us, for the market, and for most organizations is that we haven’t done enough thinking about communication and network strategies. If you don’t think through the communication and network aspects, and if you don’t understand the infrastructure you’re relying upon—you potentially have a big weakness in your infrastructure.”

The scope of technological disruption has forever heightened Bouhelier’s awareness of the need for better protection and has altered his definition of geographic disparity for disaster recovery sites.

“With this kind of incident, it can’t be 10 miles away on the other side of the river. It probably needs to be a bit more drastic than that . . . historically, people just worried about straight redundancy of data centers, but that kind of stuff has taken on a new dimension. Now, people will have to think a lot harder about what it means to be fully redundant,” he said. ###