Transcript
Cover Story
POWERED UP When the Lights Go Out | Cheryl Salerno World-class BCP and DR program makes redundancies and backups an integral part of every facet of this electronic stock market.
E
VEN IN THE GRIP OF A MASSIVE POWER OUTAGE THAT DISRUPTED AIRLINE TRAVEL, CREATED HUGE TRAFFIC
jams, and forced many people to walk home from work, many people received some very good news on the evening of Thursday, August 14, 2003:
Officials at the New York Stock Exchange and at NASDAQ vow to reopen trading on Friday morning, using backup power if necessary … As a huge sigh of relief swept through a nervous population of trading professionals, executives at the NASDAQ Stock Market were enjoying the type of confident ease that comes only with good, sound planning.
A World-Class Program, A State-of-the-Art Facility “The outage tells a pretty good story about our recovery plan,” states Steve Randich, Chief Information Officer of NASDAQ. “We were very much surrounded by the event in that our headquarters is located in Manhattan and our critical operations are conducted in Connecticut, plus a large portion of our customers are in the northeast corridor. In terms of a business interruption, however, there was barely an impact at all.” Randich credits the corporation’s resiliency during such a large-scale event to an “absolutely world-class” business continuity planning (BCP) and disaster recovery (DR) program, which makes redundancies and backups an integral part of virtually every facet of the business.
CLICK HERE FOR A FREE SUBSCRIPTION TO CONTINIUITY INSIGHTS MAGAZINE OR GO TO WWW.CONTINUITYINSIGHTS.COM s t r a t e g i e s
t o
a s s u r e
i n t e g r i t y ,
a v a i l a b i l i t y
a n d
s e c u r i t y
Cover Story: Powered Up Formed in 1971 in a rural Connecticut industrial park about sixty miles outside of New York City, NASDAQ’s executives wisely chose to keep the market’s critical data operations far removed from the vulnerabilities of a large urban environment. That location now houses the company’s primary data center, a state-of-the-art facility created six years ago that today serves as an example of how technology and forward thinking create a strong, secure infrastructure for doing business even under the most challenging conditions. NASDAQ also owns and manages its own backup data center, which is located in Maryland, well outside the Washington, DC, area and approximately three hundred miles away from the primary facility in Connecticut.
NASDAQ executives wisely chose to keep the market’s critical data operations far removed from the vulnerabilities of a large urban environment.
Executives at the NASDAQ Stock Market were enjoying the type of confident ease that comes only with good, sound planning.
Such geographic diversity has become a key focus in government legislation developed after the events of September 11th. But, according to Randich, NASDAQ needed no catastrophic event to drive home the benefits of such planning strategies. “We consider ourselves to be leaders in this respect given that we’re a key part of the national infrastructure,” he explains. “Our Maryland backup site has been in place for almost seventeen years, long before 9/11. In fact, some of the key financial industry utilities have followed NASDAQ’s lead in developing backup data centers well outside of the New York region. According to Randich, it is this strong commitment to Photos courtesy of NASDAQ.
ensuring continuity for itself and its vast network of traders that brought NASDAQ, relatively unscathed, through one of the largest power outages to hit the northeast corridor.
own and manage our facility, which means we do not have to work against onsite fuel restrictions, giving us the freedom to store a considerable amount of diesel onsite.”
Location, Location, Location
Routing Diversity … and Then Some
“Our data network is built to withstand random outages caused by events such as backhoe strikes and other isolated events,” explains Randich. “The data center is fed by two separate power feeds from separate substations. This recent outage was actually the first time in the six years since we developed this state-of-the-art facility that we lost both power feeds on an unplanned basis.” As Randich explains, the data center went into a batteryoperated mode instantaneously as soon as power was lost and, thirty seconds later, switched over to diesel-generator backup. While NASDAQ is not unique in its use of diesel generators as a backup power source, the corporation’s rural Connecticut location offered it a very unique advantage when the August 14th outage became a prolonged event. “We have four generators onsite in Connecticut, and the operation can run on one,” Randich explains. “Plus, we have enough fuel on hand to run the facility for about a week on our generator backup system without a refill. Most metro-based businesses—including the New York Stock Exchange—must deal with the space constraints, fire code regulations and onsite fuel restrictions that come with operating out of a shared facility. Such businesses simply cannot get through a twenty-four-hour power outage without a refill. When I called the Stock Exchange floor on Thursday evening during the outage, they were operating from a dark floor because, though they had diesel backup, they were trying to conserve that in case the outage lasted any length of time.” While executives at NASDAQ considered the idea of transferring the network to their Maryland location if power were not restored within several days, they decided to run the operation out of Connecticut and wait out the outage. When commercial power was finally restored, NASDAQ engineers transferred the data center from backup mode only when they were satisfied that the power source was reliable again. According to Randich, many corporations run off of backup systems that switch automatically to diesel when power is lost and then automatically back to commercial power when power is restored. When power fluctuates for a number of hours—as often occurs after an extended outage—the stress of switching back and forth can cause some systems to fail, leaving corporations without backup or primary power. “Having a backup energy source as robust as ours gives us the freedom to not only run seamlessly until power is restored, but wait until that power is stabilized before switching back to commercial energy,” Randich says. “In the case of the recent northeast outage, we waited a full six hours after commercial power was restored to switch back. We’ve learned that it is wise to ride things out for a while, but you can only do that if you have enough fuel on hand to see you through that time. We
But a large stockpile of backup fuel cannot, in itself, guarantee business as usual in the stock market arena. As Randich notes, NASDAQ’s success in maintaining business operations despite the recent outage was due, in large part, to a diverse routing system that ensured connectivity to a vast majority of clients throughout the hours of the crisis. “NASDAQ is very much a network,” states Randich. “It has no trading floor. It’s a network of disparately located traders, who communicate with each other electronically to make the market function. We have about one thousand trader sites connected to our network, representing about 870 firms [many have backup sites tied to the network, so they count as two]. Each of these firms must follow certain standards in regard to how they connect to our network.” According to Randich, NASDAQ’s diversely routed network follows approximately twenty-two points of presence located around the U.S., six of which are located in the New York area, where a large percentage of their trader population resides. Each NASDAQ client must establish a connection to the market’s managed network using diverse local access carriers, whereby two circuits—one from each local carrier— are connected to two separate points of presence along the network. NASDAQ then ensures, by working with their telecommunications provider, that those two points of presence are diversely routed without any single point of failure to both the Connecticut and the Maryland data center. “This type of connectivity ensures that: (1) any one backhoe strike or isolated outage will not cause us to lose connectivity to that customer and (2) if we were forced to route all of our operations from Connecticut to Maryland, our customers
“The unforeseen business climate of recent years has quickly moved BC and DR planning to a top business priority across all industries. By building a world-class infrastructure that supports recovery levels and capabilities unparalleled in most industries today, NASDAQ has taken a leadership role in American business through early adoption of BC/DR planning. As a long-term business partner and trusted IT advisor to NASDAQ, Forsythe has witnessed the powerful commitment of NASDAQ’s executive team in driving the development of solid BC/DR plans.” —Michael Croy, Director, Business Continuity Solutions, Forsythe Solutions Group, Inc. (Skokie, IL)
CLICK HERE FOR A FREE SUBSCRIPTION TO CONTINIUITY INSIGHTS MAGAZINE OR GO TO WWW.CONTINUITYINSIGHTS.COM s t r a t e g i e s
t o
a s s u r e
i n t e g r i t y ,
a v a i l a b i l i t y
a n d
s e c u r i t y
Cover Story: Powered Up
NASDAQ’s success in maintaining business operations despite the recent outage was due, in large part, to a diverse routing system that ensured connectivity to a vast majority of clients throughout the hours of the crisis.
would need to do nothing, because they are already dynamically connected to both.” Randich called this type of connectivity “very unique” to the trading industry; that is, to manage the whole network from end to end, all the way from the customer premise to both the primary and backup center and to do so diversely, with guaranteed routing diversity and separate vendors on the local loop. NASDAQ backs up its elaborately designed network with a well-defined plan for reaching out to clients through various alternate methods beyond the network perimeter. First, the company manages a website called NASDAQtrader.com, which features a System Status Notification screen that updates and informs clients. The company also relies on instant messaging and popup screens to disseminate vital information to employees and clients. And, finally, NASDAQ executives always can contact traders by phone or pager via an
extensive, very up-to-date list of primary contacts. “We have a fan-out approach whereby various NASDAQ employees are tied into various clients and are responsible for contacting them regarding important information,” explains Randich. “In the hours and days following both the 9/11 tragedy and this most recent northeast power outage, we set up call centers whereby we reached out to our clients to monitor their availability in regard to trading: whether they were working out of their primary or backup site, whether they were experiencing a shortage of available traders, etc. “During events like the outage, I personally will monitor the status of the top twenty clients and depending on that information NASDAQ will determine whether trading will resume. The outage occurred on Thursday, for instance, and by 8:00 Friday morning, I had verified that all but three were confirmed for trading, which was more than enough to open the market.”
Cover Story: Powered Up An Example of Efficiency While many corporations also might boast of beefed-up recovery initiatives and a more diligent approach to business continuity (BC) in the wake of the World Trade Center disaster, Randich maintains that the event brought little change to NASDAQ’s already comprehensive BCP program. In fact, since its creation six years ago, the firm’s Connecticut data center has enjoyed a rather inspirational reputation among organizations both in and outside of the industry. NASDAQ hosts tours of the facility—two or three a week—and guests range from private firms to government agencies, including the military. “I have yet to see an individual or group leave one of these tours claiming to have seen a more robust program,” remarks Randich. Randich concedes that there is never a zero possibility for losing the data center, which is why NASDAQ has duplicated all continuity initiatives and security measures at their backup facility in Maryland, including the perimeter wall and the armed guards.
Leaving Nothing to Chance It is clear that when it comes to BC, NASDAQ has left little to chance. The corporation’s infrastructure is truly a marvel in terms of redundancies. The routing of servers in the Connecticut data center is a good case in point.
“The network is designed so that every server in the data center—which totals about twenty-six hundred—has the advantage of dual power supplies,” says Randich. “So if a power cord or power supply were to crash, the server remains up and running. What’s more, if you were to trace the two power cords from each and every server operating in the center, you’d see that they traverse a diverse route through the power distribution network, out of the building, across separate commercial power feeds, back to two separate and distinct substations.” Redundancies also mark the elaborate backup power system that sustained the market during the recent northeast outage. With the backup power of four diesel generators, which feed six separate and distinct uninterruptible power supply (UPS) systems (the site can operate on just one pair), the back up strategy is, to say the least, extensive. But peace of mind is not the only payoff that NASDAQ receives from its intricate networking of UPS systems. “Our networking design has given us an extremely unique advantage when it comes to system maintenance,” Randich states. “We’ve networked the UPS systems so that the data system never has to have a maintenance power shutdown; we can continuously operate the facility 365 days a year while still doing any and all maintenance on the plant that needs to be done. I don’t know of any other organization that has that capability, and the reason it’s important is that the power-down and power-up cycles are where you see most of the attrition of equipment. Most failures occur during these maintenance phases, putting you at risk of losing as much as 5 percent of your stock, which makes for a good bit of scrambling over the weekend.”
Sailing through the Crisis
Peace of mind is not the only payoff that NASDAQ receives from its intricate networking of UPS systems.
Reflecting on the outage, Randich cannot seem to help but marvel a bit at the efficiency with which the NASDAQ Stock Market weathered an event that brought many companies to a screeching halt. “The great thing about our stock market model is that we have hundreds of market-makers, making markets and adding liquidity to the trading of NASDAQ stock every day,” he notes. “For instance, a stock like Microsoft might have more than one hundred market-makers trading Microsoft, but you don’t really need all of them in there for Microsoft to trade—you just need a critical mass. “In the case of a severe or prolonged outage, our decision to open the market for trading the next day greatly depends on being able to anticipate how much trading activity there will be in relation to each stock. In the case of the recent outage, it wasn’t even a close call. Given the geographic diversity and the availability of trade, we sailed right through it.” CI Cheryl Salerno is a contributing editor for Continuity Insights magazine. She has six years of experience covering the BC industry. She can be reached at (215) 230-9556.
©2003 Communication Technologies, Inc., All Rights Reserved.Reprinted from Continuity Insights magazine. Contents cannot be reprinted without permission from the publisher.
