NYSE Dives Into Managed Storage

Few IT managers would relish the prospect of dealing with a data repository packed with 100 billion records, although the rapidly expanding New York Stock Exchange (NYSE) has harnessed the power of just such a system.

Seven years ago the Stock Exchange began work with business intelligence specialist 1010data to build a repository capable of providing market data to its myriad of customers, which now holds a mind-boggling amount of information.

"Theres historical trades and quotes from all of the North American equity exchanges, and also Nasdaq," says Mark Schaedel, vice president of data products at the New York Stock Exchange (NYSE) Euronext, explaining that 750 million new records are often added to the system in one day. "We also have all of our transactions, executions, and orders related to NYSE and the Archipelago Exchange that we acquired three years ago."

NYSE is on something of an M&A tear at the moment, joining forces with the Euronext network of European exchanges in a $20 billion deal in 2006, and also forging partnerships with exchanges in places such as India and Brazil.

"The platform will be supporting similar data from these exchanges as well," says Schaedel, explaining that NYSE customers can access the storage system through the organization’s NYXdata.com Website or its FTP servers. "We have, as far as I am concerned, infinite capacity available on demand."The 15-Tbyte repository currently consists of six direct-attached Dell servers with direct attached SAS storage. 1010 Vision, a software tool developed by 1010data, runs across repository to analyze multiple data streams.

"That creates a grid on which queries can be parallelized and run very efficiently on that commodity hardware," explains David Frankel, 1010data’s vice president of business development. "The beauty of using a commodity hardware architecture, is that we can just add hardware as required."

Direct attached storage also offers a greater degree of reliability, according to Frankel.

"The issue with centralized storage such as SANs is that it has management benefits, but it’s not well suited to environments where you need dedicated, guaranteed performance to individual servers," he says. "Each server has its own dedicated bandwidth to the storage system that it can rely on."

NYSE’s Schaedel told Byte and Switch that the next big project for the data repository will be to integrate data from Euronext, a "certain portion" of which has already been added to the system for internal use."We started with some internal data so that our management can have some insight into what the international markets are doing," he says. "Where appropriate, [1010 data] will be supporting data from Euronext and any other partners that we acquire as part of our global expansion strategy."

Although the data repository is managed remotely by 1010data, the Stock Exchange, unlike many of the vendor’s customers, keeps the hardware within its own data center.

"This means that any data can be stored in the data center in a safe way, but still accessed globally," alluding to the stringent compliance regulations at play in the financial sector. "The data center is behind the [Stock Exchange] firewall and the API that we use to access the data is secure – only certain data sets can be accessed via the API in accordance with our security model."

Although Schaedel would not reveal the value of the Stock Exchange’s deal with 1010data, he described the rationale behind choosing a semi-managed service.

"It’s not our core business to manage data – maintenance of historical data repositories like this requires some unique expertise," he says. "It has really simplified our business, it has allowed me personally to focus on staffing and resources."Have a comment on this story? Please click "Discuss" below. If you'd like to contact Byte and Switch's editors directly, send us a message.