NYSE Euronext는 소프트웨어 라이프사이클 관리를 위하여 Helix를 사용합니다

Using Perforce, NYSE:

• Tracks creation, quality assurance, release, staging, distribution, utilization, and deprecation
• Maintains all production client specs
• Controls Helix Core access and uses Helix triggers to detect attempts to modify configurations, scripts, or binaries
• Runs nightly comparisons of files on systems with the meta-map of production in Helix, reporting any noncompliance via diff report
• Makes sure software remains on appropriate systems
• Creates common usage patterns for unique systems

NYSE now fields 198 active software projects and 6,600 production releases per year – numbers that show no sign of decreasing. NYSE has found an ideal tool for rapid application deployment, reducing latency by building a distributed global network of Perforce proxy servers. Like many Perforce customers before them, NYSE realized that the elegant design and swift performance of Helix Core makes it adaptable to a far broader application than just source code version management.

The world's most diverse exchange group is NYSE Euronext, a global stock market that includes the New York Stock Exchange (founded in 1792) and Euronext in Amsterdam (founded in 1602). Electronic commerce is only a recent development in the company's combined 400-year history. Rapid-fire mergers with electronic rival Archipelago Holdings and others in the last decade have kept NYSE at the forefront of technological innovation, but also stressed its global IT infrastructure. This is the story of how Perforce's role grew from version control to production management and, finally, into becoming the software development lifecycle management tool of record for the NYSE.

The Challenge: One Small Team Managing a Global Economic Engine

"It was the early days of the electronic exchange and Archipelago had several core trading engines in the production environment. As all companies realize when they try to scale from small sets of technology to large sets of technology, what works with one method doesn't necessarily work with the other," says Adam Breashears, NYSE's global head of release, change, configuration, automation, tools, and services.

At the time of the merger with Archipelago, there were 50 to 75 servers, and the method for updating configuration files and deploying software was via tarballs and RPM package file formats, as well as manual staging and editing of configuration files.

"When you go from 75 servers to 200 servers, a tarball will use 1GB. When you're putting it on five servers, no big deal. When you're putting it on 200 servers, suddenly it starts to be noticeable. Today, we manage 14,000 servers," Breashears explains.

His six-person team fields 198 active software projects and 6,600 production releases per year — numbers that show no sign of decreasing. Other challenges include:

• Rapid deployment
• Six company merges in as many years
• Ever more complex code
• Pressure to reduce development time to production
• A mushrooming server farm due to demand
• Zero tolerance for downtime
• A plethora of operating systems and devices
• A need for seamless software rollbacks that do not affect financial transactions
• Traceability of all activity to comply with the Securities and Exchange Commission.
• A patch-based shop: slow processes, unverifiable results

In 2007, the procedures for deploying and verifying software were informal at best, chaotic at worst. A single release would take several hours and a whole weekend of testing to make sure the system would run Monday morning.

"We became a very patch-based shop. Instead of releasing all of the binaries every time, people would start releasing binaries at different times and in different orders. So you had to version the binaries instead of the actual total package, which led to version inconsistency pretty much everywhere," recalls Breashears.

Further, manual changes were sometimes made by developers surreptitiously compiling in production and swapping binaries to fix production defects during the day, or others changing configurations at night. "You know, trying to do the right thing but making a typo, or fast anchoring it, or working on the wrong machine. So my boss, who is now the COO, came to me and said, 'We need to fix this. And we need a quick, reliable production mechanism.'"

One of the NYSE developers jokingly suggested adapting Helix, which was the company's source code repository, to broader deployment tasks; Breashears' team took the suggestion seriously.

Out of the box, Helix had it all: "We looked at what we needed from a deployment system. We said, okay, it versions all of the files individually. That's good. It's lightning fast. That's fantastic. Scales over multiples machines. Cool. Oh, it's got all this versioning and audit history. Well, that's really nice. Look, we can set up an audit and deploy all our configurations and do them in advance. We can stage. That's pretty cool," says Breashears. Finally, with Perforce seamless rollbacks would be possible.

Helix Performs Under Pressure: Deploying the Equities Platform Rebuild

The pilot test of Helix as a deployment tool came at a key juncture: an SEC-mandated critical software update to comply with new legislation that required stocks to trade similarly. "We had just merged with the NYSE at that time — everybody was watching," says Breashears.

The team used Helix Core to deploy a rebuild of the entire equities platform out to the servers on Friday night and then roll it back Sunday night so that the NYSE could run business as usual on Monday morning. "We went in on Friday night and it was like flicking a switch. And when you change a client spec from A to B, everything that was in the old client spec disappears and everything that's in the new client spec appears in the workspace. And then on Sunday afternoon, we flicked it back and bam! Everything looked exactly the way it did the day before," he says.

That success enabled NYSE to rebuild its equities trading engine in three months. "Usually when you try solutions like this, you get a lot of organizational resistance to change. This was one of those rare times when they were totally okay with it. They were like, 'No, you're good. You need to keep doing this.'"

The NYSE Software Lifecycle: Automating Global Release Management

But success comes with a price. "We're targeting 7,000 or 8,000 releases for the year, and we haven't increased head count at all. All the builds that go through the lifecycle hit production at the same time. The same team that's building those is the same team prepping those post-QA to go out," says Valerie Hendrickson, U.S. head of release management.

If more people can't lighten the load, automation must pick up the slack. The final step for Helix was becoming a global distribution, configuration, and audit tool, including enhancing security and aiding in SEC compliance — while remaining the NYSE's code repository and its end-to-end lifecycle tracking tool.

"We kept finding more and more wins because the paradigm for maintaining production distribution and production auditing is so parallel to what you're trying to do in development. It's just that usually the two domains don't cross, so people don't ask these similar questions," says Breashears.

Managing software throughout its usable lifecycle is key for the NYSE and a premier example of how the version management system can be used as a platform and adapted to tackle lifecycle tasks. Via Helix, the company:

• Tracks creation, quality assurance, release, staging, distribution, utilization, and deprecation
• Maintains all production client specs
• Controls Helix Core access and uses Helix triggers to detect attempts to modify configurations, scripts or binaries
• Runs nightly comparisons of files on systems with the meta-map of production in Helix, reporting any noncompliance via diff reports
• Makes sure software remains on appropriate systems
• Creates common usage patterns for unique systems

"There's just one section of the lifecycle where we don't have Helix: It doesn't do build for you," says Breashears. Even there, however, they place every artifact associated with the build in Helix for traceability.

Sensible Helix Versioning Engine Structure: Depot Design and Process Enforcement

There are four Helix depots: development, verification, quality assurance and production. Binaries in Helix move from development to production in stages, but development no longer has write access to verification application files, which forces all code to be submitted via Helix. Build branches are fully preserved based only on released files, and audit compliance is ensured. Because development verification zone files live in Helix, NYSE now can ensure that the binaries that pass verification are the same that go to QA. A full integration log is kept in Helix for each file.

For its part, QA now enjoys the ability to set up different environments with ease, and has full traceability back to source code for application files.

NYSE's use of Perforce Helix for rapid application deployment takes advantage of Perforce's distributed global proxies to reduce latency. Once again, there's that marvelous ability to track application files back to source code, but this time it's paired with an installation time that doesn't vary, whether installing one release or many. Finally, the installations are efficient and more accurate than ever before. Situations such as deploying only to select servers or releasing and tracking the results from eight different versions of software to 1,000 randomly selected machines are now possible. Preview and install logs can show if the actual install matches the planned one, as well as a full trace of exactly which version of which file was deployed, where and when.

Speedy Helix: Scalability Brought to You by Proxy

A key to the success of NYSE's lifecycle management approach has been its elasticity, which in turn comes from the Perforce Proxy (P4P). Perforce Helix easily handles distributed development, mediating between Helix clients and servers to cache frequently transmitted file revisions and intercepting requests for them. "That blows out almost all of your connectivity lag. From the New York environment to the Chicago server, all the binaries we're going to deploy are already loaded up on these proxies. That's one trip. All of the rest of your time to install is literally from that proxy to your machine, which is 50 feet away. So we haven't seen any scale issues," says Breashears.

There were some hardware-related hiccups, but these were easily solved. "We saw some outlier performance spikes because we were asking the system to do things that a developer would never ever do. We were connecting to some very high-end SAN [storage area network] in our production environment. But once we moved back to local disk, we were able to solve a lot of the I/O spikes." The production environment itself mushroomed from fewer than 100 machines to thousands for the electronic trading platform alone.

"Again, scaling over that, no problem. We changed every type of chipset and operating system on the planet. And fortunately, Perforce makes agents for all of them," says Breashears. "We recommend people always upgrade to the latest version of the GUI, but we don't really require it because the backward compatibility is so fantastic. And the agents themselves have been easy to deal with. It takes them almost no processing power in the production environment whatsoever."

Rollback and Hot Swap: Scary, But Possible in Helix 

Say an order flow engine in the NYSE trading system is on the fritz. In the past, the bug could stop a large house such as Goldman Sachs from trading that day — but no longer. Heads need not roll with the enhanced rollback capabilities of Helix. The company can return to an exact point in software history on hundreds of servers in minutes. This offers an easy intraday solution during an outage while also allowing for weekend testing.

There are no order dependencies when rolling back to any release version; full accuracy is based on Perforce Helix atomic changelists. This saves hours of work that used to be required to get a server back to a particular patch level. The only cons, which are mandated by security guidelines, are that operating system or third-party package dependencies must be done manually, and no changes, which require root access, can be effected via rollback.

A new feature Hendrickson is working on is Hot Swap, which enables a developer to click a button on the non-production side to initiate the deployment of a tested binary to fix a problem server on the production side in under 500 milliseconds, without causing the loss of any trades. "Helix is that fast and that accurate. The process itself makes me nervous. But the ability to do it is quite cool," says Breashears.

The Results: From Overwhelm to Calm at the Helm

Thanks to the efficiency and accuracy of Perforce Helix Development Platform as a lifecycle tool for the tiny team, they have gone from doing 500 releases a year to doing thousands — with an extremely high accuracy rate. Hendrickson and Breashears believe they could handle up to 10,000 releases a year with only six people while maintaining 99.9% accuracy. "I don't want to jinx myself, but I can't remember the last time we had a release management-induced outage."

Today, managing 14,000 servers is easier than managing a fraction of that number four years ago, and deployments are measured in milliseconds, not days. Further, triggers, traceability, and intelligent depot design discourage rogue coding and encourage good citizenship from everyone in the production chain.

The most interesting aspect of NYSE Euronext's Perforce installation, however, is how it achieves the Version Everything vision laid out by Perforce founder Christopher Seiwald. Like many Perforce customers before them, NYSE realized that the elegant design and swift performance of Helix VCS made it adaptable to a far broader application than just source code version management.

Hendrickson notes that she came across something that claimed you can't use Perforce  Helix for lifecycle management. "We both looked at that and we were like, 'Well, actually we do,'" laughs Breashears.

"This continual versioning of everything is the piece we can rely on. The deep, rich historical information that Helix has — it's become a fantastic tool for us."