23/10/2009

By Gerry DeMarco, a managing director at Morgan Stanley

Gerry DeMarco, a managing director at Morgan Stanley who is currently on a three-year secondment in London running the global datacentre projects team, overseeing the bank's facilities upgrades, discusses the best way to migrate your applications and commission new or upgraded datacentres

In the wake of the large number of mergers caused by the credit crunch over the last couple of years, a growing area of interest for IT professionals in the financial services industry is how to achieve a smooth application migration (or AppMig as I like to call it). Other drivers include real estate opportunities, footprint consolidations, or a desire to change the tiering of your datacentre to create more resilience, but what is common in every 'moving story' is that people want to know how to ensure a smooth transition.

I'd like to share some of the things that we've learnt recently at Morgan Stanley, as we've undertaken multiple concurrent application migration projects across London, New York, Hong Kong and Tokyo, involving the migration of 7,000 servers and associated infrastructure, with zero disruption to the business. I hope it's of use to you in any moves that you're planning.

First and foremost though, let's define what AppMig is. My definition of it is that it covers the events that occur after a datacentre facility is built and the necessary infrastructure (networks, file services, name services, pre-built racks, etc) are up and running. Basically, you have a datacentre that is ready to accept application workloads, or extra room in an existing datacentre to migrate applications.

My teams at Morgan Stanley, who've moved 7,000 servers don't forget, were bound by a few 'rules of engagement' but I'd expect to see similar restrictions at any firm. Ours were: our process had to have minimal impact on the developers; we were told the speed of the moves and not vice versa; and existing infrastructure staff could not absorb the AppMig workload. This led to Golden Rule #1, go out-of-band whenever and wherever possible. We hired our own Level 3 staff for Windows, Linux, storage and networking, and augmented our Level 2 with outsourcing assistance. The problem with hiring new staff is ramp up time. That led to Golden Rule #2, embed as many staff members with the incumbent infrastructure team(s) as possible. This generates goodwill and helps knowledge transfer. In addition, we embedded an application analyst in the application support and development groups, for the same reasoning. This was a crucial step in our success since most application nuances are kept in people's heads.

The Process - Step 1: Inventory
Each application and the hardware it is running on needs to be validated and inventoried. I cannot stress this point enough, you need to be 100 per cent accurate that a server you want to move is the correct one. It usually requires forensic and manual intervention, but the accuracy gained is well worth the time and money spent. This is Golden Rule #3, know what you are moving! All hardware needs to be audited and certified from a physical and logical perspective. This will prevent the situation where you work on the wrong hardware and cause an outage.

Step 2: The schedule
Now that the applications are loosely categorised in groups it is time to put together a schedule. There isn't much science here, start slowly with development servers to shake out your process and gradually ramp up based upon your risk appetite. One tip, we created a schedule that had 'tagged' or 'named' weekends rather than dates or series.

Step 3: Customer engagement
A few months prior to the scheduled move of the application we introduced application roundtables. These meetings, hosted by the analysts, aimed to engage developers on the project, communicate the proposed move weekend, and start to discuss the source and target environments. Once these architecture and dependencies are agreed upon we were then ready to place that application in a tagged weekend.

Step 4: Ten week cycle
Ten weeks before the go-live date the application team enters a standard cycle: Weeks 10 through 9 - Scheduled date and target environment are confirmed; Weeks 8 through 5 - Hardware installed, booted and burned in, servers configured to application specs; Weeks 4 through 2 - the environment is turned over to the application development team for testing and sign off. Week 1 - Final go/no go, migration will occur over the weekend.

A few final suggestions I'd recommend are: (a) Remove references to specific hostnames or IP addresses as migrating an application that has references to the old hostname or IP will cause instability; (b) Move to generic service names, thereby abstracting the physical from the logical naming; (c) Undertake any major upgrades three months prior to the migration weekend; (d) Use pseudo-environments. This is a segmented area in the source datacentre that used the new host and IP naming conventions. At a later date, these servers were installed in the main source datacentre at much greater speed than would have been possible otherwise.

By sticking to these processes, abiding by the golden rules oulined above, and constantly communicating well with all the relevant stakeholders your AppMig project should be successful and error free.