Fidelity Information Services (FIS) is one of the world’s top-ranked technology providers to the banking industry. With more than 33,000 experts in 100 countries, FIS is part of the S&P 500. FIS has also been named the number one overall financial technology provider in the world by American Banker and Financial Insights (FinTech 100).

Business Challenge
FIS was running on one of the most complex replicated environments we had ever seen.  Microsoft’s premier support couldn’t help because the environment was running on SQL2000, which was no longer supported.  SQLHA was brought in to take over project management and lead technical activity over all internal DBA staff and other external consultants.

• Consistent outages jeopardized credit card purchases and affected ability to report on credit card revenue
• Extremely diverse and complex environment running on a non-supported version of SQL Server
• Upgrade needed to be performed within 2.5 months to meet business timeline

Engagement Summary
SQLHA coordinated several resources and led an intense 2.5 month project that saw massive concurrent upgrades and support efforts.  We teamed with Microsoft Consulting Services to validate builds for OS, SAN and SQL Servers while ensuring no end users were affected by the upgrade.

• SQL Server Upgrade\Hardware Upgrade\Change the Replication from 2 way to 3 way across the country
• 1-for-1 upgrade, added a vertical tier for environment moving data bi-directionally across the country 24×7
• Design the base builds for the OS, SAN and SQL Servers
• Simultaneously support production
• Develop plan for upgrading the 2 current bi-directional replicated environments and adding a new replication direction to make it a 3 way replication environment
• Tune the replicated environment for cross-country traffic and core transactional performance
• Project Management:  2 months to plan, design, implement and performance test the environment before Black Friday

Results
The project finished successfully ahead of deadline and the business was able to function appropriately during its busiest season.  The new environment performed at a higher level on less resources, creating a far more efficient and usable environment.

• New environment is highly available, more resilient
• Performance tuning exercises reduced response time and increased peak capacity
• Performance improvements reduced the core TPS by 500% while exceeding the previous year’s load by 100%.
• Replicated environment eliminated high latency or transaction times that were experienced previously
• SQL Servers replicating bi-directionally across the country, receiving financial transactions for credit card and pre-paid card applications
• 100% of the upgrades were done 2 weeks before Black Friday

JPMorgan Chase (NYSE: JPM) is one of the oldest financial institutions in the United States. With a history dating back over 200 years.  JPMC is a leading global financial services firm with assets of $2.2 trillion, operating in more than 60 countries with more than 240,000 employees.  JPMorgan Chase serves millions of consumers, small businesses and many of the world’s most prominent corporate, institutional and government clients.

Business Challenge
JPMorgan Chase chose SQLHA for two high-profile projects intended to reduce licensing, hardware and physical footprints while increasing system performance and agility.  The first project entailed designing a SQL-as-a-Service architecture to support over 5,000 instances.  The second was a migration of over 5,000 branch locations from DB2 to SQL Server.

• Reduce physical infrastructure footprint while improving performance, scalability and reliability
• High risk- new system would change the experience and approach for the typical application that is hosted on SQL Server
• Potentially manage 5-6000 SQL servers factoring in all lifecycle management factors including security, scalability, HA, DR, monitoring, auditing and daily maintenance
• Produce brand new instances within a 24 hour turnaround time
• Develop process that would allow migration from DB2 to SQL at all branch locations
• Develop a utility that would be able to migration the financial data from DB2 to SQL Server and do this for up to 1,500 branch locations across the country in a 2 hour period
• No downtime allowed during migration- if any/all branches were down, customers would be unable to perform any banking operations at the local branches

Engagement Summary
Working with JPMC’s DB engineering team to design a SQLaaS environment, SQLHA identified the core configurations for the physical environment and designed the operational and management offerings for the environment.  This took into account security, auditing, compliance, backup strategy, HA, DR, monitoring, performance tuning and provisioning of the server itself.  Lastly, we developed a groundbreaking migration utility that migrated 1500/branches a night, all in a 2-hour period, running 600 concurrent conversions across the WAN.

• Identified the core configurations for the physical environment, designed test cases to perform against 3 vendors hardware platforms to determine the optimal configuration and hardware solution for the future state environment
• Conducted comparison by running TPCC, TPCH and batch tests across 16 VMs that were hosted on the different hardware platforms while capturing and analyzing all of the test results
• Developed a utility that would be able to migration the financial data from DB2 to SQL Server and do this for up to 1,500 branch locations across the country in a 2 hour period
• Each migration had to verify and validate the data was successfully migrated 100% since this was critical financial data
• Process had to be designed to take into account the network limitations, the limited functionality in SQL Server Express, ability to manage the processes for start, stop, rollback and failover scenarios, massively parallel and perform the migration in under 2 hours
• Process was designed using SQL Server Integration Services and allowed a single branch to be fully pre-checked, migrated and validated in an average of 10 minutes

Results
JPMC was able to meet business requirements around footprint reduction and efficiency while completing a successful migration.

• Reduce license, hardware cost
• Reduced physical server footprint by 1200%
• New virtualized base image configured to produce brand new instances within a 24 hour turnaround time
• Migration completed within timelines without end user pain

Achieving high availability is a combination of people, process, and technology. Never lead with a technological solution or technology. A solution must start with proper requirements that meet the actual need of the business.

Performance tuning can be performed on many levels including the hardware, software, schema, code base.  Focus on the hardware and software configuration first as these need to be optimal and known before reviewing the schema and code base.  I always say, “Clean the room up first so we can better see what we are dealing with.”

The Allstate Corporation is the nation’s largest publicly held personal lines insurer. A Fortune 100 company, with $131 billion in total assets, Allstate sells 13 major lines of insurance, including auto, property, life and commercial.

Business Challenge
Allstate wanted cost flexibility around licensing and physical hardware.  They also wanted to evaluate overall power use and consumption for IT infrastructure as part of a companywide environmental awareness effort.  A critical aspect of this challenge was adding Disaster Recovery sites while achieving the directive of a smaller, more efficient footprint.

• Over 1000 servers and several thousand databases and hundreds of TB of storage
• Large project that affected over 15,000 internal applications
• High risk because of the complexity of the applications using the SQL Server environment
• Update architecture to support a consolidated platform

Engagement Summary
Allstate charged SQLHA with technical project leadership, responsible for designing the future state architecture, managing the internal SQL Server team and  designing an approach and process to upgrade and consolidate the databases at the same time.

• Upgraded all of the SQL Servers to SQL 2005 from several other versions and updated the physical architecture to support a consolidated SQL Server environment
• Designed a utility to scan the network and retrieve metadata about each SQL Server host
• Developed a migration utility that migrated the database and all the related objects from one server, enabling scheduling and reducing the time to configure for each database

Results
Allstate was able to upgrade their system, save money, increase efficiency and reduce their overall IT footprint, meeting requirements for an environmentally-aware infrastructure.

• SQLHA reduced overall server footprint by 65%
• Standardized hardware, assisted in launching green datacenter
• Physical patching/maintenance process enabled better service out to lines of business
• Increased resiliency of applications with new Disaster Recovery plan
• Designed future-state  architecture, working with business on potential future service offerings and their impact on infrastructure and performance
• Enabled technology initiatives to be updated to meet best practices and support massive global business demands

Unless you were living under a rock, you couldn’t miss Microsoft’s announcements today about the next version of Windows from the //build conference. It was announced you would be able to download the developer preview here as of tonight. However, that is just the client from what I can tell. What you may not know is that you can also download the Server version – and that is what I’m more interested in. It’s up on MSDN if you have a subscription. I didn’t see it on the TechNet subscription downloads (at least not yet).

Of course, there will be failover clustering enhancements revealed this week. Once they are public, I may address some of them as they relate to SQL. However, if you plan on attending my PASS pre-conference “A Deep Dive into AlwaysOn: Failover Clustering and Availability Groups” on October 11 (and if you haven’t registered, why not?), I should have some Windows 8-based demos (fingers crossed). I just finished downloading the Windows Server Developer Preview … here we go!

It’s hard not to be a bit reflective today – 9/11. The images are just as disturbing today as they were 10 years ago. I remember exactly where I was. I had just barely opened my eyes when I saw the second plane hit. The next 100 or so minutes are seared in my memory forever. My parents grew up in New York (the Bronx to be exact), and my maternal grandfather worked in the World Trade Center for the Port Authority before he retired. So I have  a spiritual connection to New York. It’s a city I do love to visit (but those hotel room costs at certain times of the year … oy!). I remember writing the initial chapters to my SQL Server 2000 High Availability book and originally I had a 9/11 reference in it. To a person, everyone reviewing it said to take it out because the reference was too soon. And I did.

I also remember driving down to NJ about a week after 9/11 for Rosh Hashanah (the Jewish New Year). It was at night (probably 10 or 11 o’clock). When I passed the commuter parking lot at the top of the Garden State Parkway attached to the Montvale Rest Stop, I saw a lot of cars there. That was unusual. And I realized – a lot of those cars probably belonged to some of the victims of the World Trade Center. It was eerie, and a feeling I never want to have again. I can’t describe it. It’s funny what sticks to you years later.

Over the past 10 years we’ve seen a lot of natural destruction around the world – the tsunami from the Indian Ocean (which is sneaking up on its 10th as well), the recent earthquakes and tsunami in Japan this year, Hurricane Irene just a few weeks ago which has caused massive problems here in the Northeast, and Hurricane Katrina. Those are just examples – I know there are many more. Could your business stand a natural disaster that hits your area? Have you even taken it into count in your overall operations plans? If the answer is no to either, you may be in trouble. It’s not just natural events that will get you. The recent power outage in Southern California, Mexico, and Arizona is a great example of a man-made problem which can have a very adverse effect on your business if you don’t plan for those kinds of things.

At the end of the day, when the dust has settled, life usually tries to return to (relative) normal. Businesses reopen. Workers go back to work. Money is made. People get a paycheck. In this connected world, it’s not as simple as “opening the doors”. Everyone’s got e-mail systems, database systems, application servers, SANs, networking, communications … you get the idea. I talk about this with customers, but the reality is that everyone needs a disaster recovery (DR), or more accurately, a business continuity (BC) plan. BC is more accurate since there are more elements (such as human ones) that come into play.

I’m not saying you should ignore the technical side – far from it. Plan to your heart’s content. Do things like get your backups offsite. Test your DR plans. Make sure you know how to coordinate resources. Have a staff schedule. You get the idea. The trust is that all of those tactical – and necessary – tasks are very important stuff. It’s mainly what we focus on as technical folks. However …

What brought this more into focus for me recently was the death of my friend Mike which I’ve blogged about here. Besides the loved ones he left behind, he also left behind a law practice and a computer consulting business with active clients that still needed to be serviced. He may not have been Accenture or IBM Global Services, but his business still has to run until the family figures out how to transition the work properly and legally to others. From my perspective, it really got me thinking. It’s one thing to prepare for certain things, and I’ve talked about people being single points of failure in the past, but would happen if something happened to me? What would become of my business? How would those I entrusted in my will to my estate deal with things? Would they know where to find my contracts? Do they know my passwords? How would they see my receivables? Well, you get the picture. Those are very different concerns than just worrying about getting an application running – but they are every bit as valid. So whether you are a 1 person company or a 100,000 person company, these are the types of things that come with BC.

I’ve started to re-examine my own contingency plans. Have you? If not, get going!

10 Years – What A Difference

When all of the things happened around 9/11 or other events like the tsunami in the Indian Ocean, we saw the many pictures posted all around those areas of people looking for loved ones with contact information on the flyers or posters. Facebook and Twitter were not around. I believe listservs were used by the Red Cross to try to track the missing. I do remember rudimentary web pages. Today, there would be Facebook pages and real time tweets. My, how far we’ve come. I’ll be the first to admit I begrudgingly embrace social media in some aspects, but when it comes to things like this? It’s very, very good.

On Another Note

9/11 has obviously changed the way we all go about our lives whether directly or indirectly. I fly quite often, and it has been interesting to see the changes in airport security all over the world. I can remember flying a few months after 9/11 and that experience was WAY different than what we have today. It was certainly a sight to see armed military walking around. In those days immediately post-9/11, some of you may remember not being able to take ANYTHING with a battery out during flight. No laptops, no audio players, no phones – nothing. Laptops in bags had to go in the overhead. You were stuck with your travel companion (if you were with someone) and things like books/magazines/puzzles. At that point you could still take your own water (this is pre-Richard Reid). As much as I do like to work on planes sometimes, I’ve got to say, outside of boarding planes in Europe, it was the fastest boarding process I’ve ever experienced. I kinda miss it. In the USA, nowadays no matter what airline, it’s a bit chaotic with people cramming things in overheads to avoid checked baggage fees and thinking they are so self-important that they can ignore flight attendants when they are asked to power down their phone or smartphone. Hint: you’re not special. That e-mail or conversation can wait. You’re holding us all up.

I’ve also noticed a slight change in some folks (like my parents) when I go somewhere. I’ve criss-crossed the globe usually alone. My Mom doesn’t fly, and I know she gets anxious when I fly. I definitely appreciate the concern (nice to know people care!), but the reality is that if something happens, it’s going to happen. I can’t live my life in fear of “what if”. I certainly know my friend Mike didn’t. Sadly, I probably have a better chance of something happening to me driving or walking around a major city (not that I want anything to happen) than something happening on a plane. The numbers back me up on that. I can only control my actions – not anyone else’s.

Life is certainly fragile, but don’t put yourself in a cocoon. Live it to the best that you can. You’ll regret it otherwise.

I just wanted to thank everyone for their kind words and thoughts over the past week since my friend passed away. I really do appreciate it more than you know. I know I’m a little backed up in terms of correspondence and work, so I do appreciate your patience as  get fully back up to speed this week with all of the work on my plate.

As you may know, I was supposed to do a series of six webinars last week (SQL Foundations) for Penton over two days. That has been moved forward a few weeks to September 13 & 15. So if you haven’t signed up yet, now you’ve got a few more weeks. I am very thankful Penton agreed to move the dates as I know events like this are not easy to schedule. They’re a great company to work with.

Think of the hardest thing you’ve probably had to do in your daily life. Now multiply it by infinity. That is a bit how I felt yesterday at my friend’s funeral. It still hasn’t fully hit me that he is no longer with us. I’m numb today. I’ve tried to do some work to have some “normal”, but it’s like trying to comprehend rocket science – not happening at the moment. I’m not sure his passing will hit me fully any time soon.

The only funerals I have attended to date have been those of my four grandparents. While each one was devastating in their own way, you could see them coming (and I don’t mean that flippantly). My maternal grandfather, for example, had Parkinsons (which is a horrible way to die, by the way if you didn’t know). He worked for the Port Authority before he retired in the early 70s and designed some of the lighting for a few of the NYC bridges. He was so smart and mechanically oriented. He loved to fix jewellery, clocks, and watches. In many ways, I picked up a lot of his traits. But that’s a whole separate discussion. The point is, as horrible as the circumstances around his death around this extremely intelligent man who at the end, couldn’t speak or experess himself (and you could see his frustration) due to his horrible illness, even though it still is a blow when the inevitable happens, you can prepare (as much as you can) for it. With Mike, someone who was in shape, happy, successful, healthy, and seemingly had a lot of time left, it was a shock to say the least. To say it is a tragedy is an understatement.

It’s no secret I do a lot of public speaking either at events or online. Heaven knows I crank out a lot of content in any given year. But I can tell you nothing is harder (or more of an honor) than having to speak at someone’s funeral. How do you properly eulogize someone? It’s not like writing a technical document or a blog post, that’s for sure. When Mike passed away (and I still have trouble even typing those words), in the back of my mind I started formulating a few things in the event they wanted me to speak at the funeral. I was asked, and on Tuesday night after getting down to my parents, I typed up something. I had my parents read it to ensure it was okay. My Mom pointed out I had used a few things in present tense. Yeah, it is just hard to let it all sink in. While I didn’t use the script exactly, it provided the basis for what I did ultimately say. My goal was to pay a fitting tribute to my friend Mike. One that would show my experience with him – no one else’s – but that they could relate to because I’m sure they had similar experiences. And for those who may not have known him as well, I wanted to really show why he was special. I wanted to celebrate his life.

After doing a few edits, I went to bed and got up in the AM and tweaked it ever so slightly before heading north to Paramus, NJ. I didn’t rehearse it. When it came time to speak (I was 4th, after his mother and two brothers), I composed myself, and was surprisingly OK until near the end where I nearly lost it. That only lasted a few seconds but it felt like an eternity. As a guy, I think we have this “be tough” mentality that is somewhat imposed on us by society. I think it’s rather silly. Tom Hanks (in the movie A League of Their Own) said: “There’s no crying in baseball!”. True, but try to go through a funeral and not be touched. I double dare you. If you’re not, you’re soulless.

While it was nice to have some people come up to me later and tell me they thought I did a great job or liked what I said, that isn’t why I did it. I wasn’t looking for kudos. This isn’t like a technical conference where you hope you’re at or near the top of the heap in evaluations. Everyone who spoke at the funeral did a great job and brought their own unique perspective on Mike’s life to the table. There is no winner here.

I was also asked to be a pallbearer – again, a big honor. Up to that point, everything was a bit surreal. At that point, it was very real. All too real. I just can’t express how much I still can’t believe he’s gone even a few days later. I know some folks over the years have thought of me as brash, outspoken, brutally honest, insensitive, and so on. While I’m smarter these days about how I express myself, I think some people still think I’m a bit of an arrogant jerk (that’s the PG version, folks). I know this, and I doubt I’ll ever convince them otherwise. I can’t worry about that – nor do I anymore. But I’m definitely not an unfeeling ogre who doesn’t consider others. There is nothing more humbling or human than having to eulogize someone you cared about. You think 400 level technical content is hard? Ha! What would have been the point if I gave some emotionless eulogy which focused on facts only? He was my friend, not some random stranger.

In Judaism, we don’t do an open casket or things like a viewing. There are no ornate caskets or embalming to preserve the body. Jewish burial laws and customs are designed to speed the process along. If possible, someone should be buried within 24 hours or as soon as possible (for example, close relative has to travel a long way, so you delay things; that happened with one of my grandparents – the funeral was delayed a few days because of me).

This, however, was not a standard Jewish funeral. Because Mike was an EMT, his fellow sisters and brothers (fire, police, EMT) showed up in force from all over NJ, and possibly elsewhere. It was an amazing and touching sight. From the procession of emergency vehicles which led the cars to the cemetary, to the blocking off of streets (some major roads, like Route 4 in Paramus), and probably the most impressive/touching: the two Paramus fire ladder trucks set up in front of the cemetary gate with the ladders up and an American flag in the center. EMT, fire, and police lined the driveway as the mourners drove in. It really was impressive, and I’m a cynical bastard. The traditions of Judiasm were observed, as well as the traditions of his brothers and sisters who put their lives on the line every day. Mike truly was a hero in life and death, and he got a hero’s funeral.

When I atteneded the funerals of my grandparents, I was still in my 20s. I was an adult, but not in the same way I am now. I’ve got another 10+ years on the proverbial tires. In every possible way this was a different experience. I don’t know how to fully express in words how, but it just was. After yesterday, I’ve determined that the two worst sounds one could ever hear are:

1. The sound of dirt hitting a coffin. Don’t need to really elaborate on this one I hope. Horrible, horrible.
2. The playing of the song “Taps”. I’ve never attended a military funeral, but I have to assume this was as close as I’ll ever see. After the Jewish part of the ceremony was over, “Taps” was played. You know, you see it on TV in news snippets at funerals, but it is a whole different experience in person. There’s something so sad, so final about it.

Only being a little bit older than Mike by a few months, and turning 40 in November, something tragic like this really puts a lot (and I do mean a lot) of things in perspective. A few weeks ago I wrote a blog post entitled “Recharged Batteries = Full Steam Ahead”. That blog post is even truer today than when I wrote it: life is too short. Worrying about work 24×7 won’t get you anywhere. You do have to stop and smell the roses because it can all be taken away in an instant. Enjoy your time on this planet and be the best person you can be.

Mike lived his life to the fullest. He was one of the most honest, loving, trustworthy, selfless, joyful, giving, intelligent people I have ever met. He was curious about life and its experiences. You couldn’t ask for a better friend. I’m thankful for the time I had with him since 1990 and I know I’m a better person for it. I can only hope to honor my friend’s way-too-soon passing in the way I choose to live and conduct the rest of my life.