A customer of mine contacted me this week about a problem they were having. They have a two-node failover cluster on Windows Server 2008 R2 with an instance of SQL Server 2008 with SQL Server 2008 SP1. They installed SQL Server's program files to another drive - not the main system drive (C). They then configured log shipping. Everything worked fine until they were testing failover. When the instance was failed over to the node which was not the first one installed, log shipping stopped working. When they failed it back, everything worked. After a WebEx session, it looks like Setup didn't put everything in the right place on the other node. Before I rushed to any conclusions, I needed to reproduce the problem to see if it was possibly something that went wrong in their setup. Here's what I did.

1. I created a two node W2K8 R2 cluster with a slipstreamed SQL Server 2008 SP1 instance.

Here is where I installed the progams to:

and on the next dialog


So far, so good.

2. I configured log shipping to an instance on another server while the instance was on the original (first) node it was installed on. Everything worked great.

3. I failed the instance over to the other node, and log shipping failed.

4. I failed the instance back to the original node. Lo and behold, log shipping worked again.

So what happened?

If you look at the job step for the transaction log backup job, here's what it is calling:
"Z:\Program Files\Microsoft SQL Server\100\Tools\Binn\sqllogship.exe" -Backup 6B81BF42-4AA8-4DE3-8349-5E54EE0C52ED -server KILROY

Here is what the programs look like on the first (original) install node for Drive Z. Note the shared tools directory with SqlLogShip.exe.

Here is what the programs look like on the second node (add node) install:
C drive showing the shared tools directory with SqlLogShip.exe.

Z drive showing no shared tools directory.


So it's pretty clear to me that the Add Node operation is not putting the files in the same place even though the other node has the same drive structure, thus causing log shipping to stop working in a failover.

If you install things all to the original system drive (such as C), everything works fine so that is a workaround. But I know some of you like to put program files in places other than the system drive.

I have written this up over on the Connect site here, so if you want to try to get this fixed, vote for it!

The funny thing about this is that on the initial install I selected everything to go to the Z drive, but it still install some files to C. Interesting.

I just found out one of my proposed sessions was approved, so I'll be speaking at TechEd North America 2010. This year it's back in New Orleans from June 7 - 10. I was at I think the last TechEd in around 2003 or so in New Orleans (well, the last one in New Orleans *I* spoke at). Clearly it's too early to know what day or time, but I'm very happy and honored nonetheless. Stay tuned for more info.

Oh, and if you're curious, it's a clustering-related session - not a surprise.

Well, it's official - the website just went up. Can't hide it anymore ... I'm delivering a 3-day masterclass on consolidation and virtualization at World of Windows Server in Singapore as well as delivering two sessions (although the current website shows two different, it is going to be a two parter on clustering). I'm very exicted and honored to have been asked to do this.

If you don't live in Singapore or can't get there (hey, what's a few thousand US dollars to take a little excursion to Singapore at the last minute - just ask ask your boss!), I do plan on delivering the masterclass and possibly expanding it to four or five days. Or not. I've spent a lot of time over this past month in content development (official v1 of my Windows Server 2008/SQL Server failover clustering class and this upcoming delivery in Singapore), and am going to assess it after I'm done to see if any tweaks are needed. I can promise you that whether you come to Singapore or see it in a town near you, it'l be a lot of fun, technical content, and information.

Contact me if you want to know more!

PASS was a whirlwind week between the three main conference days and the two MS Insider days on either end. I don't ever remember being so busy at PASS, yet for the first time in ... well ... I don't know how long I actually attended a handful of sessions. I highly recommend the HA customer panel led by two good friends on SQLCAT Prem Mehra and Sanjay Mishra. Besides hearing about real world implementations, I was SO glad the panel talked about people and process. You don't get 12 seconds of downtime with technology alone! It wouldn't be PASS if I didn't say that it was good seeing everyone - friends new and old, including putting faces with people I had been talking to for quite awhile either over e-mail or on the phone. Oh, and Buck - stop trying to make friends with fire hydrants!

Despite all of the excitement I also managed to deliver my own, not without its own set of challenges. By the time I hit PASS, I realized that my session had been made 500 level. 500 level? I may be good, but 500 I think implies I must know a lot about SQL Server and failover clustering lol Now wanting to disappoint, my presentation and demos had major surgery before Thursday, and I hope I passed the audition (to paraphrase a famous rock band ...). I'll see when I get my scores in a few weeks. I only wish I had a two parter or more time, as it would have even been better. It was also fun sitting in for the afternoon on the failover clustering pre-con.

I did see that people posted Tweets about me on Twitter http://twitter.com/search?q=Hirt%20sqlpass. I'm not on Twitter and don't plan on it (I am on Facebook, though). Someone pointed it out to me. Seeing it was a strange experience.

In other news, I saw that Ken Simmons took the time to read and blog with a brief review of my new consolidation whitepaper. It's always nice to see people actually thinking what you do is worthwhile; sometimes it's hard to see the forest through the trees when you're writing and editing.

On Amazon, my Pro SQL Server 2005 High Availability book just got its first review after two years in publication. I was pretty shocked to see it, considering how long it's been out. I know what people have said to me about it, but funny to see a formal review after my new book was published.

Speaking of my new book, I thank those of you who have already bought Pro SQL Server 2008 Failover Clustering  in eBook or a physical format (i.e. a book of the paper kind) since its publication in July. The word on the street is it's doing fairly well, and people seem to like it. I saw at PASS it sold out, which I took to mean it was popular.

Oh, and I know I need to finish the scripts. I will soon - and one of them is a PowerShell-based script which automatically creates the scripts to install your Windows and SQL Server failover clusters pulling the info from my configuration spreadsheet. I know the scripts are way late, but I've been a tad busy and I want to give you guys good value. My apologies.

I've got a few future book ideas up my sleeve - a few of which may come to fruition over the next year or so - but I can say with relative certainty that I'll probably be doing a full HA book (not just failover clustering) for the next full version of SQL Server after SQL Server 2008 R2. Chances are it will be bigger than the 2005 HA book which is about 800 pages. Yes, you read that right. Bigger. Better start lifting weights now!

Well, in the mean time, I'm off to finish preparing for a failover clustering training class I'm delivering next week, and finalize my consolidation/virtualization class I'll be delivering in Singapore in December, along with two normal presentations (one on failover clustering, another on consolidation/virtualization).

Thankfully I'll have a little bit of downtime in Japan (before Singapore) and Hong Kong (after Singapore). I get to spend my upcoming 38th birthday on a plane flying to Tokyo. I look forward to catching the Tokyo Kosei Wind Orchestra and hopefully Stanley Clarke, Chick Corea, and Lenny White at the Blue Note while in town. I found my USA-made 1976 maple neck fretless Fender Precision there in 2008 for a great price, and it's become a favorite to play. Tokyo not only a great place to visit just as a tourist (this will be my third time there), but an awesome music town. If you're a lover of buying music in a physical format, one of the last places on Earth that still has a lot of great CD and record stores. Hopefully I'll also get to hang a bit with my friend Jason who I was in a band with back in college. My new jazz album is partially done, and I'll be doing a lot of composing and arranging while on that trip. Heaven knows I'll have a LOT of time on airplanes!

I've been talking about it for awhile, but it's finally here - my update of the old SQL Server 2000 consolidation whitepaper. It just went live less than an hour ago.

The basic info including the link to the download of the Word document can be found here:
http://msdn.microsoft.com/en-us/library/ee692366.aspx

or you can just download the Word doc from http://download.microsoft.com/download/D/B/D/DBDE7972-1EB9-470A-BA18-58849DB3EB3B/SQLServer2008Consolidation.docx.

I hope you find it useful!

I just saw that my session "Advanced Failover Clustering Installation Techniques with SQL Server 2008 and Windows Server 2008" will be on Thursday, November 5 from 1 - 2:15 in 3A-B.

I'm just starting to pull the session together and will have as much info as I can cram in 75 minutes with some demos, too :) A heads up: I will be talking about Windows (and some differences between RTM and R2) quite a bit in this one.

I'll also be giving out a copy of my book as well ...

Hope to see some of you there!

Now that Windows Server 2008 R2 is RTM and the book is done, I can get to some of the things that were impossible to do prior to getting Pro SQL Server 2008 Failover Clustering out the door.

One of the things I was not able to do was document how to create the Microsoft Distributed Transaction Coordinator using PowerShell. It is not hard, but it also isn't the most straightforward task, either. Use the script below to create DTC; all you need to do is change the parameters (self-explanatory) to the right values for your environment.

$clunm = "Windows failover cluster name"
$dtcdisknm = "Name of the disk resource to use with DTC"
$dtcdnsnm = "Name for DTC in DNS"
$dtcgrpnm = "Name of the DTC resource group"
$dtcipaddr = "IP address for DTC"
$dtcsubnet = "Subnet mask for DTC"
$dtcipresnm = "Name of the DTC IP address resource"
$dtcnetnm = "Name of the DTC network name resource"
$dtcresnm = "Name of the DTC resource name"

Add-ClusterGroup $dtcgrpnm -Cluster $clunm
Add-ClusterResource $dtcipresnm -ResourceType "IP Address" -Cluster $clunm -Group $dtcgrpnm
$ipres = Get-ClusterResource $dtcipresnm
$ipaddr = New-Object Microsoft.FailoverClusters.PowerShell.ClusterParameter $ipres,Address,$dtcipaddr
$subnet = New-Object Microsoft.FailoverClusters.PowerShell.ClusterParameter $ipres,Address,$dtcsubnet
$setparams = $ipaddr,$subnet
$setparams | Set-ClusterParameter
Add-ClusterResource $dtcnetnm -ResourceType "Network name" -Cluster $clunm -Group $dtcgrpnm
$nnres = Get-ClusterResource $dtcnetnm
$netnm = New-Object Microsoft.FailoverClusters.PowerShell.ClusterParameter $nnres,Address,$dtcdnsnm
$netnm | Set-ClusterParameter
Add-ClusterResourceDependency $dtcnetnm $dtcipresnm -Cluster $clunm
Add-ClusterResource $dtcresnm -ResourceType "Distributed Transaction Coordinator" -Cluster $clunm -Group $dtcgrpnm
Add-ClusterResourceDependency $dtcresnm $dtcnetnm -Cluster $clunm
Move-ClusterResource $dtcdisknm -Cluster $clunm -Group $dtcgrpnm
Add-ClusterResourceDependency $dtcresnm $dtcdisknm -Cluster $clunm

In my next blog post, I will show you how to map MSDTC to a specific instance of SQL Server since even the current SQL Server 2008 failover clustering whitepaper gets it slightly wrong. Screenshots will be included.

I do a bit of work in the consolidation space, and speak about it a lot. The update to my old SQL Server 2000 consolidation planning whitepaper is almost done and I should be doing a webcast next month. But I wanted to address one thing here in the interim: the biggest sticking point is not moving databases or maintenance plans; it's applications. There are lots of aspects to this which I am not addressing in this blog post. I am specifically going to address application compatibility with later versions of SQL Server such as 2005 or 2008. A common complaint I hear from many customers is that the applications they use do not support SQL Server <insert version here>.

There are a few issues with this really large, and to some degree, crippling problem:
1. Part of the problem is not the vendor, but the customer.
a. Many vendors do have later versions of their application; customers just haven't implemented it. I'm oversimplifying here; it's a deeper problem than that which is explored in the next few sub-letters.
b. Change is hard. Later versions introduce new, or changed, functionality that may not even be used. If something is changed, it may not be to a customer's liking. As far as unused functionality, there comes a point for a lot of people where what is there is "good enough" and there may be no reason in terms of features to seemingly want to upgrade.
c. If the application is highly customizable (such as Siebel), it's not a slam dunk just to implement it. There's downtime as well as retraining of end users.
d. This one is really a side effect of a, b, and c: even if it means not being supported, it sometimes may feel like it is just easier to go with what you know.
e. ... and then there is cost. Upgrades generally are not free.

You can work through downtime and any technical issues associated with an upgrade; the more intangible (perceived value) and tangible (do we have money to do this?) are a bigger problem.

2. In the cases where there is no later version of the software, it's unfortunate, but Microsoft cannot play the heavy and force vendors to make an application support a later version of SQL Server. Believe me, I wish they could, but I can only imagine the potential legalities of that. It's really up to the customers to make their voice heard to get a version of an application that supports what they need. If they can't, realistically, it may be time to move to another software package that supports the platforms you deploy. You could possibly consider virtualizing the server and its SQL Server instance, but you're still supporting the old version of SQL and you need to be sure the vendor supports you in a virtualized environment - but you do get rid of the physical box.

3. All of this is a Catch-22. Your application pain now forces you to deploy or still keep older versions of SQL Server in play, affecting your long term supportability not only from Microsoft but in terms of your DBAs. Do you want your DBAs to be a long way behind in their knowlegde of the platform? Heck no!

By the way, this  problem extends to service pack levels. In my experience, some software won't allow you to upgrade/patch your SQL Server instances to a later service pack unless they support it. That doesn't bode well for consolidating that application where you have a shared instance with N number of databases, all of which will be upgraded at the same time.

Trust me, I wish I had a magic wand to make these issues go away. They are very real and must be dealt with appropriately in a consolidation effort. What is right for one customer may not be for another. I do enjoy helping customers work through these issues, but I can assure you that it is never easy and hard decisions need to be made.

Below is my complete schedule for TechEd. I hope to see you at the panel and my session. Feel free to come by also when I'm working the DAT booth.

Monday, May 11
12 - 1PM     Speaker Idol Judge
1 - 2 PM      Panel Discussion - Planning a Windows Deployment for SQL Server 2008 Failover Clustering (with Max Verun from the SQL Server Develeopment Team and Steven Ekren from the Windows Development Clustering Team) - Room 501C
2:30 - 6PM  Answering questions in the Microsoft SQL Server booth

Tuesday, May 12
9:30 - 11:45AM  Answering questions in the Microsoft SQL Server booth
12 - 1PM     Speaker Idol Judge

Wednesday, May 13
12 - 1PM     Speaker Idol Judge
3 - 6PM  Answering questions in the Microsoft SQL Server booth

Thursday, May 14
12 - 1PM     Speaker Idol Judge
1:30 - 4:45PM  Answering questions in the Microsoft SQL Server booth
Evening - Speaker Idol Finals

I'll also probably pick up the bass at some point during the jam session. Had two gigs this past weekend and it'll be odd to go without playing for a week or so. Anyone else remember when they had them nearly every night in the older years of TechEd?

Friday, May 15
10AM - 12PM  Answering questions in the Microsoft SQL Server booth
1 - 2:15PM   DAT322 Tips and Tricks for Successful Database Mirroring Deployments with Microsoft SQL Server - Room 502A

Enjoy! Click here to listen to it.