A customer of mine contacted me this week about a problem they were having. They have a two-node failover cluster on Windows Server 2008 R2 with an instance of SQL Server 2008 with SQL Server 2008 SP1. They installed SQL Server's program files to another drive - not the main system drive (C). They then configured log shipping. Everything worked fine until they were testing failover. When the instance was failed over to the node which was not the first one installed, log shipping stopped working. When they failed it back, everything worked. After a WebEx session, it looks like Setup didn't put everything in the right place on the other node. Before I rushed to any conclusions, I needed to reproduce the problem to see if it was possibly something that went wrong in their setup. Here's what I did.

1. I created a two node W2K8 R2 cluster with a slipstreamed SQL Server 2008 SP1 instance.

Here is where I installed the progams to:

and on the next dialog


So far, so good.

2. I configured log shipping to an instance on another server while the instance was on the original (first) node it was installed on. Everything worked great.

3. I failed the instance over to the other node, and log shipping failed.

4. I failed the instance back to the original node. Lo and behold, log shipping worked again.

So what happened?

If you look at the job step for the transaction log backup job, here's what it is calling:
"Z:\Program Files\Microsoft SQL Server\100\Tools\Binn\sqllogship.exe" -Backup 6B81BF42-4AA8-4DE3-8349-5E54EE0C52ED -server KILROY

Here is what the programs look like on the first (original) install node for Drive Z. Note the shared tools directory with SqlLogShip.exe.

Here is what the programs look like on the second node (add node) install:
C drive showing the shared tools directory with SqlLogShip.exe.

Z drive showing no shared tools directory.


So it's pretty clear to me that the Add Node operation is not putting the files in the same place even though the other node has the same drive structure, thus causing log shipping to stop working in a failover.

If you install things all to the original system drive (such as C), everything works fine so that is a workaround. But I know some of you like to put program files in places other than the system drive.

I have written this up over on the Connect site here, so if you want to try to get this fixed, vote for it!

The funny thing about this is that on the initial install I selected everything to go to the Z drive, but it still install some files to C. Interesting.

So I'm finally finishing up the scripts and such for the book. Yes, they are very, very late. But you guys will benefit from that because you'll get a few things which wouldn't have been in there had I released 'em when Pro SQL Server 2008 Failover Clustering was released.

Anyway, I was creating the final versions of the checklists and configuration worksheets using Office 2010. I started them in Office 2007,  but have since installed 2010. I'm working with a client this week and I sent them the latest version of the configuration worksheet. The worksheet contains a lot of dropdown boxes to make it easier in some cases to select a value. The customer said they couldn't see the dropdown. I was baffled. Both Office 2007 and Office 2010 have the same format (.xlsx). I wasn't aware of any changes. Since I didn't have Office 2007 installed, I couldn't figure it out. I was baffled.

Today, I installed Office 2007 in one of the virtual machines on my laptop. Lo and behold, the customer was 100% correct. So what's the difference?

In Office 2007, to create a dropdown with a list of values on another sheet, here's what you do:
1. On the sheet where you want (other than the one you're working on), create the values.
2. Select the values.
3. In the Name box (next to the formula [fx] under the ribbon) enter a name for those values.
4. On the sheet where you want to use those as a dropdown, select a cel.
5. Select the Data ribbon.
6. Select Data Validation, then Data Validation.
7. On the Settings tab, select List in the Allow dropdown.
8. For Source, enter the equals sign and then the name you just created. An example would be =MyList.

In Office 2010, you no longer have to define a name for the list before using it. You are allowed to directly select the data in Step 8, meaning that steps 2 and 3 no longer need to be done.

While this is actually more intuitive, when you save it does not pop up a warning message (like it does for other instances) where this would cause a problem for previous versions of Excel. Now, I didn't select to save this as an .xls (the 97 - 2003 format). I did .xlsx which is the same format as Office 2007. So while the file format is the same, something is clearly different. The file does not error out in Office 2007 at all; it just shows the last selected value in the cel, no dropdown. Annoying.

The moral here: if you are creating documents in Excel 2010 that use dropdowns, and need them to be backwards compatible, do things the 2007 way.

Sometimes living on the bleeding edge isn't the best thing. Although you'll get both 2007 and 2010 versions of my files when I put them up for download soon, so maybe you guys all benefit from my pain ...

I just found out one of my proposed sessions was approved, so I'll be speaking at TechEd North America 2010. This year it's back in New Orleans from June 7 - 10. I was at I think the last TechEd in around 2003 or so in New Orleans (well, the last one in New Orleans *I* spoke at). Clearly it's too early to know what day or time, but I'm very happy and honored nonetheless. Stay tuned for more info.

Oh, and if you're curious, it's a clustering-related session - not a surprise.

There has been a lot of discussion lately among MVPs around “Should/can I disable IPv6 on Windows Server 2008 (RTM or R2) for servers that will act as failover clustering nodes?” Obviously this affects any products such as SQL Server or Exchange that will be then clustered on top of a Windows Server 2008 failover cluster.

In the sake of full disclosure, I have been recommending to many customers who were going to cluster over the past year or so to disable IPv6 if they were not using it. There was no definitive statement one way or the other and in all of my use and testing, I’ve seen no issues with doing so. While I didn't say to disable IPv6 in my book Pro SQL Server 2008 Failover Clustering, since I have been telling people to consider it, I had to "fess up".

Due to the MVP discussions, a PM from the Windows clustering team chimed in and they recommend that unless there is a strong reason to disable IPv6, you should not do so. Why? The following three points come directly from the PM:

• In all of our current troubleshooting and investigation of [Windows Server] 2008 and [Windows Server] 2008 R2 networking issues which may have been caused by IPv6, it has turned out that disabling it has not directly solved any problems.  These issues were resolved by disabling related networking components, such as Teredo or TCP Offload.

• By default, all clusters will communicate between nodes over IPv6.  If that is not available on network interfaces, then the cluster will try to communicate using 6-to-4 tunnels.  If IPv6 is completely unavailable in this environment, the nodes will then communicate by IPv4.  So yes, it is possible to disable either IPv4 or IPv6 and still have the cluster function correctly.

• We recommend keeping IPv6 on as this is the default configuration, which means that it is the most thoroughly tested and stable configuration.  IPv4, by itself, certainly will work, but we don’t recommend disabling IPv6 unless there is a good reason to do so.

So there you have the definitive “word” on the topic. As of the writing of this blog post, there is no official SQL Server statement on the “should I disable IPv6” topic (and if there is one at some point, I’ll update this post), but  since a SQL Server failover cluster is built on top of Windows Server 2008, I strongly suggest you follow the recommendation above for IPv6: don’t disable it. I know this will be my recommendation going forward in further talks/presentations, training, and consulting I do.

Well, it's official - the website just went up. Can't hide it anymore ... I'm delivering a 3-day masterclass on consolidation and virtualization at World of Windows Server in Singapore as well as delivering two sessions (although the current website shows two different, it is going to be a two parter on clustering). I'm very exicted and honored to have been asked to do this.

If you don't live in Singapore or can't get there (hey, what's a few thousand US dollars to take a little excursion to Singapore at the last minute - just ask ask your boss!), I do plan on delivering the masterclass and possibly expanding it to four or five days. Or not. I've spent a lot of time over this past month in content development (official v1 of my Windows Server 2008/SQL Server failover clustering class and this upcoming delivery in Singapore), and am going to assess it after I'm done to see if any tweaks are needed. I can promise you that whether you come to Singapore or see it in a town near you, it'l be a lot of fun, technical content, and information.

Contact me if you want to know more!

So it's no secret I've been using Windows Server 2008 R2 for quite some time. I like the PowerShell commandlets, and I've occasionally used the failover clustering module for PowerShell in R2 to manage "downlevel" Windows Server 2008 RTM (and SP2) based clusters with no issue.

This week I also figured, "What the heck?" and installed PowerShell 2.0 on a Windows Server 2008 SP2 cluster, and took the module from R2. It seemed to work just fine. You obviously don't get any new features, so things like Live Migration won't work. Only the basic failover clustering PowerShell commands would technically be applicable.

Despite both of these scenarios seemingly working fine in my limited use of them, they are 100% unsupported by Microsoft. So I will say this: I'm glad I tried it, but I would never recommend you doing something that would put your supportability in jeopardy - especially on a production system. If you do use these in either of the scenarios, do it at your own risk.

PASS was a whirlwind week between the three main conference days and the two MS Insider days on either end. I don't ever remember being so busy at PASS, yet for the first time in ... well ... I don't know how long I actually attended a handful of sessions. I highly recommend the HA customer panel led by two good friends on SQLCAT Prem Mehra and Sanjay Mishra. Besides hearing about real world implementations, I was SO glad the panel talked about people and process. You don't get 12 seconds of downtime with technology alone! It wouldn't be PASS if I didn't say that it was good seeing everyone - friends new and old, including putting faces with people I had been talking to for quite awhile either over e-mail or on the phone. Oh, and Buck - stop trying to make friends with fire hydrants!

Despite all of the excitement I also managed to deliver my own, not without its own set of challenges. By the time I hit PASS, I realized that my session had been made 500 level. 500 level? I may be good, but 500 I think implies I must know a lot about SQL Server and failover clustering lol Now wanting to disappoint, my presentation and demos had major surgery before Thursday, and I hope I passed the audition (to paraphrase a famous rock band ...). I'll see when I get my scores in a few weeks. I only wish I had a two parter or more time, as it would have even been better. It was also fun sitting in for the afternoon on the failover clustering pre-con.

I did see that people posted Tweets about me on Twitter http://twitter.com/search?q=Hirt%20sqlpass. I'm not on Twitter and don't plan on it (I am on Facebook, though). Someone pointed it out to me. Seeing it was a strange experience.

In other news, I saw that Ken Simmons took the time to read and blog with a brief review of my new consolidation whitepaper. It's always nice to see people actually thinking what you do is worthwhile; sometimes it's hard to see the forest through the trees when you're writing and editing.

On Amazon, my Pro SQL Server 2005 High Availability book just got its first review after two years in publication. I was pretty shocked to see it, considering how long it's been out. I know what people have said to me about it, but funny to see a formal review after my new book was published.

Speaking of my new book, I thank those of you who have already bought Pro SQL Server 2008 Failover Clustering  in eBook or a physical format (i.e. a book of the paper kind) since its publication in July. The word on the street is it's doing fairly well, and people seem to like it. I saw at PASS it sold out, which I took to mean it was popular.

Oh, and I know I need to finish the scripts. I will soon - and one of them is a PowerShell-based script which automatically creates the scripts to install your Windows and SQL Server failover clusters pulling the info from my configuration spreadsheet. I know the scripts are way late, but I've been a tad busy and I want to give you guys good value. My apologies.

I've got a few future book ideas up my sleeve - a few of which may come to fruition over the next year or so - but I can say with relative certainty that I'll probably be doing a full HA book (not just failover clustering) for the next full version of SQL Server after SQL Server 2008 R2. Chances are it will be bigger than the 2005 HA book which is about 800 pages. Yes, you read that right. Bigger. Better start lifting weights now!

Well, in the mean time, I'm off to finish preparing for a failover clustering training class I'm delivering next week, and finalize my consolidation/virtualization class I'll be delivering in Singapore in December, along with two normal presentations (one on failover clustering, another on consolidation/virtualization).

Thankfully I'll have a little bit of downtime in Japan (before Singapore) and Hong Kong (after Singapore). I get to spend my upcoming 38th birthday on a plane flying to Tokyo. I look forward to catching the Tokyo Kosei Wind Orchestra and hopefully Stanley Clarke, Chick Corea, and Lenny White at the Blue Note while in town. I found my USA-made 1976 maple neck fretless Fender Precision there in 2008 for a great price, and it's become a favorite to play. Tokyo not only a great place to visit just as a tourist (this will be my third time there), but an awesome music town. If you're a lover of buying music in a physical format, one of the last places on Earth that still has a lot of great CD and record stores. Hopefully I'll also get to hang a bit with my friend Jason who I was in a band with back in college. My new jazz album is partially done, and I'll be doing a lot of composing and arranging while on that trip. Heaven knows I'll have a LOT of time on airplanes!

I have been at a client all week implementing a SQL Server 2005 Standard Edition SP3 cluster using Windows Server 2008 R2 for the underlying operating system. Since this is W2K8 R2, it's 64-bit only, so we implemented SQL Server 2005 x64. Everything went pretty smoothly, as did the cutover from their current production box. However, we did notice one weird issue: we couldn't configure a Maintenance Plan (I'm not a huge fan - I'm more of a create it on your own kind of guy). As it turns out, I found the following blog post from a Microsoft support engineer. We had named the nodes ending in lowecase letters. The problem was that by the time we figured this out, we were already live in production, and there is no way we could take another outage, and run through the whole process of evicting and renaming nodes.

The blog only mentions Maintenance Plans and Database Mail. What it doesn't mention is that it also does not create the System/Performance Monitor counters for use, and you have replication issues if you are using a Publisher and Distributor on the same instance (although replication works, you get errors and it is just odd). If you configure replication to go against a remote Distributor, everything is just fine. We did a bunch of testing against another cluster we configured (same hardware configuration) for their soon-to-be reporting server and had none of the problems we had on the original cluster we configured as production.

The unfortunate thing is that the customer is now in production and they can't afford another outage to go and evict things and reconfigure the servers in use. Everything else is working and stable. They may consider migrating to the other cluster in another outage, but not anytime soon.

Moral of this sad tale of woe: never use lowercase letters for Windows Server 2008 (RTM or R2) node names with a SQL Server 2005 clustered implementation. It affects more than just Database Mail and Maintenance Plans.

If this isn't a reason to use SQL Server 2008, I don't know what is! :)

Just a heads up for those of you in the Pacific Northwest - I'll be presenting at the SQL Saturday on October 3 which is being held in Redmond on the Microsoft campus. I'll be doing the last talk of the day in the DBA track. Not surprisingly, I'm talking on failover clustering.

Hope to see some of you there!

I just saw that my session "Advanced Failover Clustering Installation Techniques with SQL Server 2008 and Windows Server 2008" will be on Thursday, November 5 from 1 - 2:15 in 3A-B.

I'm just starting to pull the session together and will have as much info as I can cram in 75 minutes with some demos, too :) A heads up: I will be talking about Windows (and some differences between RTM and R2) quite a bit in this one.

I'll also be giving out a copy of my book as well ...

Hope to see some of you there!