I don’t know what’s in the water in IT departments, and it’s definitely not the fact that today is Halloween. No tricks or treats in this blog post! I’ve had a lot of conversations with companies over the past year that are trying to be mission critical. By mission critical, I mean near 24×7 uptime (give or take). While I think that’s great and definitely an area I’m well suited to help them with from a consulting perspective, what I don’t like to hear is one of the following:
We don’t have any SQL Server DBAs
Our <insert worker already doing other things> is our part time DBA
I’m finding many companies are simply lacking a wholly owned, full time SQL Server DBA on their staff. I’m perplexed. Those two things – not having a DBA of any sort and being mission critical – are at odds. Let’s dispel some myths and deal with some realities.
SQL Server Manages Itself
I used to work at Microsoft and am lucky to still have a close relationship with them. Years ago Microsoft had a five nines campaign that launched around the same time we were doing a week long SQL Server 2000 high availability event. They were basically trying to show that Windows was ready for the enterprise and five nines (99.999% uptime – 5.24 minutes of downtime per year). The ad I remember most had the picture of a data center with a lone, empty rolling chair and balloons floating around insinuating everyone is out having fun and things managed themselves basically. They couldn’t have picked a worse time to send such, in my opinion, a wrong message. By the way, here is proof of that campaign. I can’t dig up the ad anywhere. If you have it or know where it is, please link below or e-mail it to me! Anyway, anyone who has done any production work knows that message in that campaign is wrong. Ever get paged at 2AM? ’nuff said.
SQL Server marketing has done an excellent job over the years of making it seem like SQL Server manages itself. To a point that is true with some of the stuff internally in the engine such as statistics (although there are cases where you may have to deal with those manually), etc. All IT shops that use SQL Server have the same basic care and feeding needs, but the scale is different in a small shop vs. a large enterprise. If you think you can effectively manage hundreds or thousands of databases and along with their accompanying instances (and all of the associated hardware) with no DBA or at least having someone on staff who knows SQL Server, I have a bridge to sell you in Brooklyn.
Sure, you can automate some tasks like monitoring and some tasks to take care of known issues, but you still need human beings to interpret what that information is saying. Some things are not obvious. A big part of monitoring is noticing trends – it’s not just for break/fix scenarios. A good DBA automates what he or she can, and deals with the rest. A good example is getting a report in the AM telling you what backups ran successfully or failed. You don’t want to connect to every instance to get that information. It’s time consuming. Spend your work hours dealing with the problems (like making sure the backups are done if failed) instead of figuring out of they ran or not.
Let’s talk about maintenance plans for a moment. I know they are popular and certainly have use. Heck, it’s easy to click some checkboxes, click Next a few times, and bam! Instant maintenance. In my estimation, 9 out of 10 times, it’s always better to code that stuff yourself and implement via SQL Server Agent jobs. That way it’s not a black box. Plus, you can easily alter things to meet your needs. For example, do you really need to rebuild ALL of your indexes at whatever interval you specify? Probably not. You most likely need to monitor what’s going on, anticipate the need based on history, and then have a job that is either manual or scheduled to fend off a problem based on that information. A maintenance plan isn’t going to work there.
Bottom line here is that over time, a DBA should be able transform a shop from being reactive to proactive. HUGE benefit and win for all. That’s why you need someone who knows what they’re doing. They make it look easy even when it’s not.
A Multi-Tasker is Better than Dedicated Staff
I think we’ve all been in roles at some point in our career where we’ve worn multiple hats. Knowing different disciplines can help any role, but you should have a primary focus. For example, a DBA should be able to understand Windows, storage, and networking, but they don’t need to be gurus. However, if you are the lone administrator responsible for everything from Active Directory and Exchange to SharePoint, SQL Server, and beyond, do you really think you’ll be efficient and a deep expert at any one of those – let alone all of them? I’m not saying it’s not possible … just improbable.
The argument, especially in very small shops, is that one person should be able to handle it all. I don’t necessarily disagree. In a small IT shop where there are not a lot of systems, the load is manageable, and nothing is very complex, a jack-of-all-trades may be perfect if they’ve got some experience under their belt. However, if you have complex deployments and numbers, that argument fades fast. For example, do you think your jack of all trades can handle multi-subnet clusters or SAN replication between sites? That’s some heady networking and storage infrastructure that needs proper planning and administration going forward. The wrong person doing the wrong thing could give you lots of downtime and problems.
Look at all of the dimensions of what a person brings to the table, not just the cost in terms of dollars.
Oh, and your developer – not such a good choice for a production SQL Server DBA for numerous reasons. I’ll leave that one alone in this blog post. Could make good fodder for another one.
Yes, Virginia – SQL Server is Mission Critical in Your Environment
That phrase is a spin on “Yes, Virginia – there is Santa Claus”. I’m never surprised anymore that some environments do not consider SQL Server essential. They’ll talk about Oracle or DB2. Yet when I go in and do some work – especially an assessment or consolidation – lo and behold we find out that there tons more SQL Server databases and instances (and the applications that use them) than they thought existed. They only knew about a subset since they were not under any kind of central administration (a different issue I’m not going to touch here). SQL Server may not be powering your Peoplesoft implementation (maybe it is; substitute your favorite application if you’d like), but it’s everywhere else under the sun and used by nearly everyone in one capacity or another. So I would say based on sheer numbers of DBs and instances alone (not even marrying that info to the apps which may be used 24×7), SQL Server is mission critical in your environment whether you like it or not. Welcome to 2011. SQL Server is ready for enterprise class stuff.
As such, it needs to be treated in the same way you do the other RDBMS choices you have. I bet you have Oracle DBAs – why not SQL Server?
The Economics
I also understand the math behind all of this. Hiring folks is not free, be it a FTE, a consultant, or an outsourced DBA service. How much is your data worth to you to ensure you have peace of mind? It’s arguably more important than your applications since your data is a big part of your IP. If you’ve got tight service level agreements (SLAs), recovery time objectives (RTOs), and recovery point objectives (RPOs), having that jack-of-all-trades-master-of-none may work in some cases, but what happens when it doesn’t? The cost to benefit ratio of having someone skilled is not a straightforward calculation that can be done by the bean counters behind a desk .
What I also see is that for whatever reason, SQL Server folks are valued lower than, say, their Oracle counterparts in enterprise IT shops. An enterprise DBA is an enterprise DBA, period. Even in terms of contract work, I sometimes chuckle at what gets sent my way. People want the sun, the moon, and the stars all in one person for very little money. I always tell whoever sends these requests that if they find that person for that rate, hang onto them.
The reality is that someone with a BI background may know very little about relational (outside of data warehouse stuff) and a relational person may know nothing about Analysis Services, Reporting Services, et al. Each one of those things is really a different role that just happens to fall under the banner of SQL Server because that is the branding. An operational DBA may do some of those other tasks, but their primary function is the care and and feeding of your SQL Server environment to ensure it remains up, healthy, and performing. Anything else in my mind is a bonus. SQL Server is a big product – understand that when you set out to find someone for your needs. You may need two or three people, not just one.
EDIT: After I posted, I saw this tweet on the #dba hashtag:
Oops!
Think this means less productivity? That’s a cost … and in this case, having the right person either may have prevented this OR would get them back up and running. Factor that in – you may cost others a lot of time and money with the wrong person, too.
Where Does This Leave Us?
I know this post is going to be a little controversial for some, and you may disagree with me (would love to see you post if so), but in the near 20 years I’ve been involved with SQL Server, I’ve never seen where having someone who really knows what they are doing is a detriment. It’s usually a benefit. 2011 is not different than 1990, 1995, 2000 or any other year – you need to know your stuff as best you can to do a good job. That person coming right out of college with no real world experience may be cheap in terms of cost, but would you trust your 24×7 system to them and no one else? Probably not. Ideally you would have a mix of senior and junior folks because both have value. The senior resources can help and mentor the junior ones.
At the end of the day, realize that if you want to be (or think you are) mission critical, you cannot ignore the simple fact that not having at least one qualified, dedicated SQL Server DBA is a surefire method that will most likely prevent you from ever achieving your goal.
As first revealed on this past Friday’s (October 21) DBAs@Midnight web show, it’s official: Allan will be writing a book for SQL Server 2012 covering mission critical to follow in the footsteps of SQL Server 2000 High Availability, Pro SQL Server 2005 High Availability, and Pro SQL Server 2008 Failover Clustering. The SQL Server 2012 book will be a return to more of what Pro SQL Server 2005 High Availability was: covering the gamut from A to Z, and not just the technology aspects. Even in terms of technology there’s more to mission critical than things like clustering and availability groups (both to be covered here): virtualization and other necessary topics will be explored.
There will be a few key differences this time around, namely:
The book will be self-published. How it will be sold is still to be determined.
Right now, the plans are to make it an eBook only and available in the most popular formats – namely Kindle and ePub (PDF). This will allow the most compatibility. This also essentially means it should be available in popular web stores as well, but again, how it will be sold is not finalized yet.
The goal is to have it out as soon as possible after SQL Server 2012 ships. Because of small changes that could happen between now and RTM (especially to UI elements), some work cannot be done until the product itself is completely finalized.
The book will have some others contributing and writing content in addition to Allan. Not only is it a lot of work to write a book with such a big scope, it helps the book get to you quicker and enhances the value of the content by having key people contributing. Names (many who you know and love) will be announced soon.
Those of you who want a physical book may be slightly disappointed which begs the question: why no physical book? It has nothing to do with self-publishing vs. using a traditional publisher.
The biggest reason is the proposed size of the book. To do the topic right, it’s not going to be 200 – 300 pages to cover all the technology along with everything else that would ensure your SQL Server 2012 deployments are mission critical. Even most traditional publishing houses these days are hesitant to do a book that is planned to be pushing 1000 (or more) pages.
To go along with #1, the cost of printing such a book is not cheap. That means expensive to Allan, expensive to you, and depending on where you live, potentially expensive to ship due to weight/size since there would be no local printing solution. Doing an eBook keeps the costs to you reasonable for such a comprehensive book. That’s the tradeoff.
An eBook allows you to print only what you need/want to have on paper. For example, want the install instructions for installing a clustered SQL Server instance and only that? You can do that.
If you want a physical book, there is the slim chance there could be a limited run if there is enough interest. If you are interested in a print book, use the “Contact Us” link at the top of the page and make your voice heard. If there’s enough response, costs will be investigated once a more definitive page count is known. Should such a print run happen, it would most likely be a on preorder basis only to ensure all costs were covered. While a few extra books may be printed, once the run is gone, it’s gone.
More details (such as if there will be early access for preorders) will be forthcoming as they are finalized. Stay tuned!
If you’ve ever seen one of my webcasts, attended a session somewhere, or looked at my books, you will tend to see a lot of references to music. Some folks have picked up on it over the years and figured it out. Since people have basically asked in a roundabout (not trying to make a Yes reference for the musically inclined among us) way, I’ll just answer the basic question: why?
It’s no secret I’m an active musician (mainly jazz stuff these days) and I’ve recorded a few albums. I’ve been playing bass (and jazz) for 30+ years now and if I ever get a couple of weeks at home and some time, I’ll finish my new jazz project (small and regular sized big band one) which is mostly in the can and needs to be mixed and mastered. It even has participation from two members of The Yellowjackets. It’s an album I’ve wanted to record since starting in big bands back in the 80s. All but one of the arrangements are mine, and some of the tunes are originals. Oh, and the “Are you related to Al Hirt?” question – heard it a million times since I picked up bass in 3rd grade. The answer is no. And no, I don’t play trumpet, either. I did play trombone starting in 5th grade because I didn’t want to learn a new clef and I knew couldn’t use my bass in marching band when I reached high school. I never practiced the thing and still wound up first chair. I don’t play trombone anymore, but still remember all the positions on the slide (7th position to do a B natural is a pain for a guy who is 5’6″ – trust me) and it’s helped with arranging since ‘bone is a hard instrument to arrange for.
Bringing up one of my scores - about to record a piano part on "Cygnus X-1" for the new jazz project (release date TBD)
I also did a bit of music journalism in college and a bit after, with some interviews for magazines like Keyboard and Bass Player. I interviewed a lot of folks (including some of my favorites – one Mr. DeYoung and one Mr. Lee included). I ran Bob Mintzer’s website for nearly 10 years. I’ve been lucky to be able to get to know some of them over the years outside of interviews. Trust me when I say they are really just like you and me.
But I digress.
Putting music references in things like names of servers, clusters, and SQL Server instances keeps it fun for something that could be mundane – such as a cluster node name. Although I have a very diverse taste in music, most of my references tend to be related to either Styx or Rush. I think some who know me well may be surprised I never use some sort of Disney reference (especially Tron or Mickey Mouse related), but I don’t think I have in all these years.
Don’t get me wrong – there are lots of bands and musicians I would consider among my favorites. Rush and Styx, however, are a bit different for me (although I’d probably lump Joe Jackson, Howard Jones, Bob Mintzer, and a few more at or near the top of the list depending on the day). Let me say up front that Styx and Rush definitely conjure up different images for different people. Most of it usually isn’t good. With Styx, folks think the kitsch of Roboto and for Rush, kimonos, math rock, and Geddy Lee’s voice.
Why Styx and Rush more often than not?
With Styx, people who really know me are aware that I’ve been a Styx fan for as long as (or longer) than I’ve been playing bass. Go ahead, snicker all you want. I cut my teeth learning to play along with Paradise Theater from the songbook. You may think I know a lot about clustering – but when it comes to Styx, I’m just as deep (so-to-speak). So it should come as no surprise that in some circles, I’m considered somewhat of an expert on Styx and its memorabilia (although I do not really collect much anymore; I’ve kept a few pieces over the years and sold others). Some of my stuff can be seen in the (in)famous Behind the Music on the band. Although I update it infrequently, I still maintain StyxCollector.com which has interviews I’ve done with band members over the years and information on the Roboto mask I own (see the SC.com website for info; it’s one of two that lights up – Dennis DeYoung has the other). I’ve been threatening to post an online discography, but this SQL thing keeps me kinda busy. Any of you SQL-type folks want to help me make a searchable one with a SQL Server backend?
Allan's Roboto Mask
I was also briefly in the VH-1 True Spin segment for “Mr. Roboto” (check just around the 1:03 mark for my few seconds of fame). That segment was taped in a Residence Inn in Rochester, NY, that I was in for the day job (I believe I was doing a gig at Xerox). It’s really the only time my Roboto mask has left where I live. The funny thing I remember about that recording was they asked me a ton of questions and I was sick as a dog. Fun times.
The Rush stuff should be a bit more obvious (I hope) since I play bass. You may hate Geddy Lee’s voice (it’s definitely a love it or hate it proposition), but his (and his fellow bandmates’) musicianship is unquestionable. I didn’t get into Rush right away after starting to play bass. I liked them mainly via MTV. I can still play along with albums like Exit Stage Left and Hold Your Fire with my eyes closed. I was even in a Rush tribute band for awhile a few years back (see pic below). I didn’t sing (thankfully!), but I did play bass and keyboards (including triggering stuff with my feet with a Roland PK-5A while playing bass and/or regular keyboards at the same time – talk about coordination). My keyboard rig went through a few iterations before the band split up. The one pictured is the last, and simplest, one.
Playing bass and keys live (foot controller not pictured)
I usually use Rush references when I have 3-server configurations (excluding domain controller) and Styx for everything else. Band members tend to be the node or server names (hence in a 3-server configuration – ALEX, GEDDY, and NEIL; in a 2-node config, DENNIS and TOMMY), albums the SQL Server instance names (such as EQUINOX or MOVING\PICTURES), the band name the domain (RUSH or STYX), etc. It makes sense to me and I’m not hopping from demo environment to demo environment trying to figure out a naming convention. I know what I’m looking at no matter which band this configuration happens to be.
There is a more serious undertone to this somewhat whimsical post: naming conventions matter. Whether you use baseball stadiums, rock bands, musicians, or something more practical like location plus some other relevant information, your servers, clusters, and instances should be consistent and make sense in your environment (such as BOSSQLPRODCLU01, where BOS = Boston, SQL = SQL Server, PROD = Production, CLU = Clustered, 01 = the first one in this type of configuration). What you want is something easy to remember that is consistently followed everywhere in your IT environment. It makes administration so much easier, and it also means you have standards and you can flag things that do not adhere to them. When you look at an object, you should instantly be able to identify what it is. Having some cryptic, million letter string may sound great to your security guys, but it makes no sense in the real world.
Devising a naming convention, no matter how cute, funny, or entertaining it appears to be, is a serious matter. Treat it as such. You’ll thank me for it later. Domo arigato!
If you haven’t noticed, SQLHA has undergone a facelift. It’s something that has been a long time coming. Don’t get me wrong – I liked the old site and made it myself. It served me well for over four years. However, sometimes you can reach the point where you need to call in a professional who really does this for a living to do it right. Much like people call me for consulting, it was time to belly up to the proverbial bar and have this site redone properly. My days of enthusiast hacking around in Dreamweaver and putzing around with various blog platforms are over.
The new SQLHA was soft launched last week during PASS. Of course, I managed to screw things up towards the end of PASS and put the site in a weird state. That has all been fixed and all of the kinks have been pretty much worked out (and we’re sorting out the final few), including the fact that all of my old blog posts should now resolve to the new structure automatically.
The new site design has a much cleaner aesthetic I always wanted but could never achieve on my own. I love how the homepage switches between two different designs. We’ve got different SQL Server tips on the bottom of many pages. You can click on the top level (such as “Why SQLHA?”) as well as the sub-menu items. There’s quite a bit of content, and more will be added. I’ve updated things like all of the Resources links. This isn’t just a simple cosmetic upgrade.
And the biggest change: I’ve got a new partner. When I started/founded SQLHA as my branding in 2007, I had no delusions of grandeur. I was just getting started and had no designs on growing much beyond just me. I was looking to survive and take it day by day, month by month, and hopefully year by year. When you go independent, you have no idea how it’s going to go. I was hoping I’d make it. Just having started year five, I think I can say I’m doing OK, and I’m very thankful to each and every one of you who has attended a class, used me for consulting, etc. One thing I’ve realized is that I’m only one person, and there’s only so much I can do. Somewhere along the way after I started on the marathon and ditched the sprint, I really thought about partnering with someone. That person is Ben DeBow (Twitter).
I’ve known Ben for years. We’ve worked on engagements together, presented at PASS together, etc. We have some intersection in terms of what we like to do in the SQL Server world which allows us to work together well, but we’re also very different people. For example, where I generally don’t do much performance tuning of things like queries (not that I can’t, I’m just so busy doing the other stuff), Ben does. I didn’t need another “me” exactly; I needed someone who complimented my skills where we could tag team on an engagement if necessary and be doubly efficient, or pursue other/different work so SQLHA can provide a wider range of services that alone I just did not have the bandwidth to do. Ben may not currently be one of the household names in the SQL Server world, but he’s worked on some of the largest and most complex deployments out there. He’s a rock star – you just don’t know it yet.
Please join me in welcoming Ben to the SQLHA fold. He’ll also be blogging here, and I know I look forward to what he has to say.
Create a feedback loop with the developers. Unless they have insight into the runtime or growth characteristics of their code, they cannot proactively address the issues they do not know about.
Effective management of the SQL Server environment is only possible by having data captured and available. At a minimum, capture SQL inventory and configuration, performance, security, audit and maintenance data.
Always design the database environment top down even if the environment consists of only a couple servers or several hundred. By approaching the design top down, many efficiencies can be gained on every level.
Independent of how much automation and self tuning a database package has, it needs to be reviewed bi-annually. A SQL health check is an important task to achieve optimal performance and stability.
Choosing between physical and virtual implementations of SQL Server does not have to be an all-or-nothing approach to your deployments. Both strategies are equally valid and can co-exist peacefully.
Testing is one of the cornerstones for ensuring success in production, including both performance and availability. DBAs and IT are responsible for testing things like updates in non-production environments prior to rollout as well as ensuring disaster recovery plans work. What you do not know can and will work against you.
