Day 22- Lessons in Migrations

This article was written by Saint Aardvark the Carpeted

I've been through two big moves in my career. The first was about four years ago when the company I was working for moved offices. It was only across the street, but it meant shifting the whole company over. We had about forty employees at the time, and maybe a hundred workstations, test servers, and production servers.

The second move was when, earlier this year at my current job, we finally got to move into our new server room. This time the scope of the move was smaller (no workstations, and about twenty servers), but the new digs were nicer. :-)

I learned a lot from these two moves. I want to pass those lessons on to you.

Have a second set of skilled hands around

At both places, I was the only sysadmin on staff. For the first move, my company hired a consultant for a few days to help me out with the move and its aftershocks. It was great to have someone else around that could help diagnose email problems, run traceroute and generally run interference while I swore at the servers.

The second time, I thought that four volunteers, plus me, would be enough... it was only twenty servers, after all. Mainly, it would be a question of cabling, then things would just fall into place after that... right?

Well, the volunteers were excellent -- I can't say enough about them, but a second set of skilled hands would have simplified things a lot. I found myself often switching between them as questions came up: How do these rack rails work? Which interface is eth0? Did you really mean to put 8U of servers into 4U of space?

Obviously, someone familiar with your network, OS/distro and thought patterns can help you with network testing, re-jigging Apache proxy directives, and finding your pizza coupons. Even something as simple as being familiar with rack rails helps a lot.

And if you're moving offices, don't do this without the support of your company. For the first move there were three of us -- including the CEO -- and I wouldn't want to do it with less bodies or less influence.

Don't underestimate how tired you'll be

In some ways, the first move was easier despite it being much more involved. We moved on a Saturday, I got machines up and running on Sunday, and on Monday, things were mostly working again. Knowing that I had the time meant that I could go home with a clear conscience.

The second move, though, was meant to be done in one day. It was gonna be simple: I had a checklist for services and boot order, the network settings were ready to go, and the new server room was quite close our old server room. How long could it take to move stuff two blocks?

Well, the moving took the morning. De-racking machines, getting stuff on the elevator and to the truck (thank goodness for strong movers), then dropping stuff off in the server room left us in a good position for lunch.

But after lunch, little things cropped up: I'd borked some netmask settings on a couple key servers. The rack I'd planned to put the firewall in was too shallow to accept it. My placement of the in-rack switches blocked some PDU outlets. Some of the rack rails were fragile, stupidly constructed, and difficult to figure out.

Each of these things were overcome, but it took time. Before I knew it, it was 7:15pm, I'd been at it for 11 hours and I was exhausted. I had to head home and finish it the next day. Fortunately, I had the support of my boss in this.

Don't make the day any worse than it has to be

At the first move, I'd decided it would be a good idea to switch to a new phone vendor as we moved into the new building.

I avoided firing by, I later calculated, the skin of my teeth.

Your move will be long. It will be stressful. You will trip over things you didn't plan for, thought you'd planned for, and were sure someone else was planning for. Don't add to the misery by making another big change at the same time. This goes double for anything involving a complicated technology with multiple vendors (including a local monopoly that Does Not Like competition) that will leave everyone very upset if it fails to work right when they come in.

Instead, mark it carefully on your calendar for five years in the future.

Set up monitoring early

For the second move, my Nagios box was second on my list of machines to boot up. I'd set it up with new addresses ahead of time, and made sure when it did start that alerts were turned off.

As machines came up, I watched the host and service checks turn green. It was a good way to ensure that I hadn't forgotten anything...if it failed, I'd either forgotten to update the address or I had a genuine problem. Either way, I knew about it quickly, and could decide whether to tackle it right away or leave it for later.

Don't forget about cabling

I planned out a lot of things for my second move, and it served me well. Service checklists, boot sequences...it had taken a long time, but it was worth it. I even had a colour-coded spreadsheet showing how many rack units, watts and network cables I'd need for each server.

Unfortunately, what I missed was thinking about the cabling itself. I'd picked out where the switch in each rack would go, I'd made sure I had lots of cables of varying lengths around, and so on. But there were some things I'd missed that experience -- or a dry run -- would have caught:

• Horizontal cable management bars blocked a couple of PDU outlets each; this was mostly, but not entirely, unavoidable.
• PDU outlets were on the wrong side for most -- but not all -- servers, which put power cables right next to network cables.
• The switches were right next to some PDU outlets -- and since the switch outlets went all the way to the side, that meant some network cables were right next to power cables.

A dry run of the cabling would not have been easy. I didn't have a spare server to rack and check for problems, and some of these things only emerged when you had a full rack. But it would have been a lot less work than doing it all on the day of the move (let alone swearing at it and leaving it for Christmas maintenance).

Getting new equipment? Make sure it works

As part of the new server room, we got a few bells and whistles. Among them were a humidifier (necessary since we didn't have a vapour barrier) and leak detectors that sat on the floor, waiting to yell at me about floods. "Woohoo!" I thought. "We're movin' on up!"

What I didn't think about was how these things worked...or rather, how I could tell that they worked. We moved in during summer, so the humidifier wasn't really necessary. But when winter came around and the humidity dropped to 15%, I realized that I had no idea how to tell if the thing was working. And when I dug up the manual, I had no idea what it was talking about.

Same with the leak detection. I knew it was there, since the sub-contractor had pointed it out. I had assumed it was managed by the monitoring box that had been installed along with it...and since I was busy right then moving in boxes and getting NFS working, I put it on the list of stuff to do later.

When I finally did tackle it later, it turned out I was wrong: it wasn't part of the other monitoring box. The box I needed to query didn't show anything about a leak detector. And I had no idea how to test the leak detection once I did figure it out.

In both cases, I erred by assuming that I could figure things out later. Most of the time, I can -- and being handy at figuring things out goes with the job. But there are limits to our expertise, our area of familiarity, and our ability to learn whole technologies at one sitting. One of the hardest things I've had to realize is that, while I like to think I'm capable of learning just about anything I'm likely to try my hand at, it's not practical -- that there are times when you have to give up and say, "That's just something I'll have to learn in my next life."

I also erred by not asking the installer to walk me through things. I should have asked for simple steps to test whether they were working, how to check for problems, and how to reset them.

Conclusion

Moving tests things and people. You (re-)learn what you forgot about; you find out how to do without missing parts; you come to terms with the limits of being human. It's no less true for being melodramatic, but a few tricks, some obsessive planning, foolhardy volunteers, and hard work will give you the best war story of all: a boring one, where everything worked out just fine in the end.

Further reading:

• LOPSA mailing list thread on Sudden server room move -- lots of good tips
• Lessons learned from a server room migration
• A forum discussion onserver room moving checklists

Day 7 - Active Directory naming is easy, right?

This article was written by Sam Cogan

Active Directory naming is easy, right? You've just got to pick a name for your domain; any name will do won't it?

This is the view many newcomers to Active Directory (AD) take, and it's the view I had when I was first introduced to AD. It often works, even for a while. Then, a few days or weeks down the line, you start to notice problems, or with greater understanding of how AD works, you realise that perhaps there was a better name. By this time, it is too late - the name is set in stone. Sure, you could rename it with the domain name rename tool (rendom), but it's likely to cause problems. Let's look at why AD naming can be problematic and what we can do to make things better.

Microsoft's decision to tie Active Directory closely to DNS, while making sense, has caused a lot of problems for inexperienced sysadmins. One of the most common problems I hear from new sysadmins working with AD for the first time is, "I setup Active Directory with our company's external domain name, but now no-one can get to the company website or ftp site!"

Why does this happen? If your AD domain is example.com, AD will answer DNS queries for that domain, which likely fails to serve external services properly, such as your corp website at www.example.com.

Using your company's external domain name for DNS seems like the perfect idea at first. Limited understanding of how AD interacts with DNS has lead to a decision that may create problems and administrative overhead. Yes, there are potential solutions to this problem: implementing split brain (aka split view) DNS, changing your AD name, or installing IIS on every domain controller to perform redirects. But it's a scary prospect for a new sysadmin who's boss is about to explode because he can't get to their website and is often enough to put them off AD for good. So yes, you can use your external domain name for AD, but in my opinion, you shouldn't. It causes problems, so why give yourself the headache?

I've found there are a number of excuses people give for using the external domain name for AD, and I've used some of them myself. For example, "We had to use our external domain because we want to use that domain name for our UPN suffix". Truthfully, you can have as many UPN suffixes as you like by adding them in the Domains and Trusts MMC. Inexperience with AD may drive assumptions as above, but after digging into it, you will find that your assumptions may not be correct about what you think you need to use as your AD domain.

So, what AD domain name should we use? There are two common schools of thought on this subject: either something like example.local, or use a subdomain of your external domain (like corp.example.com, if you own example.com). Alternately, you can use a different external domain name, but this is not recommended for the general case.

The use of the .local extension came about because it allowed the separation of the AD domain from the registered internet domain (ie; example.com) without having to buy another domain. It's also easy to get an SSL certificate for a .local domain from a trusted SSL vendor, should you need one for internal resources. The alternative is to chose a real, unowned TLD to build your domain on, but you have the obvious risk of that domain being owned by someone else.

Maybe we have a good domain decision, now, with no extra cost? Maybe not! There are problems with using the .local domain. First, it's not a reserved TLD. While it's unlikely, it's possible that IANA could choose to delegate this TLD, opening it up for registration and causing potential name conflicts. Second, the use of .local can also cause problems if you have Apple computers on the network, as it is used by the Bonjour service. Finally, because .local domains are not controlled by a registrar, someone else could be using the same domain name in another AD instance. This problem will bite you when you need to establish trusts or merge domains with another AD instance - if both of you are using example.local you will have conflicts.

Despite these problems, the use of .local is still popular especially in small companies. Microsoft's Small Business server even suggests using this when using its configuration wizard to create an AD domain.

Besides naming with .local, you could choose the name as a subdomain of your external domain, such as ad.example.com, or buy an additional domain for AD only, such as examplecorp.com. Using something like ad.example.com or corp.example.com is pretty common today; Microsoft also recommends this. This is easy and ensures ownership of that domain (unless you forget to renew example.com). Using this method means that your AD DNS server is only responsible for this subdomain and will happily forward on requests for your external websites to servers.

AD naming seems easy, but as we've shown above, there are important considerations when choosing a domain name for your Active Directory domain. This advice, "be careful about seemingly-simple decisions," carries to many other spaces than AD.

Setting up an AD infrastructure should be planned carefully. The domain name choice should be a part of this planning. Consider how your network will be used, and how it will grow over time. If you don't own a public domain name, is it worth purchasing one now so you can ensure your AD domain name is reserved, even if you never use it on the internet. If you do have an external domain name already, try and keep your internal and external DNS separate, you'll appreciate it in the long run. Finally, never use a public domain name that you don't own, you never know who might snap it up and cause you problems!

Further reading:

• Choosing a Future-Proof Internal DNS Domain Name: Mission Impossible?
• Why .com is a bad choice for a Windows Active Directory domain name.
• DNS and Active Directory
• How Domain Rename Works (Microsoft TechNet)
• Domain Rename on Server 2008

Day 4 - Communication and Organization of Changes

Change happens. Maintenance, (planned or unplanned) outages, upgrades, etc. A change can be something small, such as tweaking a configuration value, or they can be large, such as replacing power units on hundreds racks across multiple datacenters. Changes come in all shapes, sizes, scopes, and risks, and require varying degrees of planning, scheduling, and cooperation.

Communication is an important piece of the change process, and good communication allows you to inform users and groups about pending changes and change impact. Good communication requires using the right tools, so it's worth reviewing the tools that are available to you:

• Email. Email is easy to send, and maintaining mailing lists for users of any given component is pretty easy. Email seems cool by itself until you learn that nobody actually reads email (kind of like that intranet wiki...). I'll cover how to work around this shortly.
• Calendar. Everyone understands calendars. They grew up living with them, the presentation is familiar, and it is based on a concept everyone understands: time.
• Bug/Issue/Ticket systems. These systems are good for tracking work units, such as changes. They tend to have status-setting features including words like "need feedback," "in progress," and "resolved." Seems like a good fit for tracking the progress of a change.
• Phones. Phones are good when you need synchronous communication, such as when coordinating a change across geographics, or calling customers to get acknowledgement of a proposed change.
• Meetings. Meetings are good places to announce changes, scheduling, and impact. Attendees can nod quietly or object to the change or scheduling. This helps you review a change and fit its schedule to minimize risk and impact.

Every change will not need to involve every tool listed above. Further, the tools you use to communicate, plan, and log changes will depend greatly on the culture and size of your company and on the impact and risks of each change. Use the tools that fit best.

For email, have an 'it-changes' or 'ops-changes' or 'yourteamname-changes' mailing list that has your team (which can just be you) and anyone else who is interested in the changes you are making. Additionally, create a mailing list for any component that might need maintenance, such as a datacenter location, a service like Active Directory, or the network filer; document these and encourage people needing a particular component subscribe to those mailing lists. Use your judgement here. If your company is small, you can probably create fewer mailing lists.

Like engineers who need to consider unreliable networks in their design, you must consider unreliable readers in your change announcements. Folks don't read email; they skim and read things they think are important, which means they may skip your change announcement. You may have to resort to trickery in order to get people to read important change announcements; try prefixing your subject with "CAKE AND PIE" - it might work? Further, there is often no feedback from email - you won't get any acknowledgement of who has read an announcement, or more importantly, that they have understood it.

The best way to ensure your announcements are read are by targeting only the people who need the information and by repeating your message. Depending on the size and impact of the change, you may need to send your change announcement up to three times. First, to announce the scheduled change. Second, a day (or hour) before the change starts. Third, when the change starts. A final, "all clear," message should be sent when the maintenace is complete.

Email has some failings, like no acknowledgements. Calendars can do this, and more. Calendars are great visual tools for communicating schedules. Online calendars (in Exchange, Google Calendar, whatever) are great for several reasons. First, you can invite people to the event, which gives them a visual reminder in their calendar. Further, you get reminders for free: Just before the event starts, your invitees will likely get a popup reminding them of an change. Additionally, Calendars are shareable and publishable. Calendar data exchange is pretty standardized - iCalendar format sends well over email. Invitees can acknowledge receipt, helping you figure out who hasn't acknowledged and might need a phone call. Finally, calendars can often be downloaded to smartphones and other devices. All of these are excellent features of modern online calendars which will help you communicate changes more effectively.

You should schedule maintenance and outages in a calendar. Create a calendar (or multiple, using the same principles from the mailing list creation above) to track these events. Invite people and groups who need to know about the event. The data in each event should include two things: a short description (think email subject) and a link to wherever the detailed plan/discussion for the change lives, which is likely an issue/bug system. For an unplanned outage, create an event that represents the actual time and duration of the outage.

Ticket (aka bug or issue) systems should be used to track individual changes. As mentioned above, you get the state-tracking benefits (open, pending, in-progress, fixed, etc) and a reasonable place to record planning notes and actions taken during a change. Your emails and calendar entries should include links to the change ticket if it is relevant. The ticket should also, if possible, include a link to the calendar event for easy import into other calendars.

Phones and meetings are of similar use as they both grant you synchronous communication. Phones and in-person meetings are good for the planning stage. They are also both good for reviewing pending changes or for confirming that all involved acknowledge the change. Your use depends on your needs. For example, a previous job had weekly meetings to discuss impact and scheduling of planned changes. Phone is also a useful tool for calling in more experienced teammates when there's a problem with a change.

Lastly, you are a customer, too. Others will make changes that affect you. Datacenter facilities, ISPs, and other service providers make changes just like you do. I would love if my service providers sent me change notifications with calendar invites, but nobody does. Do your vendors send you change notifications? Do they include calendar data you can import into your own calendars? Do they follow the advice above? If you said no to any of these, it's worth having a chat with your vendors and providers to help work towards this. I'm trying to work with my providers to get them to do this, but it's a slow battle due to the current systems and practices, so be prepared and patient.

Remember, we are often guilty of undercommunicating and even communicating poorly. Focusing on effective communication of changes will help ensure your customers (coworkers, users, clients, etc) are well-informed of the changes that affect them. Informed customers are happy customers. As mentioned in the previous paragraph, you are a customer, too. Making you a happy may require you working with your service providers to sell them on the advice here.

Further reading:

• iCalendar and Applications that support iCalendar
• Sysadvent 2008's Visibility and Communication
• Sysadvent 2008's Change Management