Day 23 - Package vs Config management.

Written by Joshua Timberman

Package management is a best practice in system administration. So is automated configuration management. However, the maintainer scripts run by package management tools are an anti-pattern almost in direct conflict or competition with configuration management systems.

In my examples I'm going to talk about Debian packages and Chef, because that is what I use. Adapt your mindset for your own favorite distribution and configuration management tool.

Server Lifecycle

When almost all the modern, popular Linux distributions were created, servers had a general lifecycle, and an expected supportability throughout that lifecycle. Some distributions have a commercial entity that provides paid support. Others have an excellent user community that volunteers their time to help users and administrators. Many considerations in the development of the Linux distribution stem from the expectation that someone will require support, and the distribution should provide a supportable release. In addition to this, the package's maintainer scripts is what provides additional configuration, such as creating users, or starting services provided by the package.

Package Management

One of the value-adds of most Linux distributions is the package management system. Package management behavior and maintainer scripts are well documented by the distribution to be supportable by a company of support engineers, or a community of volunteers. For system administrators, however, the main reason to use package management is to get some pre-compiled software on the system, and to resolve and install any dependencies that package may have; it is less necessary to have a service start on package install. For example, CouchDB requires Erlang and various other libraries, so the package manager would install those libraries, Erlang and CouchDB. While package management has many other benefits, such as version management, and they can do things like drop off configuration files and start up daemons that were installed. There is definite business value in using packages, and that's why it is a sysadmin best practice.

Many system administrators create their own packages and host them on an internal repository. In most of the environments I've worked in, these packages were as simple as just managing the files included in the package usually ignoring the upstream culture of maintainer scripts and other policies, because the system administrator planned to use a configuration management tool to automate setup and maintenance of the software to run the business application. In these cases, the software provided by the distribution did not meet the needs of the business in some way. Perhaps an application required a newer library version, or you needed to patch in a feature or bug fix, or the default setup of a package conflicted with the way a business application was deployed.

Configuration Management

There are as many different application deployments as there are businesses. The different ways the application stacks are deployed provide a specific business value. The application stack often includes a number of the distribution-provided packages, as well as the code written by the business's software developers.

However, most companies have unique needs when it comes to how the software runs in their environment. Perhaps the HTTP server default configuration isn't properly tuned for the web application that it serves. Maybe the business requires that the MySQL server have replication slaves, and this configuration is not enabled by default. Perhaps the system administrator(s) that run the servers have tuned a particular web server for performance, but it conflicts with another web server package. The actual conflict is based on configuration, not on binaries that are created - both packages by default listen on the same port when the service is started.

For these reasons and more, automated configuration management tools such as Chef are now modern system administration best practice.

The problem we face, is that the packages that we install often run a number of maintenance scripts to ensure that the package is set up and configured. The distribution included the scripts to enforce some policy such as where to put certain configuration files, start services, or where to locate data files created by the packaged software. In some cases, the package maintainer scripts only perform actions when the package is removed (postrm in Debian/Ubuntu), and if there are problems, they don't surface until the package is removed.

Example of the Conflict

To illustrate the conflict between package maintainer scripts and configuration management systems, let's look at a couple use cases with MySQL. We are using Chef to automatically install the mysql-server package on Ubuntu 10.04 LTS running on an instance in Amazon EC2. Our two business requirements are setting a randomly generated root password and move the MySQL data directory to ephemeral storage, as the default location is on a smaller filesystem size. Normally, the package installation on Ubuntu will prompt the user for input on the password, which we then need to work around to automate the package installation. We'll need to generate a preseed file to give the proper settings to the package manager. We install mysql-server on a test system:

sudo apt-get install mysql-server

(And enter a bogus password when prompted, which is what we are trying to avoid).

To get the preseed settings for the package, we need the debconf-get-selections package:

sudo apt-get install debconf-get-selections

Then we get the mysql-server settings for our preseed file:

sudo debconf-get-selections | grep ^mysql-server > mysql-server.seed

We'll use a template that has a generated password (@mysql_root_password), along with the rest of the contents in the file:

mysql-server-5.1 mysql-server/root_password_again select <%= @mysql_root_password %>
mysql-server-5.1 mysql-server/root_password select <%= @mysql_root_password %>

And we set this up with Chef using a template and execute resource:

template "/var/cache/local/preseeding/mysql-server.seed" do
  source "mysql-server.seed.erb"
  owner "root"
  group "root"
  mode "0600"
  notifies :run, "execute[preseed mysql-server]", :immediately
end

execute "preseed mysql-server" do
  command "debconf-set-selections /var/cache/local/preseeding/mysql-server.seed"
  action :nothing
end

Then we have a package resource that installs mysql-server:

package "mysql-server"

Next, we want to configure an alternate location for the MySQL database on the ephemeral storage, as the database size may grow beyond the default root partition size (10G). An example Chef recipe to do this might look like:

service "mysql" do
  action :stop
end

execute "install-mysql" do
  command "mv /var/lib/mysql /mnt/mysql"
  not_if do FileTest.directory?("/mnt/mysql") end
end

directory "/mnt/mysql" do
  owner "mysql"
  group "mysql"
end

mount "/var/lib/mysql" do
  device "/mnt/mysql"
  fstype "none"
  options "bind,rw"
  action :mount
end

service "mysql" do
  action :start
end

We have to stop MySQL, move the directory, and restart MySQL. We use a bind mount so the configuration in /etc/mysql/my.cnf does not need to be changed. If we wanted to do that, there's additional configuration required.

Neither of these scenarios take into account the additional complexity required to manage the Debian system maintenance user set up in the MySQL package, or countless settings possible to set up MySQL tuning parameters, or database formats.

We're forced, here, to do extra work to skirt around problems created by the package management tool trying to be responsible for things outside of packages. The anti-pattern is exacerbated if we have to manage the package and installation on a different OS. Then, we'd have to redo the whole dance for another platform. If our package manager simply dropped the binaries/libraries off and we could handle this configuration directly and much in the configuration management, it would be much easier to manage in a heterogeneous environment.

Conclusion

Package management certainly has value! It allows system administrators to install a base OS image that gives all the hardware support and user-land well known and loved in Unix/Linux systems. When it comes to the application stack required by the business, custom configuration is often required. Package maintainers don't, and can't be expected to, imagine every possible custom configuration. Configuration management tools can, however, be used to cover any custom configuration, since that is their job.

After all, part of the Unix (and Linux) philosophy is that each program should do one thing well.

Further Reading

• Chef
• MySQL Chef Cookbook by Opscode
• Amazon AWS EC2
• Amazon AWS EC2 Instance Sizes
• Debian Policy: Maintainer Scrips
• Unix Philosophy
• Disabling maintiner scripts on Ubuntu - a hack that makes apt-get strip packages of maintainer scripts before they install.

About the author

Joshua Timberman is a Technical Evangelist for Opscode. He has worked for a wide range of companies as a system administrator: from small company IT support to Enterprise web infrastructure delivery for Fortune 500 companies. He helps companies and individuals learn how to use Chef and the Opscode Platform. He wrote the majority of the Chef cookbooks Opscode publishes, teaches the Chef Fundamentals class, and speaks at user groups and conferences. He can be found as jtimberman on Twitter, Skype Freenode, GitHub and more, or via email joshua@opscode.com.

Day 23 - The Dungeon Master's Guide to IT: A Standards Primer

This article written by Ben Rockwood.

A couple weeks ago, I was trying to architect the next evolution of security infrastructure and made an outline of major areas in which I need to focus. Pondering the list it occurred to me that it looked like the table of contents of a standard. I'd never really paid much attention to them, after all, everyone gripes about them and claims they are bureaucratic trash imposed on good engineers by dim-witted management. Why waste my time?

But then, I stepped back, with a child's eye, and admitted to myself that I really had no idea what any of this stuff meant. After all, thats the problem with security: when are you done? When can you say "Great! Its secure!" and move on? I've always hate security, and I think this ambiguity was precisely why.

So I brew a strong pot of coffee and create a new page in my wiki: "Industry Standards". Like many SysAdmin's I first need to get a "lay of the land", to orient myself in the subject before diving into components. ... About a week later, I think I'd started to make headway. I had no idea just how deep the rabbit whole went and quickly became obsessed with the subject.

Several things became clear during my studies. The first was that IT is struggling to leave adolescence and grow into manhood. The one thread that runs through all standards and frameworks out there is that IT can no longer be a special ops part of the company. Rather, it needs to mature and integrate with the larger corporation just like sales or marketing or facilities.

There are several reasons IT needs to stop being the corporate step-child and come under the fold, chief among them SOX compliance. Prior to SOX it was easy for management to say "Look, I don't want to know all this tech crap, just make sure our people have what they need and do your job." The blind eye of management. But with SOX, it became clear that the IT folks hold all the keys to the corporate data kingdom and needed strict oversight. I mean, the government is putting the pressure on finance, its only a matter of time they put the pressure on the people that keep the data that finance is reporting on. Is the data managed properly? Is the data secure? Is the data protected? What started with accountants is now putting the entire IT operation into question.

Thanks to SOX ambiguity, people start searching for solutions to help them comply, and thankfully a great deal of work had already been done. Thus, a variety of "frameworks" to implement controls (procedures and checks that keep things on the up-and-up; like the guy that works the register doesn't count it). Quickly auditors started agreeing that the best way to fill in the missing regulatory gaps was simply to verify the company against these frameworks and an industry of compliance and standards writing took on a whole new life.

When looking at standards there are some details to understand up front. There are "standards specification" or "requirements" that you actually can be certified against. There are "frameworks" which are series of controls which are basically like Dungeons & Dragons DM Guides, they tell you how to play the game. Lastly, there is "guidance" or "best practice", which aren't standards in the sense that you certify against them but rather you use them to help you implement the standard.

So lets start at the top: COSO. COSO is an internal control framework for corporations created in 1985 to combat the fraud and bad financial reporting of the 70's and 80's. Companies would voluntarily adopt COSO as a framework in which to run their business.

Modeled after COSO, the COBIT (Control Objectives for Information and related Technology) framework was created for IT governance. Instead of being aimed at top management on how to run the company in a responsible way, like COSO, it outlines how the IT organization should interact with the company as a whole. It tells the CEO what to expect from IT and what IT should do for the CEO.

COBIT plays a big role in de-geek-ifying IT and making it a more integrated part of the overall business, with roles and responsibilities and processes. Some of its controls include managing people, quality, problems, assets, and all sorts of not so fun stuff.

COBIT is a great framework, but how do you certify your organization? How do you implement it? This is where ITIL and ISO20K come in.

Of all the IT standards guidance, the Information Technology Infrastructure Library (ITIL), has gotten the most interest. Currently in its 3rd version, ITIL is nothing more than a series of 5 books (expensive books, $600 for the set) that define IT Service Management (ITSM) best practice guidance. The emphasis is that IT is a service organization and should align itself to service the greater corporation, so it directly supports the direction set by COBIT. However, the two are distinct and not dependent on each other.

Whether you're trying to become a compliant organization or you simply want some ideas on how to properly structure an IT group, ITIL has become the defacto authority on the subject.

Inevitably, you'll want to certify that you're running a well-oiled IT organization and that your IT governance is up to spec, and so ISO 200000-1 (ISO20K) defines "Information Technology - Service Management: Specification". If you're looking for SOX compliance this is one you may need to audit against. But lets step back.

ISO20K can provide SysAdmin's in the trenches with something very useful, a checklist that outlines what a proper IT organization should look like. Are you doing problem management? Configuration management? Change management? Do you see the value in these processes or are they just a burden? How should an IT organization be organized and run? ISO20K can help bring all these questions into a structured discussion and thought exercise. Maybe you don't agree with parts of it, or think its too much, but ISO20K gives us a stick in the sand to orbit and ponder.

So, to review so far, the big guys use the COSO framwork, the IT guys use the COBIT framework, and they turn to ITIL for guidance and certify itho touches credit card data. The telecom industry gave us TIA-942, the "Telecommunications Infrastructure Standard for Data Centers". On and on and on.

In addition to these, I want to point out that they bump up against the two big project management standards as well, namely the popular US standard PMBOK ("Project Management Body of Knowledge") and the popular European standard PRINCE2 ("PRojects IN Controlled Environments"). Whether or not you care much about project management, when you get into the standards would you will see these two pop up from time to time, so at least learn to recognize them.

--

So why am I writing sysadmins about all this? Because I think generally we're a very curious bunch and have a natural desire to organize things into efficient systems. While we also have a gung-ho DIY instinct, ultimately we do realize that having at least some point of reference is a useful measure.

Whether your in a shop thats implementing standards and your only seeing the tasks without the big picture, or your the big man in a small shop wondering how to better organize your shop, these standards and frameworks can really help you both better understand the direction of the industry and provide a helpful second opinion on your method. No process created in committee will be perfect for your specific needs, but I encourage you to at least educate yourself and see if it doesn't change the way you think about your job.

The important takeaway is this: all these standards and frameworks and guidance are just books! Read them. Understand them as much as you can. Some are free and some are not but seek them out and you'll find them. Knowledge really is power and if you want to play a bigger role in your organization or prepare yourself for the future this is the to start.

Further reading:

• COSO
• COBIT
• ITIL
• ISO20K
• PCI DSS
• ISO27K
• CMMI
• PMBOK
• PRINCE2

Day 7 - Active Directory naming is easy, right?

This article was written by Sam Cogan

Active Directory naming is easy, right? You've just got to pick a name for your domain; any name will do won't it?

This is the view many newcomers to Active Directory (AD) take, and it's the view I had when I was first introduced to AD. It often works, even for a while. Then, a few days or weeks down the line, you start to notice problems, or with greater understanding of how AD works, you realise that perhaps there was a better name. By this time, it is too late - the name is set in stone. Sure, you could rename it with the domain name rename tool (rendom), but it's likely to cause problems. Let's look at why AD naming can be problematic and what we can do to make things better.

Microsoft's decision to tie Active Directory closely to DNS, while making sense, has caused a lot of problems for inexperienced sysadmins. One of the most common problems I hear from new sysadmins working with AD for the first time is, "I setup Active Directory with our company's external domain name, but now no-one can get to the company website or ftp site!"

Why does this happen? If your AD domain is example.com, AD will answer DNS queries for that domain, which likely fails to serve external services properly, such as your corp website at www.example.com.

Using your company's external domain name for DNS seems like the perfect idea at first. Limited understanding of how AD interacts with DNS has lead to a decision that may create problems and administrative overhead. Yes, there are potential solutions to this problem: implementing split brain (aka split view) DNS, changing your AD name, or installing IIS on every domain controller to perform redirects. But it's a scary prospect for a new sysadmin who's boss is about to explode because he can't get to their website and is often enough to put them off AD for good. So yes, you can use your external domain name for AD, but in my opinion, you shouldn't. It causes problems, so why give yourself the headache?

I've found there are a number of excuses people give for using the external domain name for AD, and I've used some of them myself. For example, "We had to use our external domain because we want to use that domain name for our UPN suffix". Truthfully, you can have as many UPN suffixes as you like by adding them in the Domains and Trusts MMC. Inexperience with AD may drive assumptions as above, but after digging into it, you will find that your assumptions may not be correct about what you think you need to use as your AD domain.

So, what AD domain name should we use? There are two common schools of thought on this subject: either something like example.local, or use a subdomain of your external domain (like corp.example.com, if you own example.com). Alternately, you can use a different external domain name, but this is not recommended for the general case.

The use of the .local extension came about because it allowed the separation of the AD domain from the registered internet domain (ie; example.com) without having to buy another domain. It's also easy to get an SSL certificate for a .local domain from a trusted SSL vendor, should you need one for internal resources. The alternative is to chose a real, unowned TLD to build your domain on, but you have the obvious risk of that domain being owned by someone else.

Maybe we have a good domain decision, now, with no extra cost? Maybe not! There are problems with using the .local domain. First, it's not a reserved TLD. While it's unlikely, it's possible that IANA could choose to delegate this TLD, opening it up for registration and causing potential name conflicts. Second, the use of .local can also cause problems if you have Apple computers on the network, as it is used by the Bonjour service. Finally, because .local domains are not controlled by a registrar, someone else could be using the same domain name in another AD instance. This problem will bite you when you need to establish trusts or merge domains with another AD instance - if both of you are using example.local you will have conflicts.

Despite these problems, the use of .local is still popular especially in small companies. Microsoft's Small Business server even suggests using this when using its configuration wizard to create an AD domain.

Besides naming with .local, you could choose the name as a subdomain of your external domain, such as ad.example.com, or buy an additional domain for AD only, such as examplecorp.com. Using something like ad.example.com or corp.example.com is pretty common today; Microsoft also recommends this. This is easy and ensures ownership of that domain (unless you forget to renew example.com). Using this method means that your AD DNS server is only responsible for this subdomain and will happily forward on requests for your external websites to servers.

AD naming seems easy, but as we've shown above, there are important considerations when choosing a domain name for your Active Directory domain. This advice, "be careful about seemingly-simple decisions," carries to many other spaces than AD.

Setting up an AD infrastructure should be planned carefully. The domain name choice should be a part of this planning. Consider how your network will be used, and how it will grow over time. If you don't own a public domain name, is it worth purchasing one now so you can ensure your AD domain name is reserved, even if you never use it on the internet. If you do have an external domain name already, try and keep your internal and external DNS separate, you'll appreciate it in the long run. Finally, never use a public domain name that you don't own, you never know who might snap it up and cause you problems!

Further reading:

• Choosing a Future-Proof Internal DNS Domain Name: Mission Impossible?
• Why .com is a bad choice for a Windows Active Directory domain name.
• DNS and Active Directory
• How Domain Rename Works (Microsoft TechNet)
• Domain Rename on Server 2008