Day 3 - Development for Sysadmins

This article written by Saint Aardvark the Carpeted

In a previous life, I was the sysadmin at a small non-profit that helped organize conferences. We solicited proposals, approved budgets, and helped the organizers whenever we could. Since the organizers were scattered around the country, we used web-based tools to do all this. My newly-created position combined that of a part-time SA and part-time web developer: provide registration and payment forms, and let organizers keep track of registrations.

There were a number of problems^Wchallenges I faced:

• The database had evolved over time, and so had the web front end we used to manage it. The database schema was completely undocumented.
• Registration forms had been created anew for each conference, because each conference's requirements were different.
• Another person who worked on the database front end didn't believe in revision control, or testing his changes; it was live data or nothing, baby!
• Everything was written in PHP. PHP, used in anger, lets you get away with a lot of bad things.
• Conference season was coming up -- soon.

What to do?

I did the only thing I could do: I made mistakes. I trampled over data. I banged my head against my desk and cursed the day I'd ever taken this job. And I learned some things that developers are taught -- and that we sysadmins are rarely told.

Maybe it's assumed we'll never need to know these things....yet in my experience, we're often called on to be developers. Maybe you're at a small shop like I was, and there simply isn't anyone else to take care of The Big Important App. Maybe you're adapting a Free Software project for inside use. Or maybe you're finding that the quick hack, lashed together in desperation and anger, has suddenly become useful for other people...and now they're coming back with ideas.

Development is more than learning a language. That part is easy>, like learning how to use a hammer, or a drill, or a saw. Learning how to build something that doesn't fall apart in the slightest wind is hard, whether you're doing programming or carpentry.

When that app falls apart -- or better yet, before -- here's what to do.

Use revision control

Use it always; there is simply no excuse not to. You need it to:

• Record changes -- not just what, but why
• Revert easily to older, working versions when you screw something up
• Tell your boss what bugs you've fixed

If there's something already in place at work, or that you're already familiar with, or you're working on a project that has settled on a particular tool, use it. Otherwise, use Subversion; it's simple, it works, it's available for your OS, and you can always convert later on.

Learn how to commit and revert changes, then read up on branching, merging and tagging -- not just how, but why. Are you going to be introducing a new feature? Start a branch, then merge back to the main line. Are you going to be making a release? Start a branch, do what you need then leave it unchanged.

What if you're the only developer? Then it's at least fourteen times more important. Trust me, you're not going to remember what the difference was between "foo.php.old-1" and "foo.backup.2" at 4:15 on a Friday afternoon, let alone at 3am on a Sunday morning.

Read the books that are available for your tool, and the ones for the tools you passed up. Not only do they usually contain good introductions to revision control, but you'll get an idea of what kind of development style they encourage. Don't get hung up on that yet; just stick it in the back of your head.

Learn how to test

Here's a good mistake: on one registration form, I forgot to insert email addresses into the database. I was unaware of this until the organizer asked about emailing everyone to notify them of a change in venue. I fixed my mistake (confirmation emails CC'd to myself FTW!); then I started looking for ways to test my code.

There are four kinds of tests you'll hear about:

• Unit testing ask "Does this module/function do what I think it does?" In my case, I added a unit test that looped over all the data I thought I was inserting into the database and made sure they were there.
• Regression testing asks "Have I re-introduced any old bugs in my latest changes?" Every time you find a bug, add a test before you start work on a fix. That test should fail until you commit your fix; after that, it should continue to pass. If it doesn't, you've re-introduced a bug. (The line between unit testing and regression testing can be fuzzy; don't worry about that.)
• Integration testing asks "Does this component work well with that component?" Two components may work correctly when tested independently, but all manner of bugs can occur when you start plugging components, libraries, and other modules together.
• Acceptance testing asks "Does the customer like what I've done, and does it do what they expected?" Depending on the project, the customer may be you, your boss, your co-workers or the public. This is another way of saying "It's not done 'til they say it's done."

You'll hear people talking about test-driven development. This means you write a test before you write any code. By laying out what you expect in advance, you know exactly when you're done (because the test passes) and you can refactor (rewrite) a function easily if it becomes cumbersome. If you've got the time and you're starting from scratch, it's an excellent way of doing things. Otherwise, you may find it difficult to bolt on afterward.

The way you test your code may depend on what language you're using and what you're familiar with. In my case, I used Perl for testing, even though the code it was testing was PHP. I was more fluent in Perl, and there were two excellent modules (WWW::Automate and Test::More) that made it easy to test my forms. (I'm not familiar with tools for testing more AJAX-y web apps, but there are lots of open-source tools that are worth investigating.)

You should be testing your code at least in preparatation for beta testing; in my case, running the tests took less than a minute, so it was trivial to run them near-constantly.

Learn about development, testing, and production

If you're a sysadmin at a larger company, you probably know about the difference between these environments already...but my background is different, and this was something I had to learn the hard way.

• Development is exactly what it sounds like: you banging on your keyboard and writing stuff.
• Testing is where you put the changes you've made through all your tests.
• Production is when your code goes live.

Big places will probably have different machines and environments for each. (They probably also have golden coffee cup holders for their sysadmins.) You may find that the line between these environments is more fuzzy. But you should have some kind of sandbox for your development and/or testing.

When you're testing, think about how you're going to get convincing/realistic data. In my case, I was able to take a copy of the live database and test with that.

At a small shop, you may find it difficult to isolate yourself completely from your production environment. In that case, work hard to make it hard to shoot yourself in the foot. In my case, I was paranoid about accidentally connecting to the live database while I was testing. To get around that, my copy of the database was renamed and used completely different credentials...so I couldn't connect by accident.

You should also think about how you're going to move this to production. I set up my registration forms to automatically switch to testing mode if the file path they were in included "sandbox"; they used those different credentials for connecting to the database, emails went only to me, and more debugging info was shown. This made it easy to make it live: just move the files over.

(An important corrollary of this is: have all your configuration info in one file; do not have Multiple Sources of Truth(tm). You will forget to edit something someday, and you will unwittingly unleash the Reign of Zuul.)

Everything should be a Makefile

The developer who hired me for my first SA job had a valuable piece of advice: Everything should be a Makefile. What did he mean by that? Simple:

• You should be able to type one line, hit enter, then walk away and come back to a fully compiled and ready-to-install program.
• You should have everything you need to compile/assemble/test your program.
• You should be able to run through all tests with one command.

Makefile syntax can be hairy, no question, but it's a good way of having one command do it all. If you have to enter another command, it's not done yet.

Pick a coding style and stick with it

Placement of brackets, whitespace and functionNamingStyles are just like every other religious war: it's not usually worth your time. The point is to pick something and move on. Consistency will help you -- and your successor -- get familiar with your code again after a long absence.

If you don't know where to start, have a look at the Linux coding style document, or at Perl Best Practices by Damien Conway -- whichever is closest to hand right now. If your language offers a tool that does automagic formatting (like Perltidy or phptidy), use it. Otherwise, here are some things that work for me, cherry-picked from both of those books:

• Stick to 80 columns, and remember that a function more than 3 screens long should probably be split up. (Why? Because screen space is equivalent to brain space, and you don't have that much.)
• Don't use global variables. (Why? Because you'll change it somewhere else and think you'll keep track of it...but you won't. Or you'll have to scroll through fourteen different functions to find where it changed; see above re: screen space.)
• If a function has more than two arguments and your language supports it, name them. For example:

  foo(foo="1", bar="2", baz="aardvark", bling="emacs")

  rather than

  foo(1, 2, "aardvark", "emacs")

  (Why? Because you'll forget the right order.)

Are you detecting a pattern here? Good, because that brings me to the final point.

Remember, you're not going to remember

Documentation, comments, consistent coding style, shooting global variables on sight -- these are all synonyms for "I need to write this down." Just like you document your network, or your web server config, or your vendor support, you need to write down what you've done, you need to make it understandable, and you need to make it simple.

And keep in mind what Brian Kernighan said:

Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it?

Words worth remembering -- avoid trying to be clever.

Further reading:

• The Subversion Book
• The Git Community Book
• Pro Git Development
• Mercurial: The Definitive Guide
• Open Source Development with CVS
• Perltidy
• phptidy
• The Linux kernel coding style document
• Perl Best Practices