Day 16 - Hudson: Build Server and More

Hudson is a job monitor. It is primarily used as a build server for doing continuous builds, testing, and other build engineering activities. Continuous build and integration is a useful tool in improving the day-to-day quality of your company's code.

My first week's impression of Hudson was that it is really great software. Things work the way I expect! It has good documentation and a good web interface. Additionally, the APIs are easy to use. Compared to other learning curves, Hudson's was extremely short thanks to one part documentation and one part ease of use. I was building some work-related software packages in hudson in only a few minutes of playing with the tool.

Setting up a new job in Hudson is really easy, and every field in the job configuration interface has a little question mark icon that reveals useful documentation when clicked, so it's not often you get lost. Functionally, it has the build-related features I expect: build on demand, build if there's been new commits, email folks who break builds, show reports on build histories, etc.

Getting a bit more advanced into Hudson beyond simply building stuff or running jobs, let's talk administrative tasks. Speaking of administrative tasks, Hudson has administrative documentation detailing how to backup and restore Hudson configurations, renaming jobs, etc. Hudson's configuration is stored in XML files in a sane directory heirarchy that makes it easy to backup specific jobs or specific configurations.

Hudson also has an API. The things you need to know about the API is that any url you visit on the web interface can be accessed with the API. Adding '/api/' to any url will give you the API documentation for that url - only one thing to remember when asking "what's the API for this page?" - totally awesome. Check out these screenshots of exactly that:

I wanted to take the latest successful build of a specific job and put the resulting files (artifacts) into my local yum repo. Artifacts are what Hudson calls the files you archive after a build is complete. Fetching the artifacts from the latest successful build of any given job is fairly straightforward from the web interface. The XML api makes this easy, allowing you to find the artifacts for your builds from scripts:

% GET http://build/hudson/job/helloworld/lastSuccessfulBuild/api/xml
<?xml version="1.0"?>
<freeStyleBuild>
  ...
  <artifact>
    <displayPath>rpmbuild/RPMS/x86_64/helloworld-1.0-1.x86_64.rpm</displayPath>
    <fileName>helloworld-1.0-1.x86_64.rpm</fileName>
    <relativePath>deploy/rpmbuild/RPMS/x86_64/helloworld-1.0-1.x86_64.rpm</relativePath>
  </artifact>
  ...
</freeStyleBuild>

Parsing XML in shell without the proper tools should make anyone a sad panda. Luckily, Hudson's XML api allows you to give an XPath query to restrict the output:

# Show me the text contents of the first artifact/relativePath
% GET 'http://build/hudson/job/helloworld/lastSuccessfulBuild/api/xml?xpath=//artifact[1]/relativePath/text()'
deploy/rpmbuild/RPMS/x86_64/helloworld-1.0-1.x86_64.rpm

That path is relative to the URL without the '/api/xml' part, so fetching the RPM becomes: http://build/hudson/job/helloworld/lastSuccessfulBuild/deploy/rpmbuild/RPMS...

Fetching the RPM was only the first step of a deployment process I was building with Hudson. At work, sometimes engineers do deployments. It is not totally productive to require them to have working knowledge of rpm, yum, mrepo, puppet, linux, ssh, and other tools that may be necessary. The deployment learning curve can be reduced to almost zero if we have a one-click deployment option that would do all the right things, in the right order. Having this would also save us from having to (poorly) maintain a wiki describing the steps required to perform an on-demand upgrade.

As shown above with the API, we can fetch the latest packages. Once that happens, the exact method of deployment is really up to you and what your infrastructure needs. My version pushed the rpm to my yum master, then replicated to the per-site mirrors, then ssh'd to each related server, upgraded the package, and restarted any service required. The benefits of this are two-fold: we retire a poorly maintained upgrade howto document, and we relieve some engineers of the burdens of being intimate with the infrastructure, so they can just worry about writing and testing code.

One small trick, however. Since Hudson has a 'build now' button, I didn't want stray clicks or accidents to incur a new deployment. The workaround was to add checkbox to the build that simply asked for confirmation. Without the checkbox checked, the build would fail.

Now armed with an automated, but still human-initiated, deployment process, your deployment job can be extended to include email notifications, nagios silences, and other actions that help make your deployment more safe and reliable.

While I am not certain that Hudson is the end-game of helping coworkers do deployments to production and staging, I am comfortable with this solution given that it works well, was very easy to implement, and relieves the knoweldge transfer problems mentioned above.

Even if you don't like Hudson for deployment, it's a great build server.

Further reading:

• Hudson's home
• Continuous Integration