December 14, 2008

Day 14 - UDPcast

In the introduction of "The Practice of System and Network Administration," the authors point out a few critical things to do first. One of these things is "start every new host in a a known state."

Having a single system image for each platform you support is a great way to save yourself time and mistakes. If you deploy Ubuntu on your servers, you can save yourself time by installing Ubuntu on a single server, configuring some defaults (which you document somewhere or automate), and then take a snapshot of the harddrive for cloning. Once you've done this, you have a very easy method for moving from a host in an unknown state to a host in a known state: clone the drive.

Automation depends highly on starting from a known state and ending in a known state. Some platforms give you a system for automated system installation, such as Solaris and JumpStart, or Red Hat and KickStart. These tools often perform all the normal tasks an installation would, which may take more time than you can budget for if you need to reimage machines often. Having a standard system image you push to every server (or set of servers) will help you automate both the installation and repair processes at the same time.

Building an imaging infrastructure yourself isn't the most trivial of tasks. Lots of moving parts, depending on the tools and hardware you have: dhcp, tftp, pxe, disk cloning, an image file server, etc. There are many disk imaging and cloning tools available to you. Some are stand-alone suites, some are single tools that can help you build an imaging infrastructure.

If you don't have the budget to invest in imaging tools, or can't find a project (commercial or open source) that suits your needs, you may have to build your own. For sending and receiving disk images, you should use UDPcast.

UDPcast is a tool that can use multicast to transfer data to multiple systems at once. High-packet rates lead to dropped packets, which can lead to broken file transfers, so UDPcast compensates by periodically confirming if each connected client has received all packets and retransmits those that were dropped. It supports unicast too, if you only need one client or can't support multicast between the sender and receiver.

The UDPcast project has good documentation and tools for making a system for imaging under various situations. The UDPcast itself consists of two programs, udp-receiver and udp-sender. All configuration occurs in command-line flag arguments. To show UDPcast in a simple example, on two different hosts, A and B, on the same LAN:

HostA % echo "hello" > /tmp/udptest
HostA % udp-sender --file=/tmp/udptest
Udp-sender 2007-12-28
Using mcast address
UDP sender for /etc/motd at on eth0 
Broadcasting control to
At this point, HostA is listening on the address listed above. When it says "broadcasting control" it means it broadcast a hello packet so any already-active udp-receiver processes.
HostB % udp-receiver --file=/tmp/testoutput
Udp-receiver 2007-12-28
UDP receiver for /tmp/testoutput at on eth0
received message, cap=00000009
Connected as #0 to
Listening to multicast on
Press any key to start receiving data!
sending go signal
bytes=              9  (  0.00 Mbps)              9 
Transfer complete.

HostB % cat /tmp/testoutput
Seems simple enough. If you wanted to use it for imaging, you would send a cloned disk image with udp-sender and write it to your target harddrive. For example, "udp-receiver --file=/dev/hda"

You might get better performance if you use compression with your transfer. Keep your disk image in a compressed format and then using udp-receiver's --pipe flag to decompress it. If you compress it with gzip, using 'udp-receiver --pipe "gzip -dc"' will decompress it before writing to disk. Personally, I've had better experience with lzop for compression. Gzip beats lzop slightly in compression ratio, but lzop makes up for it by much faster decompression and less cpu usage.

How can UDPcast fit into your imaging infrastructure?

If you're stuck rolling your own system, reimaging a system might involve netbooting a linux and using udp-receiver to dump a disk image to /dev/sda. You'll need another server broadcasting these disk images on request. For that, udp-sender has a daemon mode. If daemon mode doesn't suit you, running it in a loop will work too.

Automation needs to have a starting point, and automated system installs is critical to saving yourself or others trips to the datacenter.

As a side note, udpcast works for deploying a Windows disk image too, not just Linux, Solaris, and other unix-like platforms.

Further reading:


Anonymous said...

Debian/Ubuntu have several automatic installatoin packages: FAI (which can be used for other distros/OSen), replicator, systemimager, autoinstall. You can also preseed when using debian-installer.

Jon Dowland said...

When using kickstart/FAI etc. it is entirely possible to have nuances in the default state, so it isn't perfect (just the other day I installed two machines at the exact same time and one tripped over a configuration problem that the other didn't have: some aspect of the installation is non-deterministic but I haven't figured out quite which bit yet).

However I cannot see doing a raw filesystem-level imaging being remotely feasible, unless you have totally homogeneous hardware. Do you do the imaging at the partition level, or lower? If lower, do you know the parameters for every single hard drive you might have deployed? If at the partition layer, is your system flexible enough to build the partition tables for each different scenario you might face?

Anonymous said...

I've been using mondorescue for this. Works better when drive sizes are different etc, assuming your kernel isn't tuned so tight it won't support other hardware.

Jordan Sissel said...

The deployment image I like to use is one that has fairly broad hardware support and is very small. For instance, a 2 gig disk image that contains only the necessary data.

You should have a script, that on first boot of the freshly-imaged host, will create any partitions or volumes necessary and make filesystems, or just grow the root fs, or whatever.

So, Jon, to answer your question. Yes, the system is flexible enough to build whatever you need, but the solution doesn't come from using udp-cast to blast an image, but from the startup and self-configuration decisions the host will make once it has been imaged.