December 15, 2009

Day 15 - Replacing Init Scripts with Supervisord

I find it kind of tragic that System V (SysV) init (/etc/init.d, etc) has survived as long as it has. Looking at its features, it's a pretty plain system, so why is it still here? Maybe because, historically, it has been good enough. However, in the places it hasn't been good enough, folks made workarounds, extensions, and even efforts to vanquish it entirely.

Sometimes these workarounds and extensions are done improperly. SysV init often forces upon programs common tasks such as pid file management, daemonizing (backgrounding), log management, privilege separation, etc. Most things don't use pid files correctly (nobody locks them), many startup scripts lack a functioning 'status' check, and everyone reimplements the daemonizing and privilege dropping differently and many times incorrectly. Yuck, I want less of that.

Sometimes you also need more. What was cool in 1980 may not be cool today. Providing you more is a project called supervisord.

What more is needed? Automatic restarts, centralized control, common configuration, etc.

What if you need more? That's where supervisord comes in. Supervisord gets you a few more goodies you may find useful, including a management web interface, an API, including process startup backoff.

Installation is pretty simple and comes with a sample config generator for helping decrease the learning curve. Further, the config format looks like INI format, so the learning curve there should also be pretty short.

I decided to try putting mysql into supervisord for testing, so after creating the default config file:

# echo_supervisord_conf > /etc/supervisord.conf
I checked ps(1) for how I was running mysql, and put that invocation in supervisord.conf:
command=/usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/ --skip-external-locking --port=3306 --socket=/var/run/mysqld/mysqld.sock
Then I ran supervisord (you probably want supervisord to to have an init script, so it launches on boot, etc)
% sudo supervisord

# it's running, so let's check on our mysql server:
snack(~) % sudo supervisorctl status       
mysqld                           RUNNING    pid 26028, uptime 0:00:28
Hurray! Supervisord even tries understanding what exit statuses mean. If I send SIGTERM to mysqld, then mysqld will shutdown gracefully and exit with status 0. By default, supervisord is configured that an exitcode of 0 is "expected" and thus it won't restart the process. You can change this by setting the 'autorestart' option to 'true' or by changing which exit codes supervisord understands as expected with the 'exitcodes' option:
command=/usr/sbin/mysqld ...
Supervisord can be told to reload (restarts itself) or to reread the config file. In this case, we can just tell it to reread the config file:
snack(~) % sudo supervisorctl reread 
mysqld: changed
Now any mysqld exit will be restarted automatically.

Additional features include the management web interface, which can be enabled in the config file with the [inet_http_server] configuration and also with the [unix_http_server] configuration for local-only management. The supervisorctl tool can talk to remote servers, so the http server portion isn't just for people.

Supervisord also seems easy to extend and supports event notifications if you need it. Supervisord also handles logging of stdout and stderr for you in a reasonably configurable way.

At time of writing, there are a few shortfalls.

  • No HTTPS support on the management interface
  • No decent per-program access control
  • No 'retry forever' option. startretries defaults to 3 and has no value for infinity. Perhaps setting a really huge value for startretries is a reasonable workaround. I tried startretries=100000000 which seems to work,

Supervisord may not replace all of your startup scripts, but I highly recommend it, or something like it, for your important services.

Further reading:


Cian said...

System V init seems pretty dead anyway, with the rise of upstart, and SMF on linux and Solaris respectively.

Simon said...

"you can download and install supervisor in one step using easy_install. easy_install supervisor"

I'm sorry but this sort of thing rules out a package 99% of the time. Using a non-standard install method quickly breaks down when you have more than a couple of packages doing this per machine. Unless such a package provides core functionality for the machine I will usually avoid.

Compare with a similar applications such as mmonit ( ) which is already part of many distributions and has packages available for several others.

I'd strongly advise authors of sysadmin tools to put a lot of effort into getting their tools distributed with the OSes ( RHEL, Ubuntu, Debian, SUSE, Fedora are probably the most important) plus a source distribution for the rest.

Anonymous said...

"Further, the config format looks like INI format, so the learning curve there should also be pretty short."
Assuming you are a Windows admin.
The most troubling part for me is the fact that the package has UNENCRYPTED passwords in files. Although most allow encrypted passwords, there is at least one that requires plain text.