Day 15 - Replacing Init Scripts with Supervisord

I find it kind of tragic that System V (SysV) init (/etc/init.d, etc) has survived as long as it has. Looking at its features, it's a pretty plain system, so why is it still here? Maybe because, historically, it has been good enough. However, in the places it hasn't been good enough, folks made workarounds, extensions, and even efforts to vanquish it entirely.

Sometimes these workarounds and extensions are done improperly. SysV init often forces upon programs common tasks such as pid file management, daemonizing (backgrounding), log management, privilege separation, etc. Most things don't use pid files correctly (nobody locks them), many startup scripts lack a functioning 'status' check, and everyone reimplements the daemonizing and privilege dropping differently and many times incorrectly. Yuck, I want less of that.

Sometimes you also need more. What was cool in 1980 may not be cool today. Providing you more is a project called supervisord.

What more is needed? Automatic restarts, centralized control, common configuration, etc.

What if you need more? That's where supervisord comes in. Supervisord gets you a few more goodies you may find useful, including a management web interface, an API, including process startup backoff.

Installation is pretty simple and comes with a sample config generator for helping decrease the learning curve. Further, the config format looks like INI format, so the learning curve there should also be pretty short.

I decided to try putting mysql into supervisord for testing, so after creating the default config file:

# echo_supervisord_conf > /etc/supervisord.conf

I checked ps(1) for how I was running mysql, and put that invocation in supervisord.conf:

[program:mysqld]
command=/usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-external-locking --port=3306 --socket=/var/run/mysqld/mysqld.sock

Then I ran supervisord (you probably want supervisord to to have an init script, so it launches on boot, etc)

% sudo supervisord

# it's running, so let's check on our mysql server:
snack(~) % sudo supervisorctl status       
mysqld                           RUNNING    pid 26028, uptime 0:00:28

Hurray! Supervisord even tries understanding what exit statuses mean. If I send SIGTERM to mysqld, then mysqld will shutdown gracefully and exit with status 0. By default, supervisord is configured that an exitcode of 0 is "expected" and thus it won't restart the process. You can change this by setting the 'autorestart' option to 'true' or by changing which exit codes supervisord understands as expected with the 'exitcodes' option:

[program:mysqld]
command=/usr/sbin/mysqld ...
autorestart=true

Supervisord can be told to reload (restarts itself) or to reread the config file. In this case, we can just tell it to reread the config file:

snack(~) % sudo supervisorctl reread 
mysqld: changed

Now any mysqld exit will be restarted automatically.

Additional features include the management web interface, which can be enabled in the config file with the [inet_http_server] configuration and also with the [unix_http_server] configuration for local-only management. The supervisorctl tool can talk to remote servers, so the http server portion isn't just for people.

Supervisord also seems easy to extend and supports event notifications if you need it. Supervisord also handles logging of stdout and stderr for you in a reasonably configurable way.

At time of writing, there are a few shortfalls.

• No HTTPS support on the management interface
• No decent per-program access control
• No 'retry forever' option. startretries defaults to 3 and has no value for infinity. Perhaps setting a really huge value for startretries is a reasonable workaround. I tried startretries=100000000 which seems to work,

Supervisord may not replace all of your startup scripts, but I highly recommend it, or something like it, for your important services.

Further reading:

• Init scripts considered harmful
• Supervisord covered at PHP Advent 2009