Day 8 - Sorry, Python, but I just stole virtualenv from you

Written By: Hugh Brown (@saintaardvark)
Edited By: Adam Compton (@comptona)

try:

"Python? Don't want it, don't need it." It was a Central IT voice and a Central IT joke. A bunch of us were at the pub, drawing Venn diagrams of synergies versus core competencies on beer coasters, and as usual the talk had turned simultaneously tech and trash. Tonight's flavour was Python, but Central wasn't having any of it. "Last thing I need is another language under my belt. I've got Bash, csh, Lisp, C, C#, Perl, Haskell, Scala, Make (BSD and GNU), and assembly for two abandoned middle-endian CPUs. Oh, and declaration specs for three different configuration management tools that look suspiciously like Ruby. I've got better things to do than figure out Python."

I took a swig from my beer. "Thing is, you're right."

"Huh?" Heads swivelled at this. My capitulation was obviously a surprise.

"You heard me: you're right. You don't need Python. But one of the best parts of Python isn't the language."

"What is it, then?" Central glared at me. The phrase "smart-ass" was obvious in its absence.

"Virtualenv and virtualenvwrapper. They'll make you weep with joy."

"Pfffft. I haven't wept since I was six and my VIC-20 died."

"No, really." I took another drink and lined up my thoughts. "How about...scripted creation and disposal of test environments on your desktop? A directory for each project, with customizable hooks for entry, exit and activation? And the best part..." I paused. "The best part is that you switch environments in the shell...by using the mnemonic 'workon'." Eyes widened. "Which has tab completion." Pupils dilated.

"You mean it, don't you?" Central's cynical look just, I dunno, melted. "Damn. I...I'd like to feel again."

"Um." I was unsettled.

"Tell me more."

So I did.

except ContextOverflowError:

At $WORK I switch between a lot of different tasks. Some are done quickly, but some take time and effort to finally kill. Sometimes there are a lot of dead ends to exhaust, and there's a lot of context that'll get lost between invocations if I don't commit it to non-volatile memory. This context comes in a lot of forms: scripts, shell history, Vagrant instances, clones of my Cfengine git repository, notes, tickets, whatever. I've taken to organizing all this in directories named after the project or task I'm working on, but I've begun to feel it's time to formalize it a bit more.

Ideally, what I'd like is:

• a command line-based tool that lets me create and destroy working environments easily...
• that can be customized as I like...
• which gives me frictionless switching between environments...
• with nice reminders about what I'm doing.

Turns out there's a tool that's very, very close to this. Only problem is, the Pythonistas wrote it first. Still, that's no reason to let them have all the fun.

from virtualenv* import *

I'm gonna take a moment to introduce virtualenv and virtualenvwrapper. If you're familiar with them, you might want to skip the next few paragraphs.

With any software stack, there's the question of how to manage its environment: how do you ensure that you have what you need, that you can update it easily, and that you don't stomp all over something needed by someone else? The answer is usually isolation in one form or another: VMs if you want total isolation, or lighter-weight environments that use polite walls to separate things. Python developers use the second approach, and two of their best tools are virtualenv and virtualenvwrapper.

Virtualenv came first. It does three main things:

• It creates a directory tree for each new project; this is where all the code and modules will live.
• It installs wrapper scripts for the main Python executable, as well as easy_install and pip (which let you install modules and their dependencies).
• It installs activation scripts (Bash, zsh and fish are supported) that set up environment variables to point Python, easy_install and pip to the new project directory. And as a wonderful bonus, it changes your shell prompt to reflect the project you're working on -- a small but significant thing if you're doing your work in a terminal.

With virtualenv, a non-privileged user can set up a new environment, install all requirements, and ensure that it really is isolated from everything else. However, using virtualenv can take some discipline. Where do you put the directory? What about housekeeping tasks that should be done at creation or destruction? That's where Virtualenvwrapper comes in. (Really, they need to make nicer names for these tools). It adds some welcome features:

• a central place for all your environments
• shell hooks for various stages, sourced or run as necessary
• easy commands to create and destroy environments and switch between them

Put 'em together, and you've got a toolset that no Python developer should be without.

"No weeping for me tonight, bub." Central's voice was harsh. "I told you, I don't need Python. So I don't need an isolated Python environment." Central's nostrils flared, and the glare from across the table pressed at me like a storm front.

"We're getting there. Stick with me."

do_install

The simplest way to install virtualenv and virtualenvwrapper is to let your packaging system take care of it; failing that, check out the links at the end of this article for pointers. Once you've got them installed, you need to do a couple things so your shell will use them. I'll assume you're using Bash, but the fine documentation can help you setting up zsh or others.

1. Add these lines to your .bashrc. The directory names can be adjusted to taste, of course.

   export WORKON_HOME=~/Envs
   [ -d $WORKON_HOME ] || mkdir -p $WORKON_HOME
   export VIRTUALENVWRAPPER_HOOK_DIR=~/.virtualenvwrapper_hook_dir
   [ -f /path/to/virtualenvwrapper.sh ] &&  source /path/to/virtualenvwrapper.sh
   

2. Source .bashrc or restart your shell:

   source ~/.bashrc
   # or: exec $SHELL
   

for i in task:

At last we're ready to go! Here's a typical use case for me:

A new ticket comes in, RT #4484: a user has asked for permission to bounce Tomcat on the test servers. It's an obvious case for sudo. The sudoers file is pushed out by Cfengine to each server, though, so I need to make sure I'm not going to bork things. I decide to set up a Vagrant instance, test the change, and then push it out.

Doing it by hand looks like this:

• mkdir ~/rt_4484
• cd rt_4484
• git clone /path/to/cfengine.git cfengine
• cd cfengine && git checkout -b rt_4484 && cd ..
• cp /path/to/vagrant_helper_scripts .
• vagrant init precise64
• vagrant ssh
• work on sudoers, test, finally commit, exit
• git checkout master && git merge rt_4484
• git push
• vagrant destroy
• cat notes.txt | mailx rt@example.com -s'[rt.example.com #4484] Notes for the record'
• cd ~ && rm -rf rt_4484

That's a lot of work! Virtualenv won't edit sudoers for you, but let's see how far we can get in that list.

"What would you think of someone who just did the bare minimum?"

Start by using mkvirtualenv to create a new virtual environment. It creates a new directory in $WORKON_HOME, and creates wrapper scripts to activate it:

$ mkvirtualenv rt_4484

Now we're able to use workon to activate it:

$ workon rt_4484
(rt_4484) $ cdvirtualenv
(rt_4484) $ pwd
~/Envs/rt_4484

Activation sets or changes a number of environment variables like PYTHONPATH, PATH, VIRTUALENV and PS1. (Note that we used the virtualenv alias cdvirtualenv to actually go to the new directory; we'll fix that in a moment, so that activating the environment puts you in the right place.)

If this was all we could do, it would still be useful; that prompt change alone is an elegant touch, and one directory for all your projects is nice too. But it's the hooks provided by virtualenvwrapper that really make it wonderful. Let's exit this environment, delete it, then customize the wrappers before recreating it.

To leave an environment, you use deactivate (to get you back to your original shell, with its original environment variables) or workon to switch to another environment. We've only got the one right now, so run deactivate, then rmvirtualenv to remove the whole directory:

(rt_4484) $ deactivate
$ pwd
/home/hugh
$ rmvirtualenv rt_4484
Removing rt_4484...

Virtualenvwrapper hooks: this is where the magic happens

The first time virtualenvwrapper.sh is sourced from your .bashrc, it'll create .virtualenvwrapper_hook_dir and populate it with template scripts. Some are sourced (which lets you set environment variables) and others are run; this page explains which are which, and at what stage they're invoked.

The ones I find most useful are invoked before and after:

• creation and removal of environments
• activation of environments

Let's customize them.

postmkvirtualenv

postmkvirtualenv is the hook script that gets run after the virtualenv is created, and in the directory that was created for it. Here's what's in mine:

#!/bin/bash
# This hook is run after a new virtualenv is activated.

vagrant init precise64

# This step takes a while, so I put it in the background while
# setting up my Cf3 repo.  I use my own Makefile to get
# Cfengine going, but that's just me.
(vagrant up ; vagrant ssh -- -t 'sudo apt-get update ; sudo apt-get install -y make;  make -f /vagrant/cfengine/Makefile') &

git clone ${CF3_REPO} cfengine
cd cfengine
BRANCH=$(basename $VIRTUAL_ENV)
git checkout -b $BRANCH
cd ..

That leaves me with a copy of my Cf3 repo ready to edit, and a Vagrant instance booting and about to run a custom Makefile that will set things up the way I like.

postactivate

postactivate is run when the new environment is activated by running workon. In my case, I've got a Vagrant instance ready to go and a fresh Cf3 repo to edit.

When we first created this environment, we had to cd to the new directory by hand. Let's fix that by editing postactivate:

#!/bin/bash
# This hook is run after a new virtualenv is activated.

cdvirtualenv

This is an ideal place to add other handy startup items. For example, maybe I want a way to add a comment quickly to an RT ticket. I could add this:

# I name my virtualenvs rt_\d+ if it's associated with a ticket.
TICKET=$(basename ${PWD/rt_/})
comment() {
  # Bashism ahead!
  if  [[ $TICKET =~ [[:digit:]]+ ]]; then
    echo '$*' | mailx rt@example.com -s"\[rt.example.com # #${TICKET}\] Comment" }
  else
    echo "Sorry, not a ticket."
  fi
}

prermvirtualenv

If you make a mess, you gotta clean it up -- and that's where prermvirtualenv comes in. This hook script runs just before the environment directory is deleted. Here's what's in mine:

#!/bin/bash

vagrant destroy -f
TICKET=$(dirname $(pwd) | sed -e's/rt_//')
cat notes.txt | mailx rt@example.com -s"\[rt.example.com #${TICKET}\] Notes"

We get the cleanup of the working environment for free. Sweet!

finally:

"That's not bad. Not bad at all." Central's voice was nearly a whisper. "But what about the extra Python stuff? The wrappers, the environment variables..."

"Well, I'm not a Python developer, so mostly I just ignore it. I mean, I could see the PATH munging being handy -- just drop something in the bin directory that virtualenvwrapper creates. But it's not something I really use. And yeah, there's a certain sense of waste...those poor Python folks have gone to all that effort, and here I am just using it for shell scripts."

Central nodded slowly and pushed the pitcher across the table at me.

"Thing is," I said, "there are lots of ways to improve this approach. At the very least, I should be looking at mkproject."

"At who now?"

"It's a separate part of virtualenvwrapper, and it acts a lot like mkvirtualenv. The main difference is, it lets you combine plugins -- there's one for cloning bitbucket repositories, and another that installs Django. The author of virtualenvwrapper likes to use these in combination to check out a repo, add the front end, and start going. The plugins themselves are pretty simple. You specify the ones you want to use with arguments to mkproject, so it shouldn't be hard to choose the features I want to have in different environments. This one gets Cfengine, that one gets a separate tmux session, another changes HISTFILE to capture everything I run..."

I trailed off, looking into the middle distance. Then I came to, and poured myself another beer. "But simple as it is, this really works -- for me, anyway. It lets me organize things, keep them separate, switch cleanly between them. And when I jump back into things again, everything's there. Might even be worth learning Python for."

"Word." Central nodded. And I caught just a glint of a tear.

References

• Virtualenv's documentation is here
• Doug Hellman wrote virtualenvwrapper.
• The documentation is here; the page on Tips and Tricks is also worth reading.
• There are some very good tutorials on virtualenv and virtualenvrapper; my favourite is here.