Automating Common Development Tasks in a Python Environment

Over the weekend I rolled out a few changes for this blog which made me frequently update the live codebase. After a few ssh's I decided it was time to automate these tasks so I could spend more time developing and less time doing tedious administrative things. Also, the number of times I logged onto my production server to pull from github only to realize I forgot to push is staggering...

Welcome to Fabric

In the world of Python task runners, everyone uses Fabric. However, it does not fully support Python 3 and rather than producing a series of patches and hacks to maintain backward compatibility, the developer has decided to rewrite the package and split it into three, each at a different level of abstraction. Because of this, I use Fabric (with minor hiccups) while impatiently waiting for something higher level than what Invoke has to offer.

Let's start by installing Fabric using pip with the following command:

pip install fabric

This will give you an executable fab in your terminal which will used to run the various tasks.

Fabric assumes that your tasks live in a file called fabfile.py so let's begin by creating an empty file with that name. If you want to see what a complete fabfile looks like, you can see the one I use when working on my homepage here.

To start off, let's define a simple wrapper around django's runserver command:

from fabric import local

def server():
    """ Run the local server """
    local('./manage.py runserver')

One of the functions provided by Fabric is local() which locally executes a bash command relative to the directory we ran the task from. To run this task, type the following command in the same directory as the fabfile:

fab server

Assuming the fabfile is in the same directory as a django project's manage.py, you should now see the normal server output.

Why would we ever do this? Is it really worth wrapping just a single command? Well, imagine in some project you use django-extension's handy runserver_plus as your development server and in others you don't. By wrapping both of them under the same task, you don't have to remember which is the correct command to run.

A Slightly More Complicated Example

While this seems great, there is a slight problem: local() executes the commands in the same directory as the fabfile. So how do we navigate to another directory and execute a command? We could just add a cd <target directory> to every command but that becomes tedious and is not very DRY in the case of many commands. To handle this in a cleaner fashion, Fabric uses what it calls Context Managers.

The idea is pretty simple: using a python with statement, you can prepend the cd <target directory> to each command executed by local().

The additional function that we need to be aware of is lcd() which changes the local directory in which we are executing a command.

To illustrate this, let's define a task which we'll call install_dependencies that navigates into the directory with the dependency files of my project and runs the necessary commands to install the various project dependencies.

In the case of alec.aivazis.com, I have two different package managers that I use: pip and bower. The requirements for each live in a doc directory in the root of my project (where the fabfile lives). Therefore, the task to install the dependencies looks something like:

from fabric import local, lcd

def install_dependencies():
    """ update the project dependencies """
    # go into the folder with the dependency files
    with lcd('doc'):
        # install python dependencies
        local('pip install -r requirements.txt')
        # install bower dependencies
        local('bower install')

You can then execute this task with the following command in the same directory as the fabfile to verify that it works:

fab install_dependencies

Fabric and SSH

So what's the big deal? So far, Fabric looks just looks like a clean command line wrapper around calls to python's execute() function. In fact, that is exactly what local() is. However, the real beauty of Fabric comes in the form of seamless integration with ssh.

For remote tasks, local() and lcd() are replaced with run() and cd() respectively. But before we can actually execute any tasks on our remote server, we have to tell Fabric where it is.

note: the following will change in Fabric 2.x (the developer describes this part as cryptic)

To do so we have to set some global variables in the fabfile:

  • env.hosts should be set to a list comprising of the servers on which the remote tasks will run. If multiple hosts are given, Fabric will run the task on each server.
  • env.user is a string designating the user with which to make the connection to the server.

With these two variables set, we can now define tasks which operate on the remote server.

As an example, consider deploying a django application. There are normally many steps involved with this: we need to push our recent changes, update the remote repository with those changes, migrate the database, collect any static files, and then restart the gunicorn server. The corresponding task in Fabric looks something like:

from fabric import local, cd, run

# the server hosting the project
env.hosts = ['104.236.200.165']
# the user to connect to the server as
env.user = "username"

def deploy():
    """ deploy the application """
    # push any changes before we try to pull 
    local('git push')
    # inside of the remote repository directory
    with cd('repository'):
        # update the repository
        run('git pull origin master')
        # update the database
        run('./manage.py migrate')
        # update the static files
        run('./manage.py collectstatic')
        # restart the application server
        run('sudo service gunicorn restart')

Normally deployment requires 6 separate commands. However, with this task defined, I can deploy with a single one:

fab deploy

Pretty clean, huh? And even better, there's no chance I'll ever try to pull without pushing again.

blog comments powered by Disqus