Automating Common Development Tasks in a Python Environment
Over the weekend I rolled out a few changes for this blog which made me frequently update the live codebase. After a few
ssh's I decided it was time to automate these tasks so I could spend more time developing and less time doing tedious administrative things. Also, the number of times I logged onto my production server to pull from github only to realize I forgot to push is staggering...
Welcome to Fabric
In the world of Python task runners, everyone uses Fabric. However, it does not fully support Python 3 and rather than producing a series of patches and hacks to maintain backward compatibility, the developer has decided to rewrite the package and split it into three, each at a different level of abstraction. Because of this, I use Fabric (with minor hiccups) while impatiently waiting for something higher level than what Invoke has to offer.
Let's start by installing Fabric using pip with the following command:
pip install fabric
This will give you an executable
fab in your terminal which will used to run the various tasks.
Fabric assumes that your tasks live in a file called
fabfile.py so let's begin by creating an empty file with that name. If you want to see what a complete fabfile looks like, you can see the one I use when working on my homepage here.
To start off, let's define a simple wrapper around django's
from fabric import local def server(): """ Run the local server """ local('./manage.py runserver')
One of the functions provided by Fabric is
local() which locally executes a bash command relative to the directory we ran the task from. To run this task, type the following command in the same directory as the fabfile:
Assuming the fabfile is in the same directory as a django project's manage.py, you should now see the normal server output.
Why would we ever do this? Is it really worth wrapping just a single command? Well, imagine in some project you use django-extension's handy
runserver_plus as your development server and in others you don't. By wrapping both of them under the same task, you don't have to remember which is the correct command to run.
A Slightly More Complicated Example
While this seems great, there is a slight problem:
local() executes the commands in the same directory as the fabfile. So how do we navigate to another directory and execute a command? We could just add a
cd <target directory> to every command but that becomes tedious and is not very DRY in the case of many commands. To handle this in a cleaner fashion, Fabric uses what it calls Context Managers.
The idea is pretty simple: using a python
with statement, you can prepend the
cd <target directory> to each command executed by
The additional function that we need to be aware of is
lcd() which changes the local directory in which we are executing a command.
To illustrate this, let's define a task which we'll call install_dependencies that navigates into the directory with the dependency files of my project and runs the necessary commands to install the various project dependencies.
In the case of alec.aivazis.com, I have two different package managers that I use:
bower. The requirements for each live in a doc directory in the root of my project (where the fabfile lives). Therefore, the task to install the dependencies looks something like:
from fabric import local, lcd def install_dependencies(): """ update the project dependencies """ # go into the folder with the dependency files with lcd('doc'): # install python dependencies local('pip install -r requirements.txt') # install bower dependencies local('bower install')
You can then execute this task with the following command in the same directory as the fabfile to verify that it works:
Fabric and SSH
So what's the big deal? So far, Fabric looks just looks like a clean command line wrapper around calls to python's
execute() function. In fact, that is exactly what
local() is. However, the real beauty of Fabric comes in the form of seamless integration with ssh.
For remote tasks,
lcd() are replaced with
cd() respectively. But before we can actually execute any tasks on our remote server, we have to tell Fabric where it is.
note: the following will change in Fabric 2.x (the developer describes this part as cryptic)
To do so we have to set some global variables in the fabfile:
env.hostsshould be set to a list comprising of the servers on which the remote tasks will run. If multiple hosts are given, Fabric will run the task on each server.
env.useris a string designating the user with which to make the connection to the server.
With these two variables set, we can now define tasks which operate on the remote server.
As an example, consider deploying a django application. There are normally many steps involved with this: we need to push our recent changes, update the remote repository with those changes, migrate the database, collect any static files, and then restart the gunicorn server. The corresponding task in Fabric looks something like:
from fabric import local, cd, run # the server hosting the project env.hosts = ['18.104.22.168'] # the user to connect to the server as env.user = "username" def deploy(): """ deploy the application """ # push any changes before we try to pull local('git push') # inside of the remote repository directory with cd('repository'): # update the repository run('git pull origin master') # update the database run('./manage.py migrate') # update the static files run('./manage.py collectstatic') # restart the application server run('sudo service gunicorn restart')
Normally deployment requires 6 separate commands. However, with this task defined, I can deploy with a single one:
Pretty clean, huh? And even better, there's no chance I'll ever try to pull without pushing again.