Deploy Django applications with Fabric

Sat 30 November 2013
By Tianyi

There are three main steps to get your web applications out there for the public to consume:

Start --> Develop --> Deploy

And each step contains many tasks. In my previous blog post, I wrote about how I use Fabric to bootstrap Django projects, so that's step one solved; Step two various a lot depending on the development strategy by the team or the person, I will not get into that much since it's really up to the individual to find out what's the best for themselves; So the last one left is how to deploy your shining project into the wild. In this post, I will show you how I did it with Fabric.

Deployment is hard

I wrote in the last blog post that setting up a web project is hard and I believe that deploying a web project is even harder. Because there are so many things can go wrong, that's why many people rather leave it to their system admins. Many of you might have heard of platform as a service (PaaS) business such as Heroku, Google App Engine etc. has grown so rapidly these days. All these services provide deployment strategies for your web applications which you don't have to worry about it anymore. I am the kind of the person who wants to get their hands dirty though, so before I use anything to make my life easier, I'd like to do it the hard way. To deploy a web application, I need:

  • Put my application code onto the server.
  • Install system packages needed.
  • Create a new virtualenv.
  • Install all the python packages needed using the virutalenv created.
  • Create a database if needed. (need it nearly 100% of the time).
  • Run syncdb.
  • Run south migration.
  • Run tests.
  • Set up uWsgi/gunicorn.
  • Set up Ngnix/Apache.

Phew, that's a lot of stuff need to be done. But if you read my previous post, this list only has three extra things in there, putting application code onto the server, setting up uWsgi/gunicorn and Nginx/Apache . The beauty of using Fabric is to grow your task list bigger and bigger, then reuse them over and over again. I use SVN at work and GIT for my personal projects, to create a Fabric task for this is quite simple:

@task
def svn_check_out(repo_url):
    run('svn co %s .' % repo_url)

@tasks
def git_clone(repo_url):
    run('git clone %s .' % repo_url)

And I am gonna use Nginx and uWsgi here, if you already have your uWsgi and Nginx configuration files ready, you can simply copy them from your local machine to the server, Fabric has a command put to do that. Or you can check them into your repo, so it will be checked out when you check out your application code. One problem for me is that for different servers I need to setup different configuration files, replace the IPs, port numbers and so on. Wouldn't it be good if I can use some templates? I can put project specific stuff into the server_conf.py which I talked about in the previous post, and create a Fabric task to populate the configuration files. This is my uwsgi.ini:

# mysite_uwsgi.ini file
[uwsgi]

# the base directory (full path)

chdir = /home/name/project/

# Django's wsgi file

module = example_project.wsgi

# the virtualenv (full path)

virtualenv = example_project/venv

# process-related settings
# master
master = true
# maximum number of worker processes

processes = 10

# the socket (use the full path to be safe

socket = /home/example_project/project/

# ... with appropriate permissions - may be needed
chmod-socket = 666
# clear environment on exit
vacuum = true

I need to replace each variable's value to the project specific value. How can I create a Fabric task to do that? I introduce you a well known template engine written in Python called Jinja2. If you use Django you should be pretty familiar with the syntax. It's suppose to be a faster and more powerful template engine than Django one, but these two things are not really my concern here, I chose it because as a stand alone template engine, it's so easy to use. Here is the template code for my uwsgi.ini:

# mysite_uwsgi.ini file
[uwsgi]
# the base directory (full path)
{% if chdir %}
chdir = {{ base_dir }}
{% endif %}
# Django's wsgi file
{% if project_name %}
module = {{ project_name }}.wsgi
{% endif %}
# the virtualenv (full path)
{% if virtualenv_path %}
virtualenv = {{ virtualenv_path }}
{% endif %}
# process-related settings
# master
master = true
# maximum number of worker processes
{% if processes %}
processes = {{ processes }}
{% else %}
processes = 4
{% endif %}
# the socket (use the full path to be safe
{% if socket_path %}
socket = {{ socket_path }}
{% endif %}
# ... with appropriate permissions - may be needed
chmod-socket = 666
# clear environment on exit
vacuum = true

And the Fabric task:

from jinja2 import Environment, FileSystemLoader


@task
def create_uwsgi_conf():
    templateLoader = FileSystemLoader(searchpath="templates")
    template_env = Environment(loader=templateLoader)
    template = template_env.get_template('uwsgi.ini')

    uwsgi_params = {}
    uwsgi_params['chdir'] = env.project_path
    uwsgi_params['project_name'] = env.project_name
    uwsgi_params['virtualenv_path'] = env.virtualenv_path
    uwsgi_params['processes'] = env.uwsgi_processes
    uwsgi_params['socket_path'] = en.project_path
    with open('./uwsgi.ini', 'w') as output_file:
        output_file.write(template.render(uwsgi_params))

As you can see, it's very simple. Three lines of code to create a jinja2 template:

  • Create a template loader which points to the directory containing the templates.
  • Create a jinja2 template environment.
  • Load the template which created earlier (uwsgi.ini).

Once the template is loaded, pass in a dictionary object contains the variable values and write to a new file. If you created a similar server_conf.py as me, then you've already have most of the variables filled in. :)

I don't want to make this blog post too long, but I do want to put this idea or concept out there for you to think about. There might be a better way to do this, if so, please let me know. :)

I haven't got my perfect Nginx.conf crafted yet, I will though, so keep an eye on my github repo. My previous post and this one have demonstrated how to use Fabric to do various things. I hope this can help you to get started at least. There are still more tasks need to be created for deployment:

  • Create supervisor task to manage uWsgi.
  • Create task for crontabs if you have some scheduled things need to be run.
  • Create Nginx configuration file.
  • Create tasks to start/stop/reload Nginx/uWsgi.
  • Create tasks to check uWsgi and Nginx logs.
  • Set up mail server.

There might be many more, but as I wrote above, the beauty of using Fabric is to grow your task list bigger and reuse them. I haven't finished everything I need yet, but I can already use them daily, whenever I have time or a new task idea, I can just add it to my list. I hope you will do the same. And again, like I wrote in the last post, I will check out Ansible and Saltstack, so please don't comment to say they're better you should use them etc. :)

One last thing is that I just found out, you can use -c RCFILE, --config=RCFILE to specify a configuration file for your Fabric tasks. In that file, you can overwrite default values in the env global variable. So instead of doing what I did in my FabFile.py to import the server_conf.py file, you can use fab your_task -c server_conf.py, then you don't need to import the server_conf.py anymore, all the variables you have in the server_conf.py file will be in the env global variable too. For example, if I define a variable in my server_conf.py file called something_super_cool, then you can call env.something_super_cool in your fabfile.py.

Comments