Caching Django with Nginx and Redis

Posted on December 13th, 2014 in Devops, Django, Nginx, Caching by Rich

Django's caching framework is great. Its been well thought through (as all of the Django framework has been), its easy to utilize and makes our applications much faster! Nginx is also truly amazing, one of the best pieces of software I've used in recent times. It's caching layer is also very very good, but its not scalable. Nginx & Django have plugins/packages for utilizing Redis so I've written a Python module that allows developers to use both plugins to let the respective technology do what they do best.

The drawback with Django's caching layer is that any request has to hit the Django app in order for it to be determined where to load the response from; either the cache or the Django app has to process the request then place it into the cache before its returned. Whilst this is fine, in a high-traffic environment, you're at the mercy of your WSGI and whether it can handle the load.

Nginx on the other hand, can handle load, and lots of it. Nginx's file cache is great and has served us very well, but as our web servers scale out we begin to lose the benefits of it as each server would have its own file cache, so for the same HTTP request, the database would get SQL_QUERIES * WEB_SERVERS hits due to the load balancer distributing the load across the web servers.

So how do we get round this? 

Enter Django Redis Cache. This package is great! It allows Django to communicate with Redis as a cache store very simply. Install via PIP, update your settings.py and point it to the Redis server.

The downside to this is; as mentioned above, the Django application is still doing the brunt of the work. So now what?

Enter Nginx Redis Cache!

This package is simply a wrapper on top of Django Redis Cache which stores the response body, rather than the Django HttpResponse object so that Nginx can serve from Redis directly.

pip install nginx-redis-cache

Django Configuration

The library aims to be as simple to plugin as possible. The Nginx integration is slightly more involved, but from a Django point of view, it hooks into the caching framework very simply.

# settings.py
CACHES = {
    'default': {
        'BACKEND': 'redis_cache.cache.RedisCache',
        'LOCATION': '127.0.0.1:6379:1',
        'OPTIONS': {
            'CLIENT_CLASS': 'nginx_redis_cache.clients.DefaultClient'
        }
    }
}

The configuration for the Redis Cache backend is taken directly from the django-redis package so I won't cover it here, but the only required change would be the CLIENT_CLASS option which must point to nginx_redis_cache.clients.XXX. There are equivalents for all of django-redis' clients.

If you wish to use a cache other than default, set settings.CACHE_MIDDLEWARE_ALIAS to the key of the desired cache:


# settings.py

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',
        'LOCATION': 'unique-snowflake'
    },
    'nginx_cache': {
        'BACKEND': 'redis_cache.cache.RedisCache',
        'LOCATION': '127.0.0.1:6379:1',
        'OPTIONS': {
            'CLIENT_CLASS': 'nginx_redis_cache.clients.DefaultClient'
        }
    }
}

CACHE_MIDDLEWARE_ALIAS = "nginx_cache"

Finally, all thats left is to enable the cache by adding add the cache_page decorator to your views, or add the UpdateCacheMiddleware to settings.MIDDLEWARE_CLASSES


# project/app.views.py

from nginx_redis_cache.decorators import cache_page

@cache_page(60*5)
def index(request):
    return HttpResponse(...)

# OR in settings.py

MIDDLEWARE_CLASSES = (
    'nginx_redis_cache.middleware.UpdateCacheMiddleware',
    ...
)

Nginx Installation / Setup

Unfortunately, Nginx doesn't support Redis out of the box. Fortunately however you have two options: OpenResty which bundles Nginx with many of its 3rd party plugins including Redis2 or a custom Nginx build. Whilst OpenResty is the easiest method, the custom build may work for those who are happy with the repository versions and just need the additional modules.

Custom Build

Geoff Stratton's article on rebuilding Nginx for Ubuntu was very helpful and I've had success with building it this way.

The modules that are required for the custom build are:

Nginx Configuration


server {
    listen 80 default_server;
    listen [::]:80 default_server;

    server_name _;

    location / {
        set_md5 $hash_key $http_host$request_uri;
        set $redis_key :1:$hash_key; # this is the format of the key that Django cache creates for this request
        set $redis_db 1;
        redis_pass     127.0.0.1:6379;
        default_type   text/html;
        error_page     404 @fallback;
    }

    location @fallback {
        proxy_pass       http://127.0.0.1:8001;
        proxy_set_header Host      $http_host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

Now you'll have responses being served up from Nginx from the same Redis Cache.

comments powered by Disqus