Skip to main content

Django Memcached

Introduction​

When building web applications with Django, performance becomes increasingly important as your user base grows. One of the most effective ways to improve performance is by implementing caching. In this tutorial, we'll explore Django Memcached, a popular caching backend that can significantly speed up your Django applications.

Memcached is a high-performance, distributed memory caching system that stores data in RAM rather than on disk, allowing for extremely fast data retrieval. It's designed to speed up dynamic web applications by reducing database load.

By the end of this tutorial, you'll understand how to set up, configure, and effectively use Memcached with Django to optimize your application's performance.

What is Memcached?​

Memcached is an open-source, distributed memory caching system that was originally developed by Danga Interactive for LiveJournal in 2003. It works by storing key-value pairs of data in RAM, which makes data retrieval much faster than fetching from a database or file system.

Key features of Memcached include:

  • Speed: Data is stored in memory for fast access
  • Simplicity: Simple protocol and straightforward implementation
  • Distributed: Can be spread across multiple machines
  • Scalability: Easy to scale by adding more servers

Prerequisites​

Before we get started, ensure you have:

  1. Django installed (version 3.0+ recommended)
  2. Memcached installed on your system
  3. A Django project set up
  4. python-memcached or pymemcache Python client library installed

Installing Memcached​

On Ubuntu/Debian​

bash
sudo apt-get update
sudo apt-get install memcached

On macOS (using Homebrew)​

bash
brew install memcached

On Windows​

For Windows, you can download the binary from the Memcached website or use the Windows Subsystem for Linux (WSL).

Installing the Python Client​

bash
# For python-memcached
pip install python-memcached

# OR for pymemcache (an alternative client)
pip install pymemcache

Configuring Django to Use Memcached​

To use Memcached with Django, you need to configure your cache settings in your project's settings.py file. Here's how to do it:

Basic Configuration​

python
CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.PyMemcacheCache',
        'LOCATION': '127.0.0.1:11211',
    }
}

Multiple Memcached Servers​

If you're running Memcached on multiple servers, you can configure Django to use them all:

python
CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.PyMemcacheCache',
        'LOCATION': [
            '172.19.26.240:11211',
            '172.19.26.242:11211',
        ]
    }
}

Using python-memcached Instead of pymemcache​

If you prefer to use the python-memcached library:

python
CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
        'LOCATION': '127.0.0.1:11211',
    }
}

Using Django's Cache Framework with Memcached​

Now that you have Memcached configured, let's look at how to use it in your Django application.

Site-wide Caching​

Django allows you to cache your entire site using middleware. Add the following to your settings.py:

python
MIDDLEWARE = [
    'django.middleware.cache.UpdateCacheMiddleware',
    # ... other middleware ...
    'django.middleware.cache.FetchFromCacheMiddleware',
]

CACHE_MIDDLEWARE_ALIAS = 'default'
CACHE_MIDDLEWARE_SECONDS = 600  # 10 minutes
CACHE_MIDDLEWARE_KEY_PREFIX = 'mysite'

Important: The order of middleware is crucial. UpdateCacheMiddleware should be first, and FetchFromCacheMiddleware should be last.

Per-View Caching​

For more granular control, you can cache individual views:

python
from django.views.decorators.cache import cache_page

@cache_page(60 * 15)  # cache for 15 minutes
def my_view(request):
    # View logic here
    return render(request, 'my_template.html', {'data': data})

Template Fragment Caching​

You can also cache parts of templates:

html
{% load cache %}

{% cache 500 sidebar request.user.username %}
    <!-- Expensive sidebar content -->
{% endcache %}

Low-level Cache API​

For more fine-grained control, you can use Django's low-level cache API:

python
from django.core.cache import cache

# Store a value in the cache
cache.set('my_key', 'my_value', timeout=300)  # Cache for 5 minutes

# Retrieve a value from the cache
value = cache.get('my_key')
if value is None:
    # Value not in cache, recalculate it
    value = expensive_calculation()
    cache.set('my_key', value, timeout=300)

# Delete a key from the cache
cache.delete('my_key')

# Clear the entire cache
cache.clear()

# Get or set a value (common pattern)
value = cache.get_or_set('my_key', expensive_calculation, timeout=300)

Real-world Example: Caching Database Queries​

Let's look at a real-world example of caching database queries to improve performance:

python
from django.core.cache import cache
from .models import Article

def get_latest_articles(count=10):
    cache_key = f'latest_articles_{count}'
    articles = cache.get(cache_key)
    
    if articles is None:
        # Cache miss, fetch from database
        articles = list(Article.objects.order_by('-published_date')[:count])
        cache.set(cache_key, articles, timeout=60*5)  # Cache for 5 minutes
    
    return articles

Then in your view:

python
def article_list(request):
    latest_articles = get_latest_articles()
    return render(request, 'articles/list.html', {'articles': latest_articles})

Advanced Memcached Techniques​

Cache Versioning​

You can use versioning to invalidate all cache entries at once:

python
# In settings.py
CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.PyMemcacheCache',
        'LOCATION': '127.0.0.1:11211',
        'VERSION': 1,  # Increment this to invalidate all cache entries
    }
}

Cache Keys​

Django generates cache keys based on the values you provide. To avoid key collisions, you can use prefixes:

python
# In settings.py
CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.PyMemcacheCache',
        'LOCATION': '127.0.0.1:11211',
        'KEY_PREFIX': 'mysite',  # Prefix for all cache keys
    }
}

Multiple Cache Setups​

You can define multiple cache configurations:

python
CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.PyMemcacheCache',
        'LOCATION': '127.0.0.1:11211',
        'TIMEOUT': 300,  # 5 minutes
    },
    'long_term': {
        'BACKEND': 'django.core.cache.backends.memcached.PyMemcacheCache',
        'LOCATION': '127.0.0.1:11212',
        'TIMEOUT': 3600,  # 1 hour
    }
}

And use them selectively:

python
from django.core.cache import caches

default_cache = caches['default']
long_term_cache = caches['long_term']

# Use different caches for different data
default_cache.set('user_profile', profile)
long_term_cache.set('site_settings', settings)

Common Memcached Patterns​

Cache Stampede Prevention​

When many concurrent requests try to rebuild the same expired cache entry, it can overload your database. Here's a pattern to prevent that:

python
from django.core.cache import cache
import functools

def cached_result(timeout=300):
    def decorator(func):
        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            # Create a unique cache key based on function name and arguments
            key = f"cached_{func.__name__}_{str(args)}_{str(kwargs)}"
            result = cache.get(key)
            
            if result is None:
                # Set a lock to prevent multiple rebuilds
                lock_key = f"lock_{key}"
                if not cache.get(lock_key):
                    # Set lock with a short timeout
                    cache.set(lock_key, True, timeout=10)
                    
                    # Calculate the result
                    result = func(*args, **kwargs)
                    
                    # Store in cache
                    cache.set(key, result, timeout=timeout)
                    
                    # Release lock
                    cache.delete(lock_key)
                else:
                    # Lock exists, wait and try again
                    import time
                    time.sleep(0.1)
                    return wrapper(*args, **kwargs)
            
            return result
        return wrapper
    return decorator

# Usage
@cached_result(timeout=300)
def get_expensive_data(user_id):
    # Expensive operation here
    return data

Caching Database Objects​

A common pattern is to cache model instances:

python
def get_user_profile(user_id):
    cache_key = f'user_profile_{user_id}'
    profile = cache.get(cache_key)
    
    if profile is None:
        try:
            profile = UserProfile.objects.get(user_id=user_id)
            cache.set(cache_key, profile, timeout=3600)  # Cache for 1 hour
        except UserProfile.DoesNotExist:
            # Handle case when profile doesn't exist
            return None
    
    return profile

Invalidating Cache on Model Changes​

You'll want to invalidate the cache when the underlying data changes:

python
from django.db.models.signals import post_save, post_delete
from django.dispatch import receiver
from django.core.cache import cache
from .models import UserProfile

@receiver([post_save, post_delete], sender=UserProfile)
def invalidate_profile_cache(sender, instance, **kwargs):
    cache_key = f'user_profile_{instance.user_id}'
    cache.delete(cache_key)

Performance Considerations​

Optimal Cache Key Size​

Memcached has limits on key sizes (usually 250 bytes). Keep your keys short and meaningful.

Memory Allocation​

Memcached allocates memory in fixed-size chunks. Understanding this can help you optimize your cache usage:

python
# In settings.py
CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.PyMemcacheCache',
        'LOCATION': '127.0.0.1:11211',
        'OPTIONS': {
            'server_max_value_length': 1024 * 1024 * 2,  # 2MB
        }
    }
}

Monitoring​

You can monitor your Memcached instance using various tools like:

  • memcached-tool
  • telnet to the Memcached server
  • Various monitoring systems like Datadog, New Relic, etc.

Troubleshooting​

Connection Issues​

If you're experiencing connection issues:

  1. Verify Memcached is running: ps aux | grep memcached
  2. Check that the port is open: telnet 127.0.0.1 11211
  3. Make sure your firewall allows connections to the Memcached port

Cache Not Working​

If your cache doesn't seem to be working:

  1. Check if your cache key exists by explicitly retrieving it
  2. Verify that the data is being saved to the cache correctly
  3. Check if the cache is full (Memcached will evict old entries when full)

Summary​

Django Memcached provides a powerful way to improve your application's performance by reducing database load and speeding up response times. In this tutorial, we've covered:

  • Setting up Memcached with Django
  • Different caching strategies (site-wide, per-view, template fragments)
  • Using the low-level cache API
  • Real-world examples and common patterns
  • Advanced techniques and performance considerations

By implementing appropriate caching strategies with Memcached, you can make your Django application faster and more scalable, providing a better experience for your users.

Additional Resources​

Exercises​

  1. Basic Caching: Implement view caching for a Django view that displays a list of blog posts.
  2. Template Fragment Caching: Cache a navigation menu that includes dynamic user information.
  3. Cache Invalidation: Create a signal handler that invalidates cache when a model is updated.
  4. Cache Patterns: Implement the cache stampede prevention pattern in a real application.
  5. Distributed Setup: Configure Django to use multiple Memcached instances and test the performance.

By working through these exercises, you'll gain practical experience with Django Memcached and develop a better understanding of when and how to implement caching in your applications.


If you spot any mistakes on this website, please let me know at [email protected]. I’d greatly appreciate your feedback! :)